55 Commits

Author SHA1 Message Date
fce03ebd3c Release 5.2.3 (#51) 2025-07-19 18:46:18 +09:00
f7b59909bd fix: 外部からファイルが参照できない問題を修正 (#50)
* fix: 外部からファイルが参照できない問題を修正

* Update Changelog
2025-07-19 18:44:20 +09:00
5582cc8c9b Release 5.2.2 (#49)
* Release 5.2.2

* Update CHANGELOG.md
2025-07-06 18:53:31 +09:00
7ebfe151d5 Update CHANGELOG.md 2025-07-04 10:39:38 +09:00
b0818bed3e fix: 最初のHEADリクエストにUAが反映されない問題を修正 (#48)
* fix: 最初のHEADリクエストにUAが反映されない問題を修正 (MisskeyIO#49)

* Update Changelog

---------

Co-authored-by: あわわわとーにゅ <17376330+u1-liquid@users.noreply.github.com>
2025-07-04 10:38:59 +09:00
909774bc63 update deps (#46)
* update deps

* migrate to vitest 1

* migrate to vitest 2

* migrate to vitest 3

* migrate to vitest 4

* lint fixes

* lint fixes

* lint fixes

* update deps
2025-07-04 10:33:07 +09:00
56e1137113 5.2.1 (#47) 2025-04-28 08:05:18 +09:00
441c744a35 Update CHANGELOG.md 2025-04-28 07:58:09 +09:00
632cd903ee Update CHANGELOG.md 2025-04-28 07:56:56 +09:00
45153b4f08 Merge commit from fork
Fix GHSA-7899-w6c4-vqc4
2025-04-28 07:52:26 +09:00
9e3f23691c Merge commit from fork
Fix `GHSA-jqx4-9gpq-rppm`
2025-04-28 07:52:03 +09:00
6d4ad44de3 Fix: Pass followRedirects in two spots I missed 2025-04-27 11:03:13 -04:00
23d5e03ba2 Fix GHSA-7899-w6c4-vqc4 2025-04-27 00:50:49 -04:00
dfe6451012 Fix GHSA-jqx4-9gpq-rppm 2025-04-27 00:30:13 -04:00
1b93243ff9 Release 5.2.0 (#44)
* Release 5.2.0

* Update CHANGELOG.md
2025-02-05 18:15:45 +09:00
ee06d841c2 update deps (#45)
* update deps

* lint fixes
2025-02-02 11:23:28 +09:00
e9547a556a fix: missing properties (#43)
* fix: missing properties

* Update Changelog for misskey-dev#41
2025-02-02 11:05:56 +09:00
71a6aefb8e fix(test): Github Actions上で外部サイトへの取得を行うテストをスキップするように (#38) 2025-02-02 10:59:35 +09:00
fc9f7db477 Add fediverse creator tag support (#41)
* add Fediverse Creator

* Update README.md
2025-02-02 10:59:18 +09:00
cc7ae8d00a enhance(plugin): blueskyのURLプレビューに対応 (#35)
* blueskyのURLプレビューに対応 (MisskeyIO#5)

(cherry picked from commit 5407ae09230ab44693f2198a5cf639ae3a95c941)

* Update Changelog

* refactor

* lint

* refactor

* lint

---------

Co-authored-by: たーびん <tar.bin.master@gmail.com>
2025-02-02 10:58:43 +09:00
7fbab86441 enhance(parse): sensitive判定を強化 (#36)
* enhance(parse): sensitive判定を強化

* Update Changelog
2024-11-15 17:44:17 +09:00
1b541a1418 fix(build): ビルド時にエイリアスが変換されない問題を修正 (#34) 2024-11-11 17:16:25 +09:00
3e09d27613 update deps (#33)
* update deps

* lint fixes

* fix

* trace-redirectを削除

Co-Authored-By: あわわわとーにゅ <17376330+u1-liquid@users.noreply.github.com>

* attempt to fix test

* refactor

* fix test

---------

Co-authored-by: あわわわとーにゅ <17376330+u1-liquid@users.noreply.github.com>
2024-11-11 04:14:01 +09:00
509a35abe2 Update CHANGELOG.md 2024-03-18 19:40:04 +09:00
79d9eadd58 fix: 次バージョンに向けての記載を追加 (#25) 2024-03-18 10:51:35 +09:00
c261071a82 fix: サマリ取得の動作改善+動作設定を可能にする (#23) 2024-03-17 16:53:32 +09:00
71fe234d3e chore: improve lint 2023-12-30 16:28:45 +00:00
718465e498 5.0.3 2023-12-30 10:26:31 +00:00
77c53be159 v5.0.2 2023-12-30 09:22:39 +00:00
541a0ddd0a fix workflow (v5.0.1) 2023-12-30 09:09:13 +00:00
7e94c26e9f fix workflow name 2023-12-30 08:37:41 +00:00
2f5ab74f52 v5.0.0 2023-12-30 08:36:45 +00:00
7902ded327 なんかめっちゃ変えた 2023-12-30 08:36:01 +00:00
67909d360d npm install @misskey-dev/summaly 2023-12-30 07:14:54 +00:00
0606cd3fae Update workflow to change name to Test 2023-12-30 07:04:46 +00:00
52f5692199 update typescript 2023-12-30 07:03:53 +00:00
3ecb30aefc ignore built, remove travis.yml 2023-12-30 06:59:59 +00:00
c45b3a1c63 add workflows, mod chglog, remove built 2023-12-30 06:53:31 +00:00
81de408b54 remove package-lock.json
https://github.com/misskey-dev/summaly/issues/18
2023-12-30 06:42:33 +00:00
38000dd462 feat: support 'mixi:content-rating' for sensitive (#16)
* feat: support 'mixi:content-rating' for sensitive

* fix: d7f6a88aee でビルドしてない

* ビルドする
2023-12-30 15:33:43 +09:00
d7f6a88aee fix: twitter card with name attr not working (#15) 2023-12-02 21:05:33 +09:00
d2a3e07205 fix: branch.ioを用いたディープリンクをパースできるように修正 (#13)
* (fix) branch.ioのディープリンクでうまく作動しないのを修正

* Update changelog

* fix regex
2023-11-15 19:23:03 +09:00
d2d8db4994 fix: use activity+json instead of activitypub+json (#11) 2023-08-06 19:30:20 +02:00
089a0ad8e8 feat: support <link rel=alternate type=application/activitypub+json> (#10)
* feat: support `<link rel=alternate type=application/activitypub+json>`

* Update index.ts

* build
2023-07-20 15:36:26 +09:00
77dd5654bb README.md 2023-05-07 12:19:50 +00:00
2d63e2a006 fix README.md 2023-05-04 17:18:10 +00:00
c7d71a9ec2 Merge branch 'master' of https://github.com/misskey-dev/summaly 2023-04-20 04:02:59 +00:00
994f420b46 4.0.2 2023-04-20 04:02:55 +00:00
5a3321a04f fix: allow legacy allowfullscreen (#9) 2023-04-20 12:41:11 +09:00
1bab7afee6 Merge branch 'master' of https://github.com/misskey-dev/summaly 2023-03-16 03:26:10 +00:00
441e8c22f9 v4.0.1 2023-03-16 03:26:00 +00:00
376bba9c61 fix: give null when oEmbed access fails (#8) 2023-03-16 12:22:23 +09:00
028b2fed2f fix README.md 2023-03-13 18:03:16 +00:00
90d5d0f33b Fix README.md 2023-03-13 18:02:07 +00:00
9e955d8d04 fix readme 2023-03-13 17:57:37 +00:00
76 changed files with 5412 additions and 13724 deletions

View File

@ -5,6 +5,11 @@ indent_style = tab
indent_size = 2
charset = utf-8
insert_final_newline = true
end_of_line = lf
trim_trailing_whitespace = true
[*.json]
[*.md]
trim_trailing_whitespace = false
[*.{yml,yaml}]
indent_style = space

31
.github/workflows/lint.yml vendored Normal file
View File

@ -0,0 +1,31 @@
name: Lint
on: [push, pull_request]
jobs:
lint:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [20.10.0]
steps:
- uses: actions/checkout@v4
- name: Install pnpm
uses: pnpm/action-setup@v2
with:
version: 8
run_install: false
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: 'pnpm'
- name: Install
run: |
corepack enable
pnpm i --frozen-lockfile
- name: eslint
run: |
pnpm eslint

42
.github/workflows/npm-publish.yml vendored Normal file
View File

@ -0,0 +1,42 @@
name: Publish Node.js Package
on:
release:
types: [created]
workflow_dispatch:
jobs:
publish:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write
strategy:
matrix:
node-version: [20.10.0]
steps:
- uses: actions/checkout@v4
- name: Install pnpm
uses: pnpm/action-setup@v2
with:
version: 8
run_install: false
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: 'pnpm'
registry-url: 'https://registry.npmjs.org'
- name: Publish package
run: |
corepack enable
pnpm i --frozen-lockfile
pnpm build
pnpm publish --access public --no-git-checks --provenance
env:
NODE_AUTH_TOKEN: ${{ secrets.NODE_AUTH_TOKEN }}
NPM_TOKEN: ${{ secrets.NODE_AUTH_TOKEN }}

34
.github/workflows/test.yml vendored Normal file
View File

@ -0,0 +1,34 @@
name: Test
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [20.10.0]
steps:
- uses: actions/checkout@v4
- name: Install pnpm
uses: pnpm/action-setup@v2
with:
version: 8
run_install: false
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: 'pnpm'
- name: Install
run: |
corepack enable
pnpm i --frozen-lockfile
- name: Build
run: |
pnpm build
- name: Test
run: |
SKIP_NETWORK_TEST=true pnpm test

1
.gitignore vendored
View File

@ -1,2 +1,3 @@
/node_modules
/built
npm-debug.log

View File

@ -5,5 +5,5 @@
npm-debug.log
gulpfile.js
tsconfig.json
tslint.json
.eslintrc.cjs
.editorconfig

24
.swcrc
View File

@ -1,24 +0,0 @@
{
"$schema": "https://json.schemastore.org/swcrc",
"jsc": {
"parser": {
"syntax": "typescript",
"dynamicImport": true,
"decorators": true
},
"transform": {
"legacyDecorator": true,
"decoratorMetadata": true
},
"experimental": {
"keepImportAssertions": true
},
"baseUrl": ".",
"paths": {},
"target": "es2021"
},
"module": {
"type": "es6"
},
"minify": true
}

View File

@ -1,14 +0,0 @@
# travis file
# https://docs.travis-ci.com/user/customizing-the-build
language: node_js
node_js:
- 7.5.0
before_script:
- npm run build
cache:
directories:
- node_modules

View File

@ -1,3 +1,69 @@
(unreleased)
------------------
5.2.3 / 2025/07/19
------------------
* パッケージが使用できない問題を修正
5.2.2 / 2025/07/06
------------------
* 最初のHEADリクエストにUAが反映されない問題を修正
* 依存関係の更新
* テストスイートをVitestに変更
5.2.1 / 2025/04/28
------------------
* セキュリティに関する修正
5.2.0 / 2025/02/05
------------------
* センシティブフラグの判定を `<meta property="rating">` および `rating` ヘッダでも行うように
* Blueskybsky.appのプレビューに対応
* `fediverse:creator` のパースに対応
* 依存関係の更新
* eslintの設定を更新
5.1.0 / 2024-03-18
------------------
* GETリクエストよりも前にHEADリクエストを送信し、その結果を使用して検証するように (#22)
* 下記のパラメータを`summaly`メソッドのオプションに追加
- userAgent
- responseTimeout
- operationTimeout
- contentLengthLimit
- contentLengthRequired
5.0.3 / 2023-12-30
------------------
* Fix .github/workflows/npm-publish.yml
5.0.2 / 2023-12-30
------------------
* Fix .github/workflows/npm-publish.yml
5.0.1 / 2023-12-30
------------------
* Fix .github/workflows/npm-publish.yml
5.0.0 / 2023-12-30
------------------
* support `<link rel="alternate" type="application/activitypub+json" href="{href}">` https://github.com/misskey-dev/summaly/pull/10, https://github.com/misskey-dev/summaly/pull/11
* 結果の`activityPub`プロパティでherfの内容を取得できます
* branch.ioを用いたディープリンクspotify.linkなどでパースに失敗する問題を修正 https://github.com/misskey-dev/summaly/pull/13
* Twitter Cardが読めていない問題を修正 https://github.com/misskey-dev/summaly/pull/15
* 'mixi:content-rating'をsensitive判定で見ることで、dlsiteなどでセンシティブ情報を得れるように https://github.com/misskey-dev/summaly/pull/16
* sitenameをURLから生成する場合、ポートを含むように (URL.hostname → URL.host)
* `Summary`型に`url`プロパティを追加した`SummalyResult`型をexportするように
* `IPlugin`インターフェースを`SummalyPlugin`に改称
4.0.2 / 2023-04-20
------------------
* YouTubeをフルスクリーンにできない問題を修正
4.0.1 / 2023-03-16
------------------
* oEmbedの読み込みでエラーが発生した際は、エラーにせずplayerの中身をnullにするように
4.0.0 / 2023-03-14
------------------
* oEmbed type=richの制限的なサポート

View File

@ -1,6 +1,6 @@
The MIT License (MIT)
Copyright (c) 2016-2019 syuilo
Copyright (c) 2016-2024 syuilo
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal

102
README.md
View File

@ -1,6 +1,7 @@
summaly
================================================================
[![][npm-badge]][npm-link]
[![][mit-badge]][mit]
[![][himawari-badge]][himasaku]
[![][sakurako-badge]][himasaku]
@ -8,7 +9,7 @@ summaly
Installation
----------------------------------------------------------------
```
npm install git+https://github.com/misskey-dev/summaly.git
npm install @misskey-dev/summaly
```
Usage
@ -40,17 +41,24 @@ npm run build
npm run serve
```
### Options
#### opts (SummalyOptions)
| Property | Type | Description | Default |
| :------------------ | :--------------------- | :----------------------- | :------ |
|:--------------------------|:-----------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------|
| **lang** | *string* | Accept-Language for the request | `null` |
| **followRedirects** | *boolean* | Whether follow redirects | `true` |
| **plugins** | *plugin[]* (see below) | Custom plugins | `null` |
| **agent** | *Got.Agents* | Custom HTTP agent (see below) | `null` |
| **userAgent** | *string* | User-Agent for the request | `SummalyBot/[version]` |
| **responseTimeout** | *number* | Set timeouts for each phase, such as host name resolution and socket communication. | `20000` |
| **operationTimeout** | *number* | Set the timeout from the start to the end of the request. | `60000` |
| **contentLengthLimit** | *number* | If set to true, an error will occur if the content-length value returned from the other server is larger than this parameter (or if the received body size exceeds this parameter). | `10485760` |
| **contentLengthRequired** | *boolean* | If set to true, it will be an error if the other server does not return content-length. | `false` |
#### Plugin
``` typescript
interface IPlugin {
interface SummalyPlugin {
test: (url: URL) => boolean;
summarize: (url: URL) => Promise<Summary>;
}
@ -58,32 +66,47 @@ interface IPlugin {
urls are WHATWG URL since v4.
#### Custom HTTP agent for proxy
You can specify agents to be passed to Got for proxy use, etc.
https://github.com/sindresorhus/got/blob/v12.6.0/documentation/tips.md#proxying
**⚠If you set some agent, local IP rejecting will not work.⚠️**
(Summaly usually rejects local IPs.)
(Summaly currently does not support http2.)
### Returns
A Promise of an Object that contains properties below:
※ Almost all values are nullable. player should not be null.
#### Root
#### SummalyResult
| Property | Type | Description |
| :-------------- | :------- | :------------------------------------------ |
| **description** | *string* | The description of the web page |
| **icon** | *string* | The url of the icon of the web page |
| **sitename** | *string* | The name of the web site |
| **thumbnail** | *string* | The url of the thumbnail of the web page |
| **oEmbed** | *OEmbedRichIframe* | The oEmbed rich iframe info of the web page |
|:----------------|:-------------------|:-----------------------------------------------------------|
| **title** | *string* \| *null* | The title of the web page |
| **icon** | *string* \| *null* | The url of the icon of the web page |
| **description** | *string* \| *null* | The description of the web page |
| **thumbnail** | *string* \| *null* | The url of the thumbnail of the web page |
| **sitename** | *string* \| *null* | The name of the web site |
| **player** | *Player* | The player of the web page |
| **title** | *string* | The title of the web page |
| **sensitive** | *boolean* | Whether the url is sensitive |
| **activityPub** | *string* \| *null* | The url of the ActivityPub representation of that web page |
| **fediverseCreator** | *string* \| *null* | The pages fediverse handle |
| **url** | *string* | The url of the web page |
#### Summary
`Omit<SummalyResult, "url">`
#### Player
| Property | Type | Description |
| :-------------- | :--------- | :---------------------------------------------- |
| **url** | *string* | The url of the player |
| **width** | *number* | The width of the player |
| **height** | *number* | The height of the player |
|:-----------|:-------------------|:------------------------------------------------|
| **url** | *string* \| *null* | The url of the player |
| **width** | *number* \| *null* | The width of the player |
| **height** | *number* \| *null* | The height of the player |
| **allow** | *string[]* | The names of the allowed permissions for iframe |
Currently the possible items in `allow` are:
@ -93,32 +116,46 @@ Currently the possible items in `allow` are:
* `fullscreen`
* `encrypted-media`
* `picture-in-picture`
* `web-share`
See [Permissions Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/Permissions_Policy) in MDN for details of them.
### Example
``` javascript
```javascript
import { summaly } from 'summaly';
const summary = await summaly('https://www.youtube.com/watch?v=NMIEAhH_fTU');
console.log(summary); // will be ... ↓
/*
console.log(summary);
```
will be ... ↓
```json
{
title: '【楽曲試聴】「Stage Bye Stage」(歌:島村卯月、渋谷凛、本田未央)',
icon: 'https://s.ytimg.com/yts/img/favicon-vfl8qSV2F.ico',
description: 'http://columbia.jp/idolmaster/ 2018年7月18日発売予定 THE IDOLM@STER CINDERELLA GIRLS CG STAR LIVE Stage Bye Stage 歌:島村卯月、渋谷凛、本田未央 COCC-17495CD1枚組 ¥1,200税 収録内容 Tr...',
thumbnail: 'https://i.ytimg.com/vi/NMIEAhH_fTU/maxresdefault.jpg',
player: {
url: 'https://www.youtube.com/embed/NMIEAhH_fTU',
width: 1280,
height: 720
"title": "【アイドルマスター】「Stage Bye Stage」(歌:島村卯月、渋谷凛、本田未央)",
"icon": "https://www.youtube.com/s/desktop/711fd789/img/logos/favicon.ico",
"description": "Website▶https://columbia.jp/idolmaster/Playlist▶https://www.youtube.com/playlist?list=PL83A2998CF3BBC86D2018年7月18日発売予定THE IDOLM@STER CINDERELLA GIRLS CG STAR...",
"thumbnail": "https://i.ytimg.com/vi/NMIEAhH_fTU/maxresdefault.jpg",
"player": {
"url": "https://www.youtube.com/embed/NMIEAhH_fTU?feature=oembed",
"width": 200,
"height": 113,
"allow": [
"autoplay",
"clipboard-write",
"encrypted-media",
"picture-in-picture",
"web-share",
"fullscreen",
]
},
sitename: 'YouTube',
url: 'https://www.youtube.com/watch?v=NMIEAhH_fTU'
"sitename": "YouTube",
"sensitive": false,
"activityPub": null,
"url": "https://www.youtube.com/watch?v=NMIEAhH_fTU"
}
*/
```
Testing
@ -129,5 +166,10 @@ License
----------------------------------------------------------------
[MIT](LICENSE)
[mit]: http://opensource.org/licenses/MIT
[mit-badge]: https://img.shields.io/badge/license-MIT-444444.svg?style=flat-square
[himasaku]: https://himasaku.net
[himawari-badge]: https://img.shields.io/badge/%E5%8F%A4%E8%B0%B7-%E5%90%91%E6%97%A5%E8%91%B5-1684c5.svg?style=flat-square
[sakurako-badge]: https://img.shields.io/badge/%E5%A4%A7%E5%AE%A4-%E6%AB%BB%E5%AD%90-efb02a.svg?style=flat-square
[npm-link]: https://www.npmjs.com/package/@misskey-dev/summaly
[npm-badge]: https://img.shields.io/npm/v/@misskey-dev/summaly.svg?style=flat-square

4
built/general.d.ts vendored
View File

@ -1,4 +0,0 @@
import { URL } from 'node:url';
import type { default as Summary } from './summary.js';
declare const _default: (_url: URL | string, lang?: string | null) => Promise<Summary | null>;
export default _default;

View File

@ -1,188 +0,0 @@
import { URL } from 'node:url';
import clip from './utils/clip.js';
import cleanupTitle from './utils/cleanup-title.js';
import { decode as decodeHtml } from 'html-entities';
import { get, head, scpaping } from './utils/got.js';
import * as cheerio from 'cheerio';
/**
* Contains only the html snippet for a sanitized iframe as the thumbnail is
* mostly covered in OpenGraph instead.
*
* Width should always be 100%.
*/
async function getOEmbedPlayer($, pageUrl) {
const href = $('link[type="application/json+oembed"]').attr('href');
if (!href) {
return null;
}
const oEmbed = await get((new URL(href, pageUrl)).href);
const body = (() => {
try {
return JSON.parse(oEmbed);
}
catch { }
})();
if (!body || body.version !== '1.0' || !['rich', 'video'].includes(body.type)) {
// Not a well formed rich oEmbed
return null;
}
if (!body.html.startsWith('<iframe ') || !body.html.endsWith('</iframe>')) {
// It includes something else than an iframe
return null;
}
const oEmbedHtml = cheerio.load(body.html);
const iframe = oEmbedHtml("iframe");
if (iframe.length !== 1) {
// Somehow we either have multiple iframes or none
return null;
}
if (iframe.parents().length !== 2) {
// Should only have the body and html elements as the parents
return null;
}
const url = iframe.attr('src');
if (!url) {
// No src?
return null;
}
try {
if ((new URL(url)).protocol !== 'https:') {
// Allow only HTTPS for best security
return null;
}
}
catch (e) {
return null;
}
// Height is the most important, width is okay to be null. The implementer
// should choose fixed height instead of fixed aspect ratio if width is null.
//
// For example, Spotify's embed page does not strictly follow aspect ratio
// and thus keeping the height is better than keeping the aspect ratio.
//
// Spotify gives `width: 100%, height: 152px` for iframe while `width: 456,
// height: 152` for oEmbed data, and we treat any percentages as null here.
let width = Number(iframe.attr('width') ?? body.width);
if (Number.isNaN(width)) {
width = null;
}
const height = Math.min(Number(iframe.attr('height') ?? body.height), 1024);
if (Number.isNaN(height)) {
// No proper height info
return null;
}
// TODO: This implementation only allows basic syntax of `allow`.
// Might need to implement better later.
const safeList = [
'autoplay',
'clipboard-write',
'fullscreen',
'encrypted-media',
'picture-in-picture',
'web-share',
];
// YouTube has these but they are almost never used.
const ignoredList = [
'gyroscope',
'accelerometer',
];
const allowedPermissions = (iframe.attr('allow') ?? '').split(/\s*;\s*/g)
.filter(s => s)
.filter(s => !ignoredList.includes(s));
if (allowedPermissions.some(allow => !safeList.includes(allow))) {
// This iframe is probably too powerful to be embedded
return null;
}
return {
url,
width,
height,
allow: allowedPermissions
};
}
export default async (_url, lang = null) => {
if (lang && !lang.match(/^[\w-]+(\s*,\s*[\w-]+)*$/))
lang = null;
const url = typeof _url === 'string' ? new URL(_url) : _url;
const res = await scpaping(url.href, { lang: lang || undefined });
const $ = res.$;
const twitterCard = $('meta[property="twitter:card"]').attr('content');
let title = $('meta[property="og:title"]').attr('content') ||
$('meta[property="twitter:title"]').attr('content') ||
$('title').text();
if (title === undefined || title === null) {
return null;
}
title = clip(decodeHtml(title), 100);
let image = $('meta[property="og:image"]').attr('content') ||
$('meta[property="twitter:image"]').attr('content') ||
$('link[rel="image_src"]').attr('href') ||
$('link[rel="apple-touch-icon"]').attr('href') ||
$('link[rel="apple-touch-icon image_src"]').attr('href');
image = image ? (new URL(image, url.href)).href : null;
const playerUrl = (twitterCard !== 'summary_large_image' && $('meta[property="twitter:player"]').attr('content')) ||
(twitterCard !== 'summary_large_image' && $('meta[name="twitter:player"]').attr('content')) ||
$('meta[property="og:video"]').attr('content') ||
$('meta[property="og:video:secure_url"]').attr('content') ||
$('meta[property="og:video:url"]').attr('content');
const playerWidth = parseInt($('meta[property="twitter:player:width"]').attr('content') ||
$('meta[name="twitter:player:width"]').attr('content') ||
$('meta[property="og:video:width"]').attr('content') ||
'');
const playerHeight = parseInt($('meta[property="twitter:player:height"]').attr('content') ||
$('meta[name="twitter:player:height"]').attr('content') ||
$('meta[property="og:video:height"]').attr('content') ||
'');
let description = $('meta[property="og:description"]').attr('content') ||
$('meta[property="twitter:description"]').attr('content') ||
$('meta[name="description"]').attr('content');
description = description
? clip(decodeHtml(description), 300)
: null;
if (title === description) {
description = null;
}
let siteName = decodeHtml($('meta[property="og:site_name"]').attr('content') ||
$('meta[name="application-name"]').attr('content') ||
url.hostname);
const favicon = $('link[rel="shortcut icon"]').attr('href') ||
$('link[rel="icon"]').attr('href') ||
'/favicon.ico';
const sensitive = $('.tweet').attr('data-possibly-sensitive') === 'true';
const find = async (path) => {
const target = new URL(path, url.href);
try {
await head(target.href);
return target;
}
catch (e) {
return null;
}
};
const getIcon = async () => {
return (await find(favicon)) || null;
};
const [icon, oEmbed] = await Promise.all([
getIcon(),
getOEmbedPlayer($, url.href),
]);
// Clean up the title
title = cleanupTitle(title, siteName);
if (title === '') {
title = siteName;
}
return {
title: title || null,
icon: icon?.href || null,
description: description || null,
thumbnail: image || null,
player: oEmbed ?? {
url: playerUrl || null,
width: Number.isNaN(playerWidth) ? null : playerWidth,
height: Number.isNaN(playerHeight) ? null : playerHeight,
allow: ['autoplay', 'encrypted-media', 'fullscreen'],
},
sitename: siteName || null,
sensitive,
};
};

39
built/index.d.ts vendored
View File

@ -1,39 +0,0 @@
/**
* summaly
* https://github.com/syuilo/summaly
*/
import Summary from './summary.js';
import type { IPlugin as _IPlugin } from './iplugin.js';
export declare type IPlugin = _IPlugin;
import * as Got from 'got';
import type { FastifyInstance } from 'fastify';
declare type Options = {
/**
* Accept-Language for the request
*/
lang?: string | null;
/**
* Whether follow redirects
*/
followRedirects?: boolean;
/**
* Custom Plugins
*/
plugins?: IPlugin[];
/**
* Custom HTTP agent
*/
agent?: Got.Agents;
};
declare type Result = Summary & {
/**
* The actual url of that web page
*/
url: string;
};
/**
* Summarize an web page
*/
export declare const summaly: (url: string, options?: Options | undefined) => Promise<Result>;
export default function (fastify: FastifyInstance, options: Options, done: (err?: Error) => void): void;
export {};

View File

@ -1,68 +0,0 @@
/**
* summaly
* https://github.com/syuilo/summaly
*/
import { URL } from 'node:url';
import tracer from 'trace-redirect';
import general from './general.js';
import { setAgent } from './utils/got.js';
import { plugins as builtinPlugins } from './plugins/index.js';
const defaultOptions = {
lang: null,
followRedirects: true,
plugins: [],
};
/**
* Summarize an web page
*/
export const summaly = async (url, options) => {
if (options?.agent)
setAgent(options.agent);
const opts = Object.assign(defaultOptions, options);
const plugins = builtinPlugins.concat(opts.plugins || []);
let actualUrl = url;
if (opts.followRedirects) {
// .catch(() => url)にすればいいけど、jestにtrace-redirectを食わせるのが面倒なのでtry-catch
try {
actualUrl = await tracer(url);
}
catch (e) {
actualUrl = url;
}
}
const _url = new URL(actualUrl);
// Find matching plugin
const match = plugins.filter(plugin => plugin.test(_url))[0];
// Get summary
const summary = await (match ? match.summarize : general)(_url, opts.lang || undefined);
if (summary == null) {
throw 'failed summarize';
}
return Object.assign(summary, {
url: actualUrl
});
};
export default function (fastify, options, done) {
fastify.get('/', async (req, reply) => {
const url = req.query.url;
if (url == null) {
return reply.status(400).send({
error: 'url is required'
});
}
try {
const summary = await summaly(url, {
lang: req.query.lang,
followRedirects: false,
...options,
});
return summary;
}
catch (e) {
return reply.status(500).send({
error: e
});
}
});
done();
}

7
built/iplugin.d.ts vendored
View File

@ -1,7 +0,0 @@
/// <reference types="node" />
import type { URL } from 'node:url';
import Summary from './summary.js';
export interface IPlugin {
test: (url: URL) => boolean;
summarize: (url: URL, lang?: string) => Promise<Summary>;
}

View File

@ -1 +0,0 @@
export {};

View File

@ -1,5 +0,0 @@
/// <reference types="node" />
import { URL } from 'node:url';
import summary from '../summary.js';
export declare function test(url: URL): boolean;
export declare function summarize(url: URL): Promise<summary>;

View File

@ -1,44 +0,0 @@
import { scpaping } from '../utils/got.js';
export function test(url) {
return url.hostname === 'www.amazon.com' ||
url.hostname === 'www.amazon.co.jp' ||
url.hostname === 'www.amazon.ca' ||
url.hostname === 'www.amazon.com.br' ||
url.hostname === 'www.amazon.com.mx' ||
url.hostname === 'www.amazon.co.uk' ||
url.hostname === 'www.amazon.de' ||
url.hostname === 'www.amazon.fr' ||
url.hostname === 'www.amazon.it' ||
url.hostname === 'www.amazon.es' ||
url.hostname === 'www.amazon.nl' ||
url.hostname === 'www.amazon.cn' ||
url.hostname === 'www.amazon.in' ||
url.hostname === 'www.amazon.au';
}
export async function summarize(url) {
const res = await scpaping(url.href);
const $ = res.$;
const title = $('#title').text();
const description = $('#productDescription').text() ||
$('meta[name="description"]').attr('content');
const thumbnail = $('#landingImage').attr('src');
const playerUrl = $('meta[property="twitter:player"]').attr('content') ||
$('meta[name="twitter:player"]').attr('content');
const playerWidth = $('meta[property="twitter:player:width"]').attr('content') ||
$('meta[name="twitter:player:width"]').attr('content');
const playerHeight = $('meta[property="twitter:player:height"]').attr('content') ||
$('meta[name="twitter:player:height"]').attr('content');
return {
title: title ? title.trim() : null,
icon: 'https://www.amazon.com/favicon.ico',
description: description ? description.trim() : null,
thumbnail: thumbnail ? thumbnail.trim() : null,
player: {
url: playerUrl || null,
width: playerWidth ? parseInt(playerWidth) : null,
height: playerHeight ? parseInt(playerHeight) : null,
allow: playerUrl ? ['fullscreen', 'encrypted-media'] : [],
},
sitename: 'Amazon',
};
}

View File

@ -1,2 +0,0 @@
import { IPlugin } from '@/iplugin.js';
export declare const plugins: IPlugin[];

View File

@ -1,6 +0,0 @@
import * as amazon from './amazon.js';
import * as wikipedia from './wikipedia.js';
export const plugins = [
amazon,
wikipedia,
];

View File

@ -1,5 +0,0 @@
/// <reference types="node" />
import { URL } from 'node:url';
import summary from '../summary.js';
export declare function test(url: URL): boolean;
export declare function summarize(url: URL): Promise<summary>;

View File

@ -1,37 +0,0 @@
import { get } from '../utils/got.js';
import debug from 'debug';
import clip from './../utils/clip.js';
const log = debug('summaly:plugins:wikipedia');
export function test(url) {
if (!url.hostname)
return false;
return /\.wikipedia\.org$/.test(url.hostname);
}
export async function summarize(url) {
const lang = url.host ? url.host.split('.')[0] : null;
const title = url.pathname ? url.pathname.split('/')[2] : null;
const endpoint = `https://${lang}.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro=&explaintext=&titles=${title}`;
log(`lang is ${lang}`);
log(`title is ${title}`);
log(`endpoint is ${endpoint}`);
let body = await get(endpoint);
body = JSON.parse(body);
log(body);
if (!('query' in body) || !('pages' in body.query)) {
throw 'fetch failed';
}
const info = body.query.pages[Object.keys(body.query.pages)[0]];
return {
title: info.title,
icon: 'https://wikipedia.org/static/favicon/wikipedia.ico',
description: clip(info.extract, 300),
thumbnail: `https://wikipedia.org/static/images/project-logos/${lang}wiki.png`,
player: {
url: null,
width: null,
height: null,
allow: [],
},
sitename: 'Wikipedia',
};
}

View File

@ -1 +0,0 @@
export {};

View File

@ -1,22 +0,0 @@
import * as http from 'http';
import * as Koa from 'koa';
import summaly from '../';
const app = new Koa();
app.use(async (ctx) => {
if (!ctx.query.url) {
ctx.status = 400;
return;
}
try {
const summary = await summaly(ctx.query.url, {
lang: ctx.query.lang,
followRedirects: false
});
ctx.body = summary;
}
catch (e) {
ctx.status = 500;
}
});
const server = http.createServer(app.callback());
server.listen(process.env.PORT || 80);

49
built/summary.d.ts vendored
View File

@ -1,49 +0,0 @@
declare type Summary = {
/**
* The description of that web page
*/
description: string | null;
/**
* The url of the icon of that web page
*/
icon: string | null;
/**
* The name of site of that web page
*/
sitename: string | null;
/**
* The url of the thumbnail of that web page
*/
thumbnail: string | null;
/**
* The player of that web page
*/
player: Player;
/**
* The title of that web page
*/
title: string | null;
/**
* Possibly sensitive
*/
sensitive?: boolean;
};
export default Summary;
export declare type Player = {
/**
* The url of the player
*/
url: string | null;
/**
* The width of the player
*/
width: number | null;
/**
* The height of the player
*/
height: number | null;
/**
* The allowed permissions of the iframe
*/
allow: string[];
};

View File

@ -1 +0,0 @@
export {};

View File

@ -1 +0,0 @@
export default function (title: string, siteName?: string | null): string;

View File

@ -1,19 +0,0 @@
import escapeRegExp from 'escape-regexp';
export default function (title, siteName) {
title = title.trim();
if (siteName) {
siteName = siteName.trim();
const x = escapeRegExp(siteName);
const patterns = [
`^(.+?)\\s?[\\-\\|:・]\\s?${x}$`
];
for (let i = 0; i < patterns.length; i++) {
const pattern = new RegExp(patterns[i]);
const [, match] = pattern.exec(title) || [null, null];
if (match) {
return match;
}
}
}
return title;
}

View File

@ -1 +0,0 @@
export default function (s: string, max: number): string;

View File

@ -1,13 +0,0 @@
import nullOrEmpty from './null-or-empty.js';
export default function (s, max) {
if (nullOrEmpty(s)) {
return s;
}
s = s.trim();
if (s.length > max) {
return s.substr(0, max) + '...';
}
else {
return s;
}
}

View File

@ -1,8 +0,0 @@
/// <reference types="node" />
/**
* Detect HTML encoding
* @param body Body in Buffer
* @returns encoding
*/
export declare function detectEncoding(body: Buffer): string;
export declare function toUtf8(body: Buffer, encoding: string): string;

View File

@ -1,40 +0,0 @@
import iconv from 'iconv-lite';
import jschardet from 'jschardet';
const regCharset = new RegExp(/charset\s*=\s*["']?([\w-]+)/, 'i');
/**
* Detect HTML encoding
* @param body Body in Buffer
* @returns encoding
*/
export function detectEncoding(body) {
// By detection
const detected = jschardet.detect(body, { minimumThreshold: 0.99 });
if (detected) {
const candicate = detected.encoding;
const encoding = toEncoding(candicate);
if (encoding != null)
return encoding;
}
// From meta
const matchMeta = body.toString('ascii').match(regCharset);
if (matchMeta) {
const candicate = matchMeta[1];
const encoding = toEncoding(candicate);
if (encoding != null)
return encoding;
}
return 'utf-8';
}
export function toUtf8(body, encoding) {
return iconv.decode(body, encoding);
}
function toEncoding(candicate) {
if (iconv.encodingExists(candicate)) {
if (['shift_jis', 'shift-jis', 'windows-31j', 'x-sjis'].includes(candicate.toLowerCase()))
return 'cp932';
return candicate;
}
else {
return null;
}
}

20
built/utils/got.d.ts vendored
View File

@ -1,20 +0,0 @@
import * as Got from 'got';
import * as cheerio from 'cheerio';
export declare let agent: Got.Agents;
export declare function setAgent(_agent: Got.Agents): void;
export declare type GotOptions = {
url: string;
method: 'GET' | 'POST' | 'HEAD';
body?: string;
headers: Record<string, string | undefined>;
typeFilter?: RegExp;
};
export declare function scpaping(url: string, opts?: {
lang?: string;
}): Promise<{
body: string;
$: cheerio.CheerioAPI;
response: Got.Response<string>;
}>;
export declare function get(url: string): Promise<string>;
export declare function head(url: string): Promise<Got.Response<string>>;

View File

@ -1,124 +0,0 @@
import got, * as Got from 'got';
import { StatusError } from './status-error.js';
import { detectEncoding, toUtf8 } from './encoding.js';
import * as cheerio from 'cheerio';
import PrivateIp from 'private-ip';
import { dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { readFileSync } from 'node:fs';
const _filename = fileURLToPath(import.meta.url);
const _dirname = dirname(_filename);
export let agent = {};
export function setAgent(_agent) {
agent = _agent || {};
}
const repo = JSON.parse(readFileSync(`${_dirname}/../../package.json`, 'utf8'));
const RESPONSE_TIMEOUT = 20 * 1000;
const OPERATION_TIMEOUT = 60 * 1000;
const MAX_RESPONSE_SIZE = 10 * 1024 * 1024;
const BOT_UA = `SummalyBot/${repo.version}`;
export async function scpaping(url, opts) {
const response = await getResponse({
url,
method: 'GET',
headers: {
'accept': 'text/html,application/xhtml+xml',
'user-agent': BOT_UA,
'accept-language': opts?.lang
},
typeFilter: /^(text\/html|application\/xhtml\+xml)/,
});
// SUMMALY_ALLOW_PRIVATE_IPはテスト用
const allowPrivateIp = process.env.SUMMALY_ALLOW_PRIVATE_IP === 'true' || Object.keys(agent).length > 0;
if (!allowPrivateIp && response.ip && PrivateIp(response.ip)) {
throw new StatusError(`Private IP rejected ${response.ip}`, 400, 'Private IP Rejected');
}
const encoding = detectEncoding(response.rawBody);
const body = toUtf8(response.rawBody, encoding);
const $ = cheerio.load(body);
return {
body,
$,
response,
};
}
export async function get(url) {
const res = await getResponse({
url,
method: 'GET',
headers: {
'accept': '*/*',
},
});
return await res.body;
}
export async function head(url) {
const res = await getResponse({
url,
method: 'HEAD',
headers: {
'accept': '*/*',
},
});
return await res;
}
async function getResponse(args) {
const timeout = RESPONSE_TIMEOUT;
const operationTimeout = OPERATION_TIMEOUT;
const req = got(args.url, {
method: args.method,
headers: args.headers,
body: args.body,
timeout: {
lookup: timeout,
connect: timeout,
secureConnect: timeout,
socket: timeout,
response: timeout,
send: timeout,
request: operationTimeout, // whole operation timeout
},
agent,
http2: false,
retry: {
limit: 0,
},
});
return await receiveResponse({ req, typeFilter: args.typeFilter });
}
async function receiveResponse(args) {
const req = args.req;
const maxSize = MAX_RESPONSE_SIZE;
req.on('response', (res) => {
// Check html
if (args.typeFilter && !res.headers['content-type']?.match(args.typeFilter)) {
// console.warn(res.headers['content-type']);
req.cancel(`Rejected by type filter ${res.headers['content-type']}`);
return;
}
// 応答ヘッダでサイズチェック
const contentLength = res.headers['content-length'];
if (contentLength != null) {
const size = Number(contentLength);
if (size > maxSize) {
req.cancel(`maxSize exceeded (${size} > ${maxSize}) on response`);
}
}
});
// 受信中のデータでサイズチェック
req.on('downloadProgress', (progress) => {
if (progress.transferred > maxSize && progress.percent !== 1) {
req.cancel(`maxSize exceeded (${progress.transferred} > ${maxSize}) on response`);
}
});
// 応答取得 with ステータスコードエラーの整形
const res = await req.catch(e => {
if (e instanceof Got.HTTPError) {
throw new StatusError(`${e.response.statusCode} ${e.response.statusMessage}`, e.response.statusCode, e.response.statusMessage);
}
else {
throw e;
}
});
return res;
}

View File

@ -1 +0,0 @@
export default function (val: string): boolean;

View File

@ -1,14 +0,0 @@
export default function (val) {
if (val === undefined) {
return true;
}
else if (val === null) {
return true;
}
else if (val.trim() === '') {
return true;
}
else {
return false;
}
}

View File

@ -1,7 +0,0 @@
export declare class StatusError extends Error {
name: string;
statusCode: number;
statusMessage?: string;
isPermanentError: boolean;
constructor(message: string, statusCode: number, statusMessage?: string);
}

View File

@ -1,9 +0,0 @@
export class StatusError extends Error {
constructor(message, statusCode, statusMessage) {
super(message);
this.name = 'StatusError';
this.statusCode = statusCode;
this.statusMessage = statusMessage;
this.isPermanentError = typeof this.statusCode === 'number' && this.statusCode >= 400 && this.statusCode < 500;
}
}

38
eslint.config.js Normal file
View File

@ -0,0 +1,38 @@
import pluginMisskey from '@misskey-dev/eslint-plugin';
import tsParser from '@typescript-eslint/parser';
//@ts-check
/** @type {import('eslint').Linter.Config[]} */
export default [ // eslint-disable-line import/no-default-export
...pluginMisskey.configs['recommended'],
{
ignores: [
'**/node_modules',
'src/@types/package.json.d.ts',
'built',
'vitest.config.ts',
'test',
],
},
{
files: ['**/*.ts', '**/*.tsx'],
languageOptions: {
parserOptions: {
parser: tsParser,
project: ['./tsconfig.json', './test/tsconfig.json'],
sourceType: 'module',
tsConfigRootDir: import.meta.dirname,
},
},
rules: {
// 空文字でもフォールバックしたいので無効
'@typescript-eslint/prefer-nullish-coalescing': 'off',
},
},
{
files: ['**/*.js', '**/*.cjs'],
rules: {
'@typescript-eslint/no-var-requires': 'off',
},
},
];

View File

@ -1,208 +0,0 @@
/*
* For a detailed explanation regarding each configuration property and type check, visit:
* https://jestjs.io/docs/en/configuration.html
*/
export default {
// All imported modules in your tests should be mocked automatically
// automock: false,
// Stop running tests after `n` failures
// bail: 0,
// The directory where Jest should store its cached dependency information
// cacheDirectory: "C:\\Users\\ai\\AppData\\Local\\Temp\\jest",
// Automatically clear mock calls and instances between every test
// clearMocks: false,
// Indicates whether the coverage information should be collected while executing the test
// collectCoverage: false,
// An array of glob patterns indicating a set of files for which coverage information should be collected
collectCoverageFrom: ['src/**/*.ts'],
// The directory where Jest should output its coverage files
coverageDirectory: "coverage",
// An array of regexp pattern strings used to skip coverage collection
// coveragePathIgnorePatterns: [
// "\\\\node_modules\\\\"
// ],
// Indicates which provider should be used to instrument code for coverage
coverageProvider: "v8",
// A list of reporter names that Jest uses when writing coverage reports
// coverageReporters: [
// "json",
// "text",
// "lcov",
// "clover"
// ],
// An object that configures minimum threshold enforcement for coverage results
// coverageThreshold: undefined,
// A path to a custom dependency extractor
// dependencyExtractor: undefined,
// Make calling deprecated APIs throw helpful error messages
// errorOnDeprecated: false,
// Force coverage collection from ignored files using an array of glob patterns
// forceCoverageMatch: [],
// A path to a module which exports an async function that is triggered once before all test suites
// globalSetup: undefined,
// A path to a module which exports an async function that is triggered once after all test suites
// globalTeardown: undefined,
// A set of global variables that need to be available in all test environments
globals: {
},
// The maximum amount of workers used to run your tests. Can be specified as % or a number. E.g. maxWorkers: 10% will use 10% of your CPU amount + 1 as the maximum worker number. maxWorkers: 2 will use a maximum of 2 workers.
// maxWorkers: "50%",
// An array of directory names to be searched recursively up from the requiring module's location
// moduleDirectories: [
// "node_modules"
// ],
// An array of file extensions your modules use
// moduleFileExtensions: [
// "js",
// "json",
// "jsx",
// "ts",
// "tsx",
// "node"
// ],
// A map from regular expressions to module names or to arrays of module names that allow to stub out resources with a single module
moduleNameMapper: {
// Do not resolve .wasm.js to .wasm by the rule below
'^(.+)\\.wasm\\.js$': '$1.wasm.js',
// SWC converts @/foo/bar.js to `../../src/foo/bar.js`, and then this rule
// converts it again to `../../src/foo/bar` which then can be resolved to
// `.ts` files.
// See https://github.com/swc-project/jest/issues/64#issuecomment-1029753225
// TODO: Use `--allowImportingTsExtensions` on TypeScript 5.0 so that we can
// directly import `.ts` files without this hack.
'^(\\.{1,2}/.*)\\.js$': '$1',
},
// An array of regexp pattern strings, matched against all module paths before considered 'visible' to the module loader
// modulePathIgnorePatterns: [],
// Activates notifications for test results
// notify: false,
// An enum that specifies notification mode. Requires { notify: true }
// notifyMode: "failure-change",
// A preset that is used as a base for Jest's configuration
//preset: "ts-jest/presets/js-with-ts-esm",
// Run tests from one or more projects
// projects: undefined,
// Use this configuration option to add custom reporters to Jest
// reporters: undefined,
// Automatically reset mock state between every test
// resetMocks: false,
// Reset the module registry before running each individual test
// resetModules: false,
// A path to a custom resolver
// resolver: './jest-resolver.cjs',
// Automatically restore mock state between every test
restoreMocks: true,
// The root directory that Jest should scan for tests and modules within
// rootDir: undefined,
// A list of paths to directories that Jest should use to search for files in
roots: [
"<rootDir>"
],
// Allows you to use a custom runner instead of Jest's default test runner
// runner: "jest-runner",
// The paths to modules that run some code to configure or set up the testing environment before each test
// setupFiles: [],
// A list of paths to modules that run some code to configure or set up the testing framework before each test
// setupFilesAfterEnv: [],
// The number of seconds after which a test is considered as slow and reported as such in the results.
// slowTestThreshold: 5,
// A list of paths to snapshot serializer modules Jest should use for snapshot testing
// snapshotSerializers: [],
// The test environment that will be used for testing
testEnvironment: "node",
// Options that will be passed to the testEnvironment
// testEnvironmentOptions: {},
// Adds a location field to test results
// testLocationInResults: false,
// The glob patterns Jest uses to detect test files
testMatch: [
"<rootDir>/test/index.ts",
],
// An array of regexp pattern strings that are matched against all test paths, matched tests are skipped
// testPathIgnorePatterns: [
// "\\\\node_modules\\\\"
// ],
// The regexp pattern or array of patterns that Jest uses to detect test files
// testRegex: [],
// This option allows the use of a custom results processor
// testResultsProcessor: undefined,
// This option allows use of a custom test runner
// testRunner: "jasmine2",
// This option sets the URL for the jsdom environment. It is reflected in properties such as location.href
// testURL: "http://localhost",
// Setting this value to "fake" allows the use of fake timers for functions such as "setTimeout"
// timers: "real",
// A map from regular expressions to paths to transformers
transform: {
"^.+\\.(t|j)sx?$": ["@swc/jest"],
},
// An array of regexp pattern strings that are matched against all source file paths, matched files will skip transformation
// transformIgnorePatterns: [
// "\\\\node_modules\\\\",
// "\\.pnp\\.[^\\\\]+$"
// ],
// An array of regexp pattern strings that are matched against all modules before the module loader will automatically return a mock for them
// unmockedModulePathPatterns: undefined,
// Indicates whether each individual test should be reported during the run
// verbose: undefined,
// An array of regexp patterns that are matched against all source file paths before re-running tests in watch mode
// watchPathIgnorePatterns: [],
// Whether to use watchman for file crawling
// watchman: true,
extensionsToTreatAsEsm: ['.ts'],
};

9241
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@ -1,6 +1,6 @@
{
"name": "summaly",
"version": "4.0.0",
"name": "@misskey-dev/summaly",
"version": "5.2.3",
"description": "Get web page's summary",
"author": "syuilo <syuilotan@yahoo.co.jp>",
"license": "MIT",
@ -9,37 +9,40 @@
"main": "./built/index.js",
"type": "module",
"types": "./built/index.d.ts",
"packageManager": "pnpm@9.12.3",
"files": [
"built",
"LICENSE"
],
"scripts": {
"build": "tsc",
"test": "node --experimental-vm-modules node_modules/jest/bin/jest.js --silent=false --verbose false",
"build": "tsc && tsc-alias",
"eslint": "eslint",
"test": "vitest run ./test/index.test.ts",
"serve": "fastify start ./built/index.js"
},
"devDependencies": {
"@jest/globals": "^29.4.2",
"@swc/core": "^1.3.35",
"@swc/jest": "^0.2.24",
"@types/cheerio": "0.22.18",
"@types/debug": "4.1.7",
"@types/escape-regexp": "^0.0.1",
"@types/node": "16.11.12",
"debug": "^4.3.4",
"fastify": "^4.13.0",
"fastify-cli": "^5.7.1",
"jest": "^29.4.2",
"typescript": "4.5.3"
"@misskey-dev/eslint-plugin": "^2.1.0",
"@types/debug": "4.1.12",
"@types/escape-regexp": "^0.0.3",
"@types/node": "22.16.0",
"@typescript-eslint/eslint-plugin": "^8.35.1",
"@typescript-eslint/parser": "^8.35.1",
"debug": "^4.4.1",
"eslint": "^9.30.1",
"eslint-plugin-import": "^2.32.0",
"fastify": "^5.4.0",
"fastify-cli": "^7.4.0",
"tsc-alias": "^1.8.16",
"typescript": "5.8.3",
"vitest": "^3.2.4"
},
"dependencies": {
"cheerio": "1.0.0-rc.12",
"cheerio": "1.1.0",
"escape-regexp": "0.0.1",
"got": "^12.6.0",
"html-entities": "2.3.2",
"got": "^14.4.7",
"html-entities": "2.6.0",
"iconv-lite": "0.6.3",
"jschardet": "3.0.0",
"private-ip": "2.3.3",
"trace-redirect": "1.0.6"
"jschardet": "3.1.4",
"private-ip": "3.0.2"
}
}

6677
pnpm-lock.yaml generated

File diff suppressed because it is too large Load Diff

View File

@ -1,12 +1,10 @@
import { URL } from 'node:url';
import clip from './utils/clip.js';
import cleanupTitle from './utils/cleanup-title.js';
import { decode as decodeHtml } from 'html-entities';
import { get, head, scpaping } from './utils/got.js';
import type { default as Summary, Player } from './summary.js';
import * as cheerio from 'cheerio';
import type { default as Summary, Player } from '@/summary.js';
import { clip } from '@/utils/clip.js';
import { cleanupTitle } from '@/utils/cleanup-title.js';
import { get, head, scpaping } from '@/utils/got.js';
/**
* Contains only the html snippet for a sanitized iframe as the thumbnail is
@ -20,11 +18,24 @@ async function getOEmbedPlayer($: cheerio.CheerioAPI, pageUrl: string): Promise<
return null;
}
const oEmbed = await get((new URL(href, pageUrl)).href);
const oEmbedUrl = (() => {
try {
return new URL(href, pageUrl);
} catch { return null; }
})();
if (!oEmbedUrl) {
return null;
}
const oEmbed = await get(oEmbedUrl.href).catch(() => null);
if (!oEmbed) {
return null;
}
const body = (() => {
try {
return JSON.parse(oEmbed);
} catch {}
} catch { /* empty */ }
})();
if (!body || body.version !== '1.0' || !['rich', 'video'].includes(body.type)) {
@ -38,7 +49,7 @@ async function getOEmbedPlayer($: cheerio.CheerioAPI, pageUrl: string): Promise<
}
const oEmbedHtml = cheerio.load(body.html);
const iframe = oEmbedHtml("iframe");
const iframe = oEmbedHtml('iframe');
if (iframe.length !== 1) {
// Somehow we either have multiple iframes or none
@ -61,7 +72,7 @@ async function getOEmbedPlayer($: cheerio.CheerioAPI, pageUrl: string): Promise<
// Allow only HTTPS for best security
return null;
}
} catch (e) {
} catch {
return null;
}
@ -102,6 +113,9 @@ async function getOEmbedPlayer($: cheerio.CheerioAPI, pageUrl: string): Promise<
(iframe.attr('allow') ?? '').split(/\s*;\s*/g)
.filter(s => s)
.filter(s => !ignoredList.includes(s));
if (iframe.attr('allowfullscreen') === '') {
allowedPermissions.push('fullscreen');
}
if (allowedPermissions.some(allow => !safeList.includes(allow))) {
// This iframe is probably too powerful to be embedded
return null;
@ -111,24 +125,71 @@ async function getOEmbedPlayer($: cheerio.CheerioAPI, pageUrl: string): Promise<
url,
width,
height,
allow: allowedPermissions
}
allow: allowedPermissions,
};
}
export default async (_url: URL | string, lang: string | null = null): Promise<Summary | null> => {
export type GeneralScrapingOptions = {
lang?: string | null;
userAgent?: string;
followRedirects?: boolean;
responseTimeout?: number;
operationTimeout?: number;
contentLengthLimit?: number;
contentLengthRequired?: boolean;
};
export async function general(_url: URL | string, opts?: GeneralScrapingOptions): Promise<Summary | null> {
let lang = opts?.lang;
if (lang && !lang.match(/^[\w-]+(\s*,\s*[\w-]+)*$/)) lang = null;
const url = typeof _url === 'string' ? new URL(_url) : _url;
const res = await scpaping(url.href, { lang: lang || undefined });
const res = await scpaping(url.href, {
lang: lang || undefined,
userAgent: opts?.userAgent,
followRedirects: opts?.followRedirects,
responseTimeout: opts?.responseTimeout,
operationTimeout: opts?.operationTimeout,
contentLengthLimit: opts?.contentLengthLimit,
contentLengthRequired: opts?.contentLengthRequired,
});
return await parseGeneral(url, res);
}
function headerEqualValueContains(search: string, headerValue: string | string[] | undefined) {
if (!headerValue) {
return false;
}
if (Array.isArray(headerValue)) {
return headerValue.some(value => value.toLowerCase() === search.toLowerCase());
}
return headerValue.toLowerCase() === search.toLowerCase();
}
export async function parseGeneral(_url: URL | string, res: Awaited<ReturnType<typeof scpaping>>): Promise<Summary | null> {
const url = typeof _url === 'string' ? new URL(_url) : _url;
const $ = res.$;
const twitterCard = $('meta[property="twitter:card"]').attr('content');
const twitterCard =
$('meta[name="twitter:card"]').attr('content') ||
$('meta[property="twitter:card"]').attr('content');
// According to docs, name attribute of meta tag is used for twitter card but for compatibility,
// this library will also look for property attribute.
// See https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/summary
// Property attribute is used for open graph.
// See https://ogp.me/
let title: string | null | undefined =
$('meta[property="og:title"]').attr('content') ||
$('meta[name="twitter:title"]').attr('content') ||
$('meta[property="twitter:title"]').attr('content') ||
$('title').text();
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
if (title === undefined || title === null) {
return null;
}
@ -137,6 +198,7 @@ export default async (_url: URL | string, lang: string | null = null): Promise<S
let image: string | null | undefined =
$('meta[property="og:image"]').attr('content') ||
$('meta[name="twitter:image"]').attr('content') ||
$('meta[property="twitter:image"]').attr('content') ||
$('link[rel="image_src"]').attr('href') ||
$('link[rel="apple-touch-icon"]').attr('href') ||
@ -145,26 +207,27 @@ export default async (_url: URL | string, lang: string | null = null): Promise<S
image = image ? (new URL(image, url.href)).href : null;
const playerUrl =
(twitterCard !== 'summary_large_image' && $('meta[property="twitter:player"]').attr('content')) ||
(twitterCard !== 'summary_large_image' && $('meta[name="twitter:player"]').attr('content')) ||
(twitterCard !== 'summary_large_image' && $('meta[property="twitter:player"]').attr('content')) ||
$('meta[property="og:video"]').attr('content') ||
$('meta[property="og:video:secure_url"]').attr('content') ||
$('meta[property="og:video:url"]').attr('content');
const playerWidth = parseInt(
$('meta[property="twitter:player:width"]').attr('content') ||
$('meta[name="twitter:player:width"]').attr('content') ||
$('meta[property="twitter:player:width"]').attr('content') ||
$('meta[property="og:video:width"]').attr('content') ||
'');
const playerHeight = parseInt(
$('meta[property="twitter:player:height"]').attr('content') ||
$('meta[name="twitter:player:height"]').attr('content') ||
$('meta[property="twitter:player:height"]').attr('content') ||
$('meta[property="og:video:height"]').attr('content') ||
'');
let description: string | null | undefined =
$('meta[property="og:description"]').attr('content') ||
$('meta[name="twitter:description"]').attr('content') ||
$('meta[property="twitter:description"]').attr('content') ||
$('meta[name="description"]').attr('content');
@ -176,10 +239,10 @@ export default async (_url: URL | string, lang: string | null = null): Promise<S
description = null;
}
let siteName = decodeHtml(
const siteName = decodeHtml(
$('meta[property="og:site_name"]').attr('content') ||
$('meta[name="application-name"]').attr('content') ||
url.hostname
url.host,
);
const favicon =
@ -187,26 +250,38 @@ export default async (_url: URL | string, lang: string | null = null): Promise<S
$('link[rel="icon"]').attr('href') ||
'/favicon.ico';
const sensitive = $('.tweet').attr('data-possibly-sensitive') === 'true'
const activityPub =
$('link[rel="alternate"][type="application/activity+json"]').attr('href') || null;
const fediverseCreator: string | null =
$('meta[name=\'fediverse:creator\']').attr('content') || null;
// https://developer.mixi.co.jp/connect/mixi_plugin/mixi_check/spec_mixi_check/#toc-18-
const sensitive =
$('meta[property=\'mixi:content-rating\']').attr('content') === '1' ||
headerEqualValueContains('adult', res.response.headers.rating) ||
headerEqualValueContains('RTA-5042-1996-1400-1577-RTA', res.response.headers.rating) ||
$('meta[name=\'rating\']').attr('content') === 'adult' ||
$('meta[name=\'rating\']').attr('content')?.toUpperCase() === 'RTA-5042-1996-1400-1577-RTA';
const find = async (path: string) => {
const target = new URL(path, url.href);
try {
await head(target.href);
return target;
} catch (e) {
} catch {
return null;
}
};
const getIcon = async () => {
return (await find(favicon)) || null;
}
};
const [icon, oEmbed] = await Promise.all([
getIcon(),
getOEmbedPlayer($, url.href),
])
]);
// Clean up the title
title = cleanupTitle(title, siteName);
@ -228,5 +303,7 @@ export default async (_url: URL | string, lang: string | null = null): Promise<S
},
sitename: siteName || null,
sensitive,
activityPub,
fediverseCreator,
};
};
}

View File

@ -1,20 +1,19 @@
/**
* summaly
* https://github.com/syuilo/summaly
* https://github.com/misskey-dev/summaly
*/
import { URL } from 'node:url';
import tracer from 'trace-redirect';
import Summary from './summary.js';
import type { IPlugin as _IPlugin } from './iplugin.js';
export type IPlugin = _IPlugin;
import general from './general.js';
import * as Got from 'got';
import { setAgent } from './utils/got.js';
import { got, type Agents as GotAgents } from 'got';
import type { FastifyInstance } from 'fastify';
import { plugins as builtinPlugins } from './plugins/index.js';
import { SummalyResult } from '@/summary.js';
import { SummalyPlugin as _SummalyPlugin } from '@/iplugin.js';
import { general, type GeneralScrapingOptions } from '@/general.js';
import { DEFAULT_BOT_UA, DEFAULT_OPERATION_TIMEOUT, DEFAULT_RESPONSE_TIMEOUT, agent, setAgent } from '@/utils/got.js';
import { plugins as builtinPlugins } from '@/plugins/index.js';
type Options = {
export type SummalyPlugin = _SummalyPlugin;
export type SummalyOptions = {
/**
* Accept-Language for the request
*/
@ -28,34 +27,56 @@ type Options = {
/**
* Custom Plugins
*/
plugins?: IPlugin[];
plugins?: SummalyPlugin[];
/**
* Custom HTTP agent
*/
agent?: Got.Agents;
};
agent?: GotAgents;
type Result = Summary & {
/**
* The actual url of that web page
* User-Agent for the request
*/
url: string;
userAgent?: string;
/**
* Response timeout.
* Set timeouts for each phase, such as host name resolution and socket communication.
*/
responseTimeout?: number;
/**
* Operation timeout.
* Set the timeout from the start to the end of the request.
*/
operationTimeout?: number;
/**
* Maximum content length.
* If set to true, an error will occur if the content-length value returned from the other server is larger than this parameter (or if the received body size exceeds this parameter).
*/
contentLengthLimit?: number;
/**
* Content length required.
* If set to true, it will be an error if the other server does not return content-length.
*/
contentLengthRequired?: boolean;
};
const defaultOptions = {
export const summalyDefaultOptions = {
lang: null,
followRedirects: true,
plugins: [],
} as Options;
} as SummalyOptions;
/**
* Summarize an web page
*/
export const summaly = async (url: string, options?: Options): Promise<Result> => {
export const summaly = async (url: string, options?: SummalyOptions): Promise<SummalyResult> => {
if (options?.agent) setAgent(options.agent);
const opts = Object.assign(defaultOptions, options);
const opts = Object.assign(summalyDefaultOptions, options);
const plugins = builtinPlugins.concat(opts.plugins || []);
@ -63,8 +84,32 @@ export const summaly = async (url: string, options?: Options): Promise<Result> =
if (opts.followRedirects) {
// .catch(() => url)にすればいいけど、jestにtrace-redirectを食わせるのが面倒なのでtry-catch
try {
actualUrl = await tracer(url);
} catch (e) {
const timeout = opts.responseTimeout ?? DEFAULT_RESPONSE_TIMEOUT;
const operationTimeout = opts.operationTimeout ?? DEFAULT_OPERATION_TIMEOUT;
actualUrl = await got
.head(url, {
headers: {
accept: 'text/html,application/xhtml+xml',
'user-agent': opts.userAgent ?? DEFAULT_BOT_UA,
'accept-language': opts.lang ?? undefined,
},
timeout: {
lookup: timeout,
connect: timeout,
secureConnect: timeout,
socket: timeout, // read timeout
response: timeout,
send: timeout,
request: operationTimeout, // whole operation timeout
},
agent,
http2: false,
retry: {
limit: 0,
},
})
.then(res => res.url);
} catch {
actualUrl = url;
}
}
@ -75,18 +120,30 @@ export const summaly = async (url: string, options?: Options): Promise<Result> =
const match = plugins.filter(plugin => plugin.test(_url))[0];
// Get summary
const summary = await (match ? match.summarize : general)(_url, opts.lang || undefined);
const scrapingOptions: GeneralScrapingOptions = {
lang: opts.lang,
userAgent: opts.userAgent,
responseTimeout: opts.responseTimeout,
followRedirects: opts.followRedirects,
operationTimeout: opts.operationTimeout,
contentLengthLimit: opts.contentLengthLimit,
contentLengthRequired: opts.contentLengthRequired,
};
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
const summary = await (match ? match.summarize : general)(_url, scrapingOptions);
if (summary == null) {
throw 'failed summarize';
throw new Error('failed summarize');
}
return Object.assign(summary, {
url: actualUrl
url: actualUrl,
});
};
export default function (fastify: FastifyInstance, options: Options, done: (err?: Error) => void) {
// eslint-disable-next-line import/no-default-export
export default function (fastify: FastifyInstance, options: SummalyOptions, done: (err?: Error) => void) {
fastify.get<{
Querystring: {
url?: string;
@ -94,9 +151,10 @@ export default function (fastify: FastifyInstance, options: Options, done: (err?
};
}>('/', async (req, reply) => {
const url = req.query.url as string;
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
if (url == null) {
return reply.status(400).send({
error: 'url is required'
error: 'url is required',
});
}
@ -110,7 +168,7 @@ export default function (fastify: FastifyInstance, options: Options, done: (err?
return summary;
} catch (e) {
return reply.status(500).send({
error: e
error: e,
});
}
});

View File

@ -1,7 +1,7 @@
import type { URL } from 'node:url';
import Summary from './summary.js';
import type Summary from '@/summary.js';
import type { GeneralScrapingOptions } from '@/general.js';
export interface IPlugin {
export interface SummalyPlugin {
test: (url: URL) => boolean;
summarize: (url: URL, lang?: string) => Promise<Summary>;
summarize: (url: URL, opts?: GeneralScrapingOptions) => Promise<Summary | null>;
}

View File

@ -1,6 +1,5 @@
import { URL } from 'node:url';
import { scpaping } from '../utils/got.js';
import summary from '../summary.js';
import { scpaping } from '@/utils/got.js';
import summary from '@/summary.js';
export function test(url: URL): boolean {
return url.hostname === 'www.amazon.com' ||
@ -55,5 +54,7 @@ export async function summarize(url: URL): Promise<summary> {
allow: playerUrl ? ['fullscreen', 'encrypted-media'] : [],
},
sitename: 'Amazon',
activityPub: null,
fediverseCreator: null,
};
}

26
src/plugins/bluesky.ts Normal file
View File

@ -0,0 +1,26 @@
import * as cheerio from 'cheerio';
import type Summary from '@/summary.js';
import { getResponse, getGotOptions } from '@/utils/got.js';
import { parseGeneral, type GeneralScrapingOptions } from '@/general.js';
export function test(url: URL): boolean {
return url.hostname === 'bsky.app';
}
export async function summarize(url: URL, opts?: GeneralScrapingOptions): Promise<Summary | null> {
const args = getGotOptions(url.href, opts);
// HEADで取ると404が返るためGETのみで取得
const res = await getResponse({
...args,
method: 'GET',
});
const body = res.body;
const $ = cheerio.load(body);
return await parseGeneral(url, {
body,
$,
response: res,
});
}

View File

@ -0,0 +1,16 @@
import { general, type GeneralScrapingOptions } from '@/general.js';
import Summary from '@/summary.js';
export function test(url: URL): boolean {
// Branch.io を使用したディープリンクにマッチ
return /^[a-zA-Z0-9]+\.app\.link$/.test(url.hostname) ||
url.hostname === 'spotify.link';
}
export async function summarize(url: URL, opts?: GeneralScrapingOptions): Promise<Summary | null> {
// https://help.branch.io/using-branch/docs/creating-a-deep-link#redirections
// Web版に強制リダイレクトすることでbranch.ioの独自ページが開くのを防ぐ
url.searchParams.append('$web_only', 'true');
return await general(url, opts);
}

View File

@ -1,8 +1,12 @@
import { IPlugin } from '@/iplugin.js';
import * as amazon from './amazon.js';
import * as bluesky from './bluesky.js';
import * as wikipedia from './wikipedia.js';
import * as branchIoDeeplinks from './branchio-deeplinks.js';
import { SummalyPlugin } from '@/iplugin.js';
export const plugins: IPlugin[] = [
export const plugins: SummalyPlugin[] = [
amazon,
bluesky,
wikipedia,
branchIoDeeplinks,
];

View File

@ -1,8 +1,7 @@
import { URL } from 'node:url';
import { get } from '../utils/got.js';
import debug from 'debug';
import summary from '../summary.js';
import clip from './../utils/clip.js';
import { get } from '@/utils/got.js';
import summary from '@/summary.js';
import { clip } from '@/utils/clip.js';
const log = debug('summaly:plugins:wikipedia');
@ -20,12 +19,13 @@ export async function summarize(url: URL): Promise<summary> {
log(`title is ${title}`);
log(`endpoint is ${endpoint}`);
// eslint-disable-next-line @typescript-eslint/no-explicit-any
let body = await get(endpoint) as any;
body = JSON.parse(body);
log(body);
if (!('query' in body) || !('pages' in body.query)) {
throw 'fetch failed';
throw new Error('fetch failed');
}
const info = body.query.pages[Object.keys(body.query.pages)[0]];
@ -42,5 +42,7 @@ export async function summarize(url: URL): Promise<summary> {
allow: [],
},
sitename: 'Wikipedia',
activityPub: null,
fediverseCreator: null,
};
}

View File

@ -1,8 +1,8 @@
type Summary = {
/**
* The description of that web page
* The title of that web page
*/
description: string | null;
title: string | null;
/**
* The url of the icon of that web page
@ -10,31 +10,49 @@ type Summary = {
icon: string | null;
/**
* The name of site of that web page
* The description of that web page
*/
sitename: string | null;
description: string | null;
/**
* The url of the thumbnail of that web page
*/
thumbnail: string | null;
/**
* The name of site of that web page
*/
sitename: string | null;
/**
* The player of that web page
*/
player: Player;
/**
* The title of that web page
*/
title: string | null;
/**
* Possibly sensitive
*/
sensitive?: boolean;
/**
* The url of the ActivityPub representation of that web page
*/
activityPub: string | null;
/**
* The @ handle of a fediverse user (https://blog.joinmastodon.org/2024/07/highlighting-journalism-on-mastodon/)
*/
fediverseCreator: string | null;
};
export type SummalyResult = Summary & {
/**
* The actual url of that web page
*/
url: string;
};
// eslint-disable-next-line import/no-default-export
export default Summary;
export type Player = {

View File

@ -1,6 +1,7 @@
/* eslint-disable no-param-reassign */
import escapeRegExp from 'escape-regexp';
export default function(title: string, siteName?: string | null): string {
export function cleanupTitle(title: string, siteName?: string | null): string {
title = title.trim();
if (siteName) {
@ -9,12 +10,12 @@ export default function(title: string, siteName?: string | null): string {
const x = escapeRegExp(siteName);
const patterns = [
`^(.+?)\\s?[\\-\\|:・]\\s?${x}$`
`^(.+?)\\s?[\\-\\|:・]\\s?${x}$`,
];
for (let i = 0; i < patterns.length; i++) {
const pattern = new RegExp(patterns[i]);
const [, match] = pattern.exec(title) || [null, null];
const [, match] = pattern.exec(title) ?? [null, null];
if (match) {
return match;
}

View File

@ -1,10 +1,11 @@
import nullOrEmpty from './null-or-empty.js';
import { nullOrEmpty } from './null-or-empty.js';
export default function(s: string, max: number): string {
export function clip(s: string, max: number): string {
if (nullOrEmpty(s)) {
return s;
}
// eslint-disable-next-line no-param-reassign
s = s.trim();
if (s.length > max) {

View File

@ -11,6 +11,7 @@ const regCharset = new RegExp(/charset\s*=\s*["']?([\w-]+)/, 'i');
export function detectEncoding(body: Buffer): string {
// By detection
const detected = jschardet.detect(body, { minimumThreshold: 0.99 });
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
if (detected) {
const candicate = detected.encoding;
const encoding = toEncoding(candicate);

View File

@ -1,17 +1,20 @@
import got, * as Got from 'got';
import { StatusError } from './status-error.js';
import { detectEncoding, toUtf8 } from './encoding.js';
import * as cheerio from 'cheerio';
import PrivateIp from 'private-ip';
import { dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { readFileSync } from 'node:fs';
import got, * as Got from 'got';
import * as cheerio from 'cheerio';
import PrivateIp from 'private-ip';
import type { GeneralScrapingOptions } from '@/general.js';
import { StatusError } from '@/utils/status-error.js';
import { detectEncoding, toUtf8 } from '@/utils/encoding.js';
const _filename = fileURLToPath(import.meta.url);
const _dirname = dirname(_filename);
export let agent: Got.Agents = {};
export function setAgent(_agent: Got.Agents) {
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
agent = _agent || {};
}
@ -21,34 +24,48 @@ export type GotOptions = {
body?: string;
headers: Record<string, string | undefined>;
typeFilter?: RegExp;
}
followRedirects?: boolean;
responseTimeout?: number;
operationTimeout?: number;
contentLengthLimit?: number;
contentLengthRequired?: boolean;
};
const repo = JSON.parse(readFileSync(`${_dirname}/../../package.json`, 'utf8'));
const RESPONSE_TIMEOUT = 20 * 1000;
const OPERATION_TIMEOUT = 60 * 1000;
const MAX_RESPONSE_SIZE = 10 * 1024 * 1024;
const BOT_UA = `SummalyBot/${repo.version}`;
export const DEFAULT_RESPONSE_TIMEOUT = 20 * 1000;
export const DEFAULT_OPERATION_TIMEOUT = 60 * 1000;
export const DEFAULT_MAX_RESPONSE_SIZE = 10 * 1024 * 1024;
export const DEFAULT_BOT_UA = `SummalyBot/${repo.version}`;
export async function scpaping(url: string, opts?: { lang?: string; }) {
const response = await getResponse({
export function getGotOptions(url: string, opts?: GeneralScrapingOptions): Omit<GotOptions, 'method'> {
return {
url,
method: 'GET',
headers: {
'accept': 'text/html,application/xhtml+xml',
'user-agent': BOT_UA,
'accept-language': opts?.lang
'user-agent': opts?.userAgent ?? DEFAULT_BOT_UA,
'accept-language': opts?.lang ?? undefined,
},
typeFilter: /^(text\/html|application\/xhtml\+xml)/,
followRedirects: opts?.followRedirects,
responseTimeout: opts?.responseTimeout,
operationTimeout: opts?.operationTimeout,
contentLengthLimit: opts?.contentLengthLimit,
contentLengthRequired: opts?.contentLengthRequired,
};
}
export async function scpaping(
url: string,
opts?: GeneralScrapingOptions,
) {
const args = getGotOptions(url, opts);
const response = await getResponse({
...args,
method: 'GET',
});
// SUMMALY_ALLOW_PRIVATE_IPはテスト用
const allowPrivateIp = process.env.SUMMALY_ALLOW_PRIVATE_IP === 'true' || Object.keys(agent).length > 0;
if (!allowPrivateIp && response.ip && PrivateIp(response.ip)) {
throw new StatusError(`Private IP rejected ${response.ip}`, 400, 'Private IP Rejected');
}
const encoding = detectEncoding(response.rawBody);
const body = toUtf8(response.rawBody, encoding);
const $ = cheerio.load(body);
@ -69,24 +86,22 @@ export async function get(url: string) {
},
});
return await res.body;
return res.body;
}
export async function head(url: string) {
const res = await getResponse({
return await getResponse({
url,
method: 'HEAD',
headers: {
'accept': '*/*',
},
});
return await res;
}
async function getResponse(args: GotOptions) {
const timeout = RESPONSE_TIMEOUT;
const operationTimeout = OPERATION_TIMEOUT;
export async function getResponse(args: GotOptions) {
const timeout = args.responseTimeout ?? DEFAULT_RESPONSE_TIMEOUT;
const operationTimeout = args.operationTimeout ?? DEFAULT_OPERATION_TIMEOUT;
const req = got<string>(args.url, {
method: args.method,
@ -101,6 +116,7 @@ async function getResponse(args: GotOptions) {
send: timeout,
request: operationTimeout, // whole operation timeout
},
followRedirect: args.followRedirects,
agent,
http2: false,
retry: {
@ -108,30 +124,45 @@ async function getResponse(args: GotOptions) {
},
});
return await receiveResponse({ req, typeFilter: args.typeFilter });
}
const res = await receiveResponse({ req, opts: args });
async function receiveResponse<T>(args: { req: Got.CancelableRequest<Got.Response<T>>, typeFilter?: RegExp }) {
const req = args.req;
const maxSize = MAX_RESPONSE_SIZE;
// SUMMALY_ALLOW_PRIVATE_IPはテスト用
// TODO: Try moving this to receiveResponse- ATM `got` doesn't provide a means
// to check the IP/response header data while streaming the response...
const allowPrivateIp = process.env.SUMMALY_ALLOW_PRIVATE_IP === 'true' || Object.keys(agent).length > 0;
if (!allowPrivateIp && res.ip && PrivateIp(res.ip)) {
throw new StatusError(`Private IP rejected ${res.ip}`, 400, 'Private IP Rejected');
}
req.on('response', (res: Got.Response) => {
// Check html
if (args.typeFilter && !res.headers['content-type']?.match(args.typeFilter)) {
// console.warn(res.headers['content-type']);
req.cancel(`Rejected by type filter ${res.headers['content-type']}`);
return;
const contentType = res.headers['content-type'];
if (args.typeFilter && !contentType?.match(args.typeFilter)) {
throw new Error(`Rejected by type filter ${contentType}`);
}
// 応答ヘッダでサイズチェック
const contentLength = res.headers['content-length'];
if (contentLength != null) {
if (contentLength) {
const maxSize = args.contentLengthLimit ?? DEFAULT_MAX_RESPONSE_SIZE;
const size = Number(contentLength);
if (size > maxSize) {
req.cancel(`maxSize exceeded (${size} > ${maxSize}) on response`);
throw new Error(`maxSize exceeded (${size} > ${maxSize}) on response`);
}
} else {
if (args.contentLengthRequired) {
throw new Error('content-length required');
}
}
});
return res;
}
async function receiveResponse<T>(args: {
req: Got.CancelableRequest<Got.Response<T>>,
opts: GotOptions,
}) {
const req = args.req;
const maxSize = args.opts.contentLengthLimit ?? DEFAULT_MAX_RESPONSE_SIZE;
// 受信中のデータでサイズチェック
req.on('downloadProgress', (progress: Got.Progress) => {

View File

@ -1,4 +1,5 @@
export default function(val: string): boolean {
/* eslint-disable @typescript-eslint/no-unnecessary-condition */
export function nullOrEmpty(val: string): boolean {
if (val === undefined) {
return true;
} else if (val === null) {

View File

@ -0,0 +1,3 @@
<!DOCTYPE html>
<meta charset="utf-8">
<link rel="alternate" type="application/activity+json" href="https://misskey.test/notes/abcdefg">

View File

@ -7,6 +7,6 @@
</head>
<body>
<h1>KISS principle</h1>
<p>KISS is an acronym for "Keep it simple, stupid" as a design principle noted by the U.S. Navy in 1960.</p>
<p>KISS is an acronym for Keep it simple, stupid as a design principle noted by the U.S. Navy in 1960.</p>
</body>
</html>

View File

@ -1,9 +1,13 @@
<!doctype html>
<html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta property="og:site_name" content="Alice's Site">
<title>Strawberry Pasta | Alice's Site</title>
</head>
<body>
<h1>Strawberry Pasta</h1>
<p>Strawberry pasta is a kind of pasta with strawberry sauce.</p>
</body>
</html>

View File

@ -0,0 +1,13 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="fediverse:creator" content="@test@example.com">
<title>Meow</title>
</head>
<body>
<h1>Hellooo!</h1>
<p>:3</p>
</body>
</html>

View File

@ -0,0 +1,13 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="rating" content="adult">
<title>SENSITIVE CONTENT!!</title>
</head>
<body>
<h1>Yo</h1>
<p>Hey hey hey syuilo.</p>
</body>
</html>

View File

@ -0,0 +1,13 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="rating" content="RTA-5042-1996-1400-1577-RTA">
<title>SENSITIVE CONTENT!!</title>
</head>
<body>
<h1>Yo</h1>
<p>Hey hey hey syuilo.</p>
</body>
</html>

View File

@ -0,0 +1,13 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta property="mixi:content-rating" content="1">
<title>SENSITIVE CONTENT!!</title>
</head>
<body>
<h1>Yo</h1>
<p>Hey hey hey syuilo.</p>
</body>
</html>

View File

@ -3,9 +3,10 @@
<html lang="en">
<head>
<meta charset="utf-8">
<title>KISS principle</title>
</head>
<body>
<h1>KISS principle</h1>
<p>KISS is an acronym for "Keep it simple, stupid" as a design principle noted by the U.S. Navy in 1960.</p>
<p>KISS is an acronym for Keep it simple, stupid as a design principle noted by the U.S. Navy in 1960.</p>
</body>
</html>

View File

@ -1,2 +1,3 @@
<!DOCTYPE html>
<link type="application/json+oembed" href="http://localhost:3060/oembe.json" />
<meta property="og:description" content="nonexistent">

View File

@ -0,0 +1,3 @@
<!DOCTYPE html>
<link type="application/json+oembed" href="http://localhost:+3060/oembed.json" />
<meta property="og:description" content="wrong url">

View File

@ -1,2 +0,0 @@
<!DOCTYPE html>
<link type="application/json+oembed" href="http://localhost+:3060/oembed.json" />

733
test/index.test.ts Normal file
View File

@ -0,0 +1,733 @@
/**
* Tests!
*/
'use strict';
/* dependencies below */
import fs, { readdirSync } from 'node:fs';
import process from 'node:process';
import { dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { Agent as httpAgent } from 'node:http';
import { Agent as httpsAgent } from 'node:https';
import { expect, test, describe, beforeEach, afterEach } from 'vitest';
import fastify, { type FastifyInstance } from 'fastify';
import { summaly } from '@/index.js';
import { StatusError } from '@/utils/status-error.js';
const _filename = fileURLToPath(import.meta.url);
const _dirname = dirname(_filename);
/* settings below */
Error.stackTraceLimit = Infinity;
// During the test the env variable is set to test
process.env.NODE_ENV = 'test';
process.env.SUMMALY_ALLOW_PRIVATE_IP = 'true';
const port = 3060;
const host = `http://localhost:${port}`;
// Display detail of unhandled promise rejection
process.on('unhandledRejection', console.dir);
let app: FastifyInstance | null = null;
function skippableTest(name: string, fn: () => void) {
if (process.env.SKIP_NETWORK_TEST === 'true') {
console.log(`[SKIP] ${name}`);
test.skip(name, fn);
} else {
test(name, fn);
}
}
beforeEach(() => {
// Allow private IPs by default, since a lot of the tests rely on old behvior
process.env.SUMMALY_ALLOW_PRIVATE_IP = 'true';
});
afterEach(async () => {
if (app) {
await app.close();
app = null;
}
});
/* tests below */
test('basic', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
expect(await summaly(host)).toEqual({
title: 'KISS principle',
icon: null,
description: null,
thumbnail: null,
player: {
url: null,
width: null,
height: null,
'allow': [
'autoplay',
'encrypted-media',
'fullscreen',
],
},
sitename: 'localhost:3060',
sensitive: false,
url: host + '/',
activityPub: null,
fediverseCreator: null,
});
});
skippableTest('Stage Bye Stage', async () => {
// If this test fails, you must rewrite the result data and the example in README.md.
const summary = await summaly('https://www.youtube.com/watch?v=NMIEAhH_fTU');
expect(summary).toEqual(
{
'title': '【アイドルマスター】「Stage Bye Stage」(歌:島村卯月、渋谷凛、本田未央)',
'icon': 'https://www.youtube.com/s/desktop/78bc1359/img/logos/favicon.ico',
'description': 'Website▶https://columbia.jp/idolmaster/Playlist▶https://www.youtube.com/playlist?list=PL83A2998CF3BBC86D2018年7月18日発売予定THE IDOLM@STER CINDERELLA GIRLS CG STAR...',
'thumbnail': 'https://i.ytimg.com/vi/NMIEAhH_fTU/maxresdefault.jpg',
'player': {
'url': 'https://www.youtube.com/embed/NMIEAhH_fTU?feature=oembed',
'width': 200,
'height': 113,
'allow': [
'autoplay',
'clipboard-write',
'encrypted-media',
'picture-in-picture',
'web-share',
'fullscreen',
],
},
'sitename': 'YouTube',
'sensitive': false,
'activityPub': null,
'fediverseCreator': null,
'url': 'https://www.youtube.com/watch?v=NMIEAhH_fTU',
},
);
});
test('faviconがHTML上で指定されていないが、ルートに存在する場合、正しく設定される', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/no-favicon.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
app.get('/favicon.ico', (_, reply) => reply.status(200).send());
await app.listen({ port });
const summary = await summaly(host);
expect(summary.icon).toBe(`${host}/favicon.ico`);
});
test('faviconがHTML上で指定されていなくて、ルートにも存在しなかった場合 null になる', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/no-favicon.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
app.get('*', (_, reply) => reply.status(404).send());
await app.listen({ port });
const summary = await summaly(host);
expect(summary.icon).toBe(null);
});
test('titleがcleanupされる', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/og-title.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.title).toBe('Strawberry Pasta');
});
describe('Private IP blocking', () => {
beforeEach(() => {
process.env.SUMMALY_ALLOW_PRIVATE_IP = 'false';
app = fastify();
app.get('*', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/og-title.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
return app.listen({ port });
});
test('private ipなサーバーの情報を取得できない', async () => {
const summary = await summaly(host).catch((e: StatusError) => e);
if (summary instanceof StatusError) {
expect(summary.name).toBe('StatusError');
} else {
expect(summary).toBeInstanceOf(StatusError);
}
});
test('agentが指定されている場合はprivate ipを許可', async () => {
const summary = await summaly(host, {
agent: {
http: new httpAgent({ keepAlive: true }),
https: new httpsAgent({ keepAlive: true }),
},
});
expect(summary.title).toBe('Strawberry Pasta');
});
test('agentが空のオブジェクトの場合はprivate ipを許可しない', async () => {
const summary = await summaly(host, { agent: {} }).catch((e: StatusError) => e);
if (summary instanceof StatusError) {
expect(summary.name).toBe('StatusError');
} else {
expect(summary).toBeInstanceOf(StatusError);
}
});
afterEach(() => {
process.env.SUMMALY_ALLOW_PRIVATE_IP = 'true';
});
});
describe('OGP', () => {
test('title', async () => {
app = fastify();
app.get('*', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/og-title.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.title).toBe('Strawberry Pasta');
});
test('description', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/og-description.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.description).toBe('Strawberry Pasta');
});
test('site_name', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/og-site_name.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.sitename).toBe('Strawberry Pasta');
});
test('thumbnail', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/og-image.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.thumbnail).toBe('https://himasaku.net/himasaku.png');
});
});
describe('TwitterCard', () => {
test('title', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/twitter-title.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.title).toBe('Strawberry Pasta');
});
test('description', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/twitter-description.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.description).toBe('Strawberry Pasta');
});
test('thumbnail', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/twitter-image.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.thumbnail).toBe('https://himasaku.net/himasaku.png');
});
test('Player detection - PeerTube:video => video', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/player-peertube-video.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/embedurl');
expect(summary.player.allow).toStrictEqual(['autoplay', 'encrypted-media', 'fullscreen']);
});
test('Player detection - Pleroma:video => video', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/player-pleroma-video.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/embedurl');
expect(summary.player.allow).toStrictEqual(['autoplay', 'encrypted-media', 'fullscreen']);
});
test('Player detection - Pleroma:image => image', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/player-pleroma-image.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.thumbnail).toBe('https://example.com/imageurl');
});
});
describe('oEmbed', () => {
const setUpFastify = async (oEmbedPath: string, htmlPath = 'htmls/oembed.html') => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(new URL(htmlPath, import.meta.url));
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
app.get('/oembed.json', (request, reply) => {
const content = fs.readFileSync(new URL(oEmbedPath, new URL('oembed/', import.meta.url)));
reply.header('content-length', content.length);
reply.header('content-type', 'application/json');
return reply.send(content);
});
await app.listen({ port });
};
for (const filename of readdirSync(new URL('oembed/invalid', import.meta.url))) {
test(`Invalidity test: ${filename}`, async () => {
await setUpFastify(`invalid/${filename}`);
const summary = await summaly(host);
expect(summary.player.url).toBe(null);
});
}
test('basic properties', async () => {
await setUpFastify('oembed.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.width).toBe(500);
expect(summary.player.height).toBe(300);
});
test('type: video', async () => {
await setUpFastify('oembed-video.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.width).toBe(500);
expect(summary.player.height).toBe(300);
});
test('max height', async () => {
await setUpFastify('oembed-too-tall.json');
const summary = await summaly(host);
expect(summary.player.height).toBe(1024);
});
test('children are ignored', async () => {
await setUpFastify('oembed-iframe-child.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
});
test('allows fullscreen', async () => {
await setUpFastify('oembed-allow-fullscreen.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.allow).toStrictEqual(['fullscreen']);
});
test('allows legacy allowfullscreen', async () => {
await setUpFastify('oembed-allow-fullscreen-legacy.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.allow).toStrictEqual(['fullscreen']);
});
test('allows safelisted permissions', async () => {
await setUpFastify('oembed-allow-safelisted-permissions.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.allow).toStrictEqual([
'autoplay', 'clipboard-write', 'fullscreen',
'encrypted-media', 'picture-in-picture', 'web-share',
]);
});
test('ignores rare permissions', async () => {
await setUpFastify('oembed-ignore-rare-permissions.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.allow).toStrictEqual(['autoplay']);
});
test('oEmbed with relative path', async () => {
await setUpFastify('oembed.json', 'htmls/oembed-relative.html');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
});
test('oEmbed with nonexistent path', async () => {
await setUpFastify('oembed.json', 'htmls/oembed-nonexistent-path.html');
const summary = await summaly(host);
expect(summary.player.url).toBe(null);
expect(summary.description).toBe('nonexistent');
});
test('oEmbed with wrong path', async () => {
await setUpFastify('oembed.json', 'htmls/oembed-wrong-path.html');
const summary = await summaly(host);
expect(summary.player.url).toBe(null);
expect(summary.description).toBe('wrong url');
});
test('oEmbed with OpenGraph', async () => {
await setUpFastify('oembed.json', 'htmls/oembed-and-og.html');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.description).toBe('blobcats rule the world');
});
test('Invalid oEmbed with valid OpenGraph', async () => {
await setUpFastify('invalid/oembed-insecure.json', 'htmls/oembed-and-og.html');
const summary = await summaly(host);
expect(summary.player.url).toBe(null);
expect(summary.description).toBe('blobcats rule the world');
});
test('oEmbed with og:video', async () => {
await setUpFastify('oembed.json', 'htmls/oembed-and-og-video.html');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.allow).toStrictEqual([]);
});
test('width: 100%', async () => {
await setUpFastify('oembed-percentage-width.json');
const summary = await summaly(host);
expect(summary.player.width).toBe(null);
expect(summary.player.height).toBe(300);
});
});
describe('ActivityPub', () => {
test('Basic', async () => {
app = fastify();
app.get('*', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/activitypub.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.activityPub).toBe('https://misskey.test/notes/abcdefg');
});
test('Null', async () => {
app = fastify();
app.get('*', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.activityPub).toBe(null);
});
});
describe('Fediverse Creator', () => {
test('Basic', async () => {
app = fastify();
app.get('*', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/fediverse-creator.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.fediverseCreator).toBe('@test@example.com');
});
test('Null', async () => {
app = fastify();
app.get('*', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.fediverseCreator).toBeNull();
});
});
describe('sensitive', () => {
test('default', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
expect((await summaly(host)).sensitive).toBe(false);
});
test('mixi:content-rating 1', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/mixi-sensitive.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
expect((await summaly(host)).sensitive).toBe(true);
});
test('meta rating adult', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/meta-adult-sensitive.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
expect((await summaly(host)).sensitive).toBe(true);
});
test('meta rating rta', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/meta-rta-sensitive.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
expect((await summaly(host)).sensitive).toBe(true);
});
test('HTTP Header rating adult', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
reply.header('rating', 'adult');
return reply.send(content);
});
await app.listen({ port });
expect((await summaly(host)).sensitive).toBe(true);
});
test('HTTP Header rating rta', async () => {
app = fastify();
app.get('/', (request, reply) => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
reply.header('content-length', content.length);
reply.header('content-type', 'text/html');
reply.header('rating', 'RTA-5042-1996-1400-1577-RTA');
return reply.send(content);
});
await app.listen({ port });
expect((await summaly(host)).sensitive).toBe(true);
});
});
describe('UserAgent', () => {
test('UA設定が反映されていること', async () => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
let ua: string | undefined = undefined;
app = fastify();
app.get('/', (request, reply) => {
reply.header('content-length', content.byteLength);
reply.header('content-type', 'text/html');
ua = request.headers['user-agent'];
return reply.send(content);
});
await app.listen({ port });
await summaly(host, { userAgent: 'test-ua' });
expect(ua).toBe('test-ua');
});
});
describe('content-length limit', () => {
test('content-lengthの上限以内であればエラーが起こらないこと', async () => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
app = fastify();
app.get('/', (request, reply) => {
reply.header('content-length', content.byteLength);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
expect(await summaly(host, { contentLengthLimit: content.byteLength })).toBeDefined();
});
test('content-lengthの上限を超えているとエラーになる事', async () => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
app = fastify();
app.get('/', (request, reply) => {
reply.header('content-length', content.byteLength);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
await expect(summaly(host, { contentLengthLimit: content.byteLength - 1 })).rejects.toThrow();
});
});
describe('content-length required', () => {
test('[オプション有効化時] content-lengthが返された場合はエラーとならないこと', async () => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
app = fastify();
app.get('/', (request, reply) => {
reply.header('content-length', content.byteLength);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
expect(await summaly(host, { contentLengthRequired: true, contentLengthLimit: content.byteLength })).toBeDefined();
});
test('[オプション有効化時] content-lengthが返されない場合はエラーとなること', async () => {
app = fastify();
app.get('/', (request, reply) => {
reply.header('content-type', 'text/html');
// streamで渡さないとcontent-lengthを自動で設定されてしまう
return reply.send(fs.createReadStream(_dirname + '/htmls/basic.html'));
});
await app.listen({ port });
await expect(summaly(host, { contentLengthRequired: true })).rejects.toThrow();
});
test('[オプション無効化時] content-lengthが返された場合はエラーとならないこと', async () => {
const content = fs.readFileSync(_dirname + '/htmls/basic.html');
app = fastify();
app.get('/', (request, reply) => {
reply.header('content-length', content.byteLength);
reply.header('content-type', 'text/html');
return reply.send(content);
});
await app.listen({ port });
expect(await summaly(host, { contentLengthRequired: false, contentLengthLimit: content.byteLength })).toBeDefined();
});
test('[オプション無効化時] content-lengthが返されなくてもエラーとならないこと', async () => {
app = fastify();
app.get('/', (request, reply) => {
reply.header('content-type', 'text/html');
// streamで渡さないとcontent-lengthを自動で設定されてしまう
return reply.send(fs.createReadStream(_dirname + '/htmls/basic.html'));
});
await app.listen({ port });
expect(await summaly(host, { contentLengthRequired: false })).toBeDefined();
});
});

View File

@ -1,360 +0,0 @@
/**
* Tests!
*/
'use strict';
/* dependencies below */
import fs, { readdirSync } from 'node:fs';
import process from 'node:process';
import fastify from 'fastify';
import { summaly } from '../src/index.js';
import { dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { expect, jest, test, describe, beforeEach, afterEach } from '@jest/globals';
import { Agent as httpAgent } from 'node:http';
import { Agent as httpsAgent } from 'node:https';
import { StatusError } from '../src/utils/status-error.js';
const _filename = fileURLToPath(import.meta.url);
const _dirname = dirname(_filename);
/* settings below */
Error.stackTraceLimit = Infinity;
// During the test the env variable is set to test
process.env.NODE_ENV = 'test';
process.env.SUMMALY_ALLOW_PRIVATE_IP = 'true';
const port = 3060;
const host = `http://localhost:${port}`;
// Display detail of unhandled promise rejection
process.on('unhandledRejection', console.dir);
let app: ReturnType<typeof fastify> | null = null;
let n = 0;
afterEach(async () => {
if (app) {
await app.close();
app = null;
}
});
/* tests below */
test('faviconがHTML上で指定されていないが、ルートに存在する場合、正しく設定される', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/no-favicon.html'));
});
app.get('/favicon.ico', (_, reply) => reply.status(200).send());
await app.listen({ port });
const summary = await summaly(host);
expect(summary.icon).toBe(`${host}/favicon.ico`);
});
test('faviconがHTML上で指定されていなくて、ルートにも存在しなかった場合 null になる', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/no-favicon.html'));
});
app.get('*', (_, reply) => reply.status(404).send());
await app.listen({ port });
const summary = await summaly(host);
expect(summary.icon).toBe(null);
});
test('titleがcleanupされる', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/dirty-title.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.title).toBe('Strawberry Pasta');
});
describe('Private IP blocking', () => {
beforeEach(() => {
process.env.SUMMALY_ALLOW_PRIVATE_IP = 'false';
app = fastify();
app.get('*', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/og-title.html'));
});
return app.listen({ port });
});
test('private ipなサーバーの情報を取得できない', async () => {
const summary = await summaly(host).catch((e: StatusError) => e);
if (summary instanceof StatusError) {
expect(summary.name).toBe('StatusError');
} else {
expect(summary).toBeInstanceOf(StatusError);
}
});
test('agentが指定されている場合はprivate ipを許可', async () => {
const summary = await summaly(host, {
agent: {
http: new httpAgent({ keepAlive: true }),
https: new httpsAgent({ keepAlive: true }),
}
});
expect(summary.title).toBe('Strawberry Pasta');
});
test('agentが空のオブジェクトの場合はprivate ipを許可しない', async () => {
const summary = await summaly(host, { agent: {} }).catch((e: StatusError) => e);
if (summary instanceof StatusError) {
expect(summary.name).toBe('StatusError');
} else {
expect(summary).toBeInstanceOf(StatusError);
}
});
afterEach(() => {
process.env.SUMMALY_ALLOW_PRIVATE_IP = 'true';
});
});
describe('OGP', () => {
test('title', async () => {
app = fastify();
app.get('*', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/og-title.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.title).toBe('Strawberry Pasta');
});
test('description', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/og-description.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.description).toBe('Strawberry Pasta');
});
test('site_name', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/og-site_name.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.sitename).toBe('Strawberry Pasta');
});
test('thumbnail', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/og-image.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.thumbnail).toBe('https://himasaku.net/himasaku.png');
});
});
describe('TwitterCard', () => {
test('title', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/twitter-title.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.title).toBe('Strawberry Pasta');
});
test('description', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/twitter-description.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.description).toBe('Strawberry Pasta');
});
test('thumbnail', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/twitter-image.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.thumbnail).toBe('https://himasaku.net/himasaku.png');
});
test('Player detection - PeerTube:video => video', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/player-peertube-video.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/embedurl');
expect(summary.player.allow).toStrictEqual(['autoplay', 'encrypted-media', 'fullscreen']);
});
test('Player detection - Pleroma:video => video', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/player-pleroma-video.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/embedurl');
expect(summary.player.allow).toStrictEqual(['autoplay', 'encrypted-media', 'fullscreen']);
});
test('Player detection - Pleroma:image => image', async () => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(_dirname + '/htmls/player-pleroma-image.html'));
});
await app.listen({ port });
const summary = await summaly(host);
expect(summary.thumbnail).toBe('https://example.com/imageurl');
});
});
describe("oEmbed", () => {
const setUpFastify = async (oEmbedPath: string, htmlPath = 'htmls/oembed.html') => {
app = fastify();
app.get('/', (request, reply) => {
return reply.send(fs.createReadStream(new URL(htmlPath, import.meta.url)));
});
app.get('/oembed.json', (request, reply) => {
return reply.send(fs.createReadStream(
new URL(oEmbedPath, new URL('oembed/', import.meta.url))
));
});
await app.listen({ port });
}
for (const filename of readdirSync(new URL('oembed/invalid', import.meta.url))) {
test(`Invalidity test: ${filename}`, async () => {
await setUpFastify(`invalid/${filename}`);
const summary = await summaly(host);
expect(summary.player.url).toBe(null);
});
}
test('basic properties', async () => {
await setUpFastify('oembed.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.width).toBe(500);
expect(summary.player.height).toBe(300);
});
test('type: video', async () => {
await setUpFastify('oembed-video.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.width).toBe(500);
expect(summary.player.height).toBe(300);
});
test('max height', async () => {
await setUpFastify('oembed-too-tall.json');
const summary = await summaly(host);
expect(summary.player.height).toBe(1024);
});
test('children are ignored', async () => {
await setUpFastify('oembed-iframe-child.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
});
test('allows fullscreen', async () => {
await setUpFastify('oembed-allow-fullscreen.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.allow).toStrictEqual(['fullscreen'])
});
test('allows safelisted permissions', async () => {
await setUpFastify('oembed-allow-safelisted-permissions.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.allow).toStrictEqual([
'autoplay', 'clipboard-write', 'fullscreen',
'encrypted-media', 'picture-in-picture', 'web-share',
]);
});
test('ignores rare permissions', async () => {
await setUpFastify('oembed-ignore-rare-permissions.json');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.allow).toStrictEqual(['autoplay']);
});
test('oEmbed with relative path', async () => {
await setUpFastify('oembed.json', 'htmls/oembed-relative.html');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
});
test('oEmbed with nonexistent path', async () => {
await setUpFastify('oembed.json', 'htmls/oembed-nonexistent-path.html');
await expect(summaly(host)).rejects.toThrow('404 Not Found');
});
test('oEmbed with wrong path', async () => {
await setUpFastify('oembed.json', 'htmls/oembed-wrong-path.html');
await expect(summaly(host)).rejects.toThrow();
});
test('oEmbed with OpenGraph', async () => {
await setUpFastify('oembed.json', 'htmls/oembed-and-og.html');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.description).toBe('blobcats rule the world');
});
test('Invalid oEmbed with valid OpenGraph', async () => {
await setUpFastify('invalid/oembed-insecure.json', 'htmls/oembed-and-og.html');
const summary = await summaly(host);
expect(summary.player.url).toBe(null);
expect(summary.description).toBe('blobcats rule the world');
});
test('oEmbed with og:video', async () => {
await setUpFastify('oembed.json', 'htmls/oembed-and-og-video.html');
const summary = await summaly(host);
expect(summary.player.url).toBe('https://example.com/');
expect(summary.player.allow).toStrictEqual([]);
});
test('width: 100%', async () => {
await setUpFastify('oembed-percentage-width.json');
const summary = await summaly(host);
expect(summary.player.width).toBe(null);
expect(summary.player.height).toBe(300);
});
});

View File

@ -0,0 +1,7 @@
{
"version": "1.0",
"type": "rich",
"html": "<iframe src='https://example.com/' allowfullscreen></iframe>",
"width": 500,
"height": 300
}

45
test/tsconfig.json Normal file
View File

@ -0,0 +1,45 @@
{
"compilerOptions": {
"allowJs": true,
"noEmitOnError": false,
"noImplicitAny": true,
"noImplicitReturns": true,
"noUnusedParameters": false,
"noUnusedLocals": false,
"noFallthroughCasesInSwitch": true,
"declaration": true,
"sourceMap": false,
"target": "es2021",
"module": "nodenext",
"moduleResolution": "nodenext",
"allowSyntheticDefaultImports": true,
"removeComments": false,
"noLib": false,
"strict": true,
"strictNullChecks": true,
"strictPropertyInitialization": false,
"skipLibCheck": true,
"experimentalDecorators": true,
"emitDecoratorMetadata": true,
"resolveJsonModule": true,
"isolatedModules": true,
"baseUrl": "./../",
"paths": {
"@/*": [
"./src/*"
]
},
"outDir": "./built",
"typeRoots": [
"./node_modules/@types",
"./src/@types"
],
"lib": [
"esnext"
]
},
"compileOnSave": false,
"include": [
"./**/*"
],
}

View File

@ -10,8 +10,8 @@
"declaration": true,
"sourceMap": false,
"target": "es2021",
"module": "esnext",
"moduleResolution": "node",
"module": "nodenext",
"moduleResolution": "nodenext",
"allowSyntheticDefaultImports": true,
"removeComments": false,
"noLib": false,
@ -23,7 +23,6 @@
"emitDecoratorMetadata": true,
"resolveJsonModule": true,
"isolatedModules": true,
"rootDir": "./src",
"baseUrl": "./",
"paths": {
"@/*": [
@ -44,6 +43,9 @@
},
"compileOnSave": false,
"include": [
"./src/**/*.ts"
"./src/**/*"
],
"exclude": [
"node_modules",
]
}

View File

@ -1,86 +0,0 @@
{
"rules": {
"align": [true,
"parameters",
"statements"
],
"ban": false,
"class-name": true,
"comment-format": [true,
"check-upper-case"
],
"curly": true,
"eofline": true,
"forin": false,
"indent": [true, "tabs"],
"interface-name": false,
"jsdoc-format": true,
"label-position": true,
"label-undefined": true,
"max-line-length": false,
"member-access": false,
"member-ordering": [true,
"static-before-instance",
"variables-before-functions"
],
"no-any": false,
"no-arg": true,
"no-bitwise": true,
"no-console": [true,
"debug",
"info",
"time",
"timeEnd",
"trace"
],
"no-consecutive-blank-lines": true,
"no-construct": true,
"no-constructor-vars": true,
"no-debugger": true,
"no-duplicate-key": true,
"no-shadowed-variable": false,
"no-duplicate-variable": true,
"no-empty": true,
"no-eval": true,
"no-internal-module": true,
"no-require-imports": false,
"no-string-literal": false,
"no-switch-case-fall-through": true,
"no-trailing-whitespace": true,
"no-unreachable": true,
"no-unused-expression": true,
"no-unused-variable": true,
"no-use-before-declare": true,
"no-var-keyword": true,
"no-var-requires": false,
"one-line": [true,
"check-catch",
"check-whitespace"
],
"quotemark": false,
"radix": true,
"semicolon": true,
"switch-default": false,
"triple-equals": false,
"typedef": [true,
"call-signature",
"property-declaration"
],
"typedef-whitespace": [true, {
"call-signature": "nospace",
"index-signature": "nospace",
"parameter": "nospace",
"property-declaration": "nospace",
"variable-declaration": "nospace"
}],
"use-strict": false,
"variable-name": false,
"whitespace": [true,
"check-branch",
"check-decl",
"check-operator",
"check-separator",
"check-type"
]
}
}

10
vitest.config.ts Normal file
View File

@ -0,0 +1,10 @@
import { fileURLToPath } from 'node:url';
import { defineConfig } from 'vitest/config';
export default defineConfig({
resolve: {
alias: {
'@': fileURLToPath(new URL('./src', import.meta.url)),
},
},
});