feat(worker): rate limits for the whole site (4.5.0) (#167)

Every request that runs the Worker is counted per client (IPv4, or IPv6
/64): RL_SITE 120/min (app pages, public /api, the old apps' /data JSON,
misses), RL_IMG 600/min (covers), RL_ADMIN 300/min (/admin + /api/admin);
RL_SUGGEST 5 and RL_GATE 20 stay on top for their POSTs. Verified bots are
not counted. Over a limit: 429 + Retry-After 60 (JSON for the API, covers,
files and /data with CORS; a script-less EN/VI page for app pages). Fails
open.

Every rate-limit namespace moves to 8801-8806 (staging 8901-8906):
namespace ids are account-wide, and other Workers on the account use
1001-1006, so RL_SUGGEST / RL_GATE shared their counters.

docs/operations.md section 1 step 15: the zone's WAF rate-limiting rule as
the first layer (Free plan: one rule, IP, 10 s; 300 requests / 10 s).

[skip-discuss]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 authored and GitHub committed 2026-10-27 06:02:49 +07:00
1 parent 84ab9b7761
commit 7920011c70
23 files changed
+493 -27

No files matched your search

+2 -1
View File
@@ -3,12 +3,13 @@
All notable changes to this project, newest first. The notes of each release live in
[`changelog/`](changelog/), one file per minor version: `changelog/<major>.x/ver<major>.<minor>.x.md`.
**Latest release: [4.4.3](changelog/4.x/ver4.4.x.md#443---2026-10-26-dependency-and-security-updates) (2026-10-26) — Dependency and security updates.**
**Latest release: [4.5.0](changelog/4.x/ver4.5.x.md#450---2026-10-27-rate-limits-for-the-whole-site) (2026-10-27) — Rate limits for the whole site.**
## 4.x
| Version | Date | Highlights |
|---|---|---|
| [4.5.0](changelog/4.x/ver4.5.x.md#450---2026-10-27-rate-limits-for-the-whole-site) | 2026-10-27 | Rate limits for the whole site (HTTP 429), WAF guide |
| [4.4.3](changelog/4.x/ver4.4.x.md#443---2026-10-26-dependency-and-security-updates) | 2026-10-26 | Dependency and security updates (build-tool advisories closed) |
| [4.4.2](changelog/4.x/ver4.4.x.md#442---2026-10-22-toolchain-upgrade-code-scan-fixes) | 2026-10-22 | Toolchain upgrade (TypeScript 7, Node 26, Python 3.15), code-scan fixes |
| [4.4.1](changelog/4.x/ver4.4.x.md#441---2026-10-16-extension-and-bot-retired) | 2026-10-16 | Extension and bot retired (archived repos), ingest API removed |
+3
View File
@@ -42,6 +42,7 @@ Local admin without Access: put `DEV_ADMIN_EMAIL=you@example.com` in `webapp/.de
## Worker (`webapp/worker/`)
- `index.ts` routes: `/img/*` (img.ts, behind the gate), `/api/health`, `/api/verify` (gate.ts), `/api/suggest` (suggest.ts), `/api/admin/*` + `/admin*` (admin.ts), the readable `/data/*.json` (legacy-data.ts), everything else that misses an asset → `spa.ts`. `run_worker_first` also lists `/` (otherwise served straight from `dist/index.html`) so the gate sees it, and the five legacy catalog files one by one (never a `!` pattern — gotcha 26). `scheduled()` (cron `23 */4 * * *`), four steps each in its own try/catch: `discover.ts` (one itch.io RSS feed per run, 6 h back-off on 429) → `QueueStore.reconcile` (queued → ingested / failed with the pipeline's outcome, read at head through the GitHub App; else in the catalog → ingested, > 3 days → failed `timeout`) → `automation.ts` (auto-approve + followed collections, ≤ 30 itch.io pages a run, ONE `[auto]` commit, D1 written only after it lands; the run's summary in settings `automation_last_run`) → `purge` (Suggest notes 180 d). The account is on **Workers Paid** (30 s CPU): admin catalog edits parse and rewrite ~3.8 MB in the Worker (≈ 25–30 ms for 100 removals).
- `ratelimit.ts` (since 4.5): every Worker request is counted first, per IP (IPv6 /64, `rateKey`) and per Cloudflare location (approximate, fails open), in one `ratelimits` binding per kind of traffic — `RL_SITE` 120/60 s (pages, public `/api`, legacy `/data`, misses), `RL_IMG` 600, `RL_ADMIN` 300; `RL_SUGGEST` 5 and `RL_GATE` 20 still apply on top to those POSTs. Verified bots (the gate's header) are not counted. Over a limit: 429 + `Retry-After: 60` — JSON `rate_limited` for `/api`, `/img`, files and `/data` (with CORS), the script-less EN/VI page of `gate-page.ts` for app pages. Keep every `period` 60 and a unique `namespace_id` in 88xx (staging 89xx): namespaces are **account-wide** and other Workers on the account (repolens, ticktune…) use 1001–1006, so the old 1001 / 1003 shared their counters (`ratelimit.test.ts` reads `wrangler.jsonc`). Layer 1 is the zone's WAF rate-limiting rule (Free plan: one rule, IP, 10 s; `docs/operations.md` §1 step 15): only it saves the Worker request a flood costs, and only it covers static assets.
- `img.ts`: `/img/<160|640>/<path on img.itch.zone>` — allow-listed against `data/urls.json`, edge cache → R2 `THUMBS` (`w{w}/<sha256>.webp`) → `cf.image` transform (stored once) → fallback streams the original with a 1-day cache (quota error 9422). The fallback original and a missing upstream image are also kept in `caches.default` for a day (GET only); widths are the exact strings `160` / `640` (a `Map`, no `Number()` aliases); background puts log failures. Upstream errors → controlled 502.
- `spa.ts`: `not_found_handling` is `"none"`, so misses reach the Worker: `/assets/*`, `/data/*` (a pack from an older deploy included — `no-store`, CORS) and file-like paths get a real 404 (`no-store`); app routes get the shell with 200; unknown paths and game slugs not in the catalog get the shell with **404**; with the gate on, a visitor without a pass gets the verification page instead (checked before the shell is fetched, so never a 304). Keep its `ROUTES` in sync with `src/App.tsx` and `src/hooks/useSeoHead.ts`.
- `gate.ts` + `gate-page.ts`: the **verification gate** (web only). App pages and `/img` need a pass: the `__Host-fig_gate` cookie (HMAC of its issue time, HttpOnly, `BOT_GATE_TTL_HOURS`, default 48) set by `POST /api/verify` after a Turnstile check (action `gate`, `RL_GATE`, same-origin); `GET /api/verify` reports `{enabled, valid, via, ttl}`. The page is self-contained (inline CSS/JS with a per-response CSP nonce, EN/VI, the site's og tags, 200 + noindex). Verified bots pass with `x-fig-known-bot: <GATE_BOT_TOKEN>` from a zone Transform Rule; clients that only claim to be a crawler get 503. `/assets`, `/data`, `/api/*`, `/admin*` and the other static files stay open. `turnstile.ts`: siteverify shared with Suggest (hostname + action checked; the test secrets skip both).
@@ -161,6 +162,8 @@ Code **MIT** (`LICENSE`); catalog data **CC BY 4.0** for the maintainer's contri
## Current state (Oct 2026)
**v4.5.0** (2026-10-27): rate limits for the whole site — `worker/ratelimit.ts` + `RL_SITE` / `RL_IMG` / `RL_ADMIN`, every namespace moved to 8801–8806 / staging 8901–8906 (account-wide ids; 1001 / 1003 were shared with other Workers), the 429 page, admin wording, `docs/operations.md` §1 step 15 (the WAF rule: `not cf.client.bot`, not `/assets/` or `/data/pack/`, 300 requests / 10 s, block 10 s — set by the owner in the dashboard). No policy change (`LEGAL_VERSION` unchanged; SECURITY.md EN + VI describe the limits).
**v4.4.3** (2026-10-26): dependency and security updates only — the week's npm updates (Dependabot #159–#163 in one PR, plus lucide-react / Zustand; `knip` kept at 6.42.1 by the 3-day cooldown), wrangler 4.152 with regenerated `worker-configuration.d.ts` (a wrangler bump fails `cf:types -- --check` until it is regenerated — why Dependabot's build-tooling PR failed), `overrides.undici` (gotcha 42), zod 4.6.7, `cargo update` (tauri 2.12.5, tokio 1.54.1; `cargo audit`: 0 vulnerabilities, 5 upstream warnings — glib, zbus family, proc-macro-error), certifi / coverage / Ruff via pip-tools on 3.14, `actions/setup-python` 7.1.0. Repository security state that day: CodeQL default setup **not configured** (last analyses May 2026, 0 open alerts), Dependabot **security updates disabled** in Settings (version updates run), secret scanning + push protection on, SonarQube Cloud gate OK with 0 issues.
**SonarQube Cloud CI** (2026-10-22, shipped in 4.4.3's notes): CI-based analysis with test coverage replaced Automatic Analysis — `sonarcloud.yml`, `sonar-project.properties` (tests marked as tests, coverage exclusions, `.sonarcloud.properties` removed), `pytest-cov` + `@vitest/coverage-v8` (gotcha 41).
+1 -1
View File
@@ -1,6 +1,6 @@
# Free Itch.io Games List
[![Version](https://img.shields.io/badge/version-4.4.0-blue.svg)](https://github.com/poli0981/free-games-itchio-list/releases/latest)
[![Version](https://img.shields.io/badge/version-4.5.0-blue.svg)](https://github.com/poli0981/free-games-itchio-list/releases/latest)
[![Website](https://img.shields.io/badge/website-freeitchgames.win-purple.svg)](https://freeitchgames.win)
[![Stars](https://img.shields.io/github/stars/poli0981/free-games-itchio-list?style=social)](https://github.com/poli0981/free-games-itchio-list/stargazers)
[![Forks](https://img.shields.io/github/forks/poli0981/free-games-itchio-list?style=social)](https://github.com/poli0981/free-games-itchio-list/network/members)
+1 -1
View File
@@ -1,6 +1,6 @@
# Danh sách Game Itch.io Miễn Phí
[![Version](https://img.shields.io/badge/version-4.4.0-blue.svg)](https://github.com/poli0981/free-games-itchio-list/releases/latest)
[![Version](https://img.shields.io/badge/version-4.5.0-blue.svg)](https://github.com/poli0981/free-games-itchio-list/releases/latest)
[![Website](https://img.shields.io/badge/website-freeitchgames.win-purple.svg)](https://freeitchgames.win)
[![Stars](https://img.shields.io/github/stars/poli0981/free-games-itchio-list?style=social)](https://github.com/poli0981/free-games-itchio-list/stargazers)
[![Forks](https://img.shields.io/github/forks/poli0981/free-games-itchio-list?style=social)](https://github.com/poli0981/free-games-itchio-list/network/members)
+1 -1
View File
@@ -76,7 +76,7 @@ What the repository actually does:
- **Dependabot** ([`.github/dependabot.yml`](.github/dependabot.yml)) is configured to keep npm, pip, GitHub Actions and cargo dependencies current.
- Every third-party **GitHub Action is pinned by full commit SHA**, and each workflow declares least-privilege `permissions`.
- Writes to the repository from the website's admin go through a **GitHub App** (verified commits), and the admin page is behind Cloudflare Access (GitHub login or one-time email PIN) and limited to the Maintainer.
- The Suggest form is protected by Cloudflare Turnstile and rate limiting.
- The website limits how many requests one address can send in a short time (a Cloudflare WAF rule, then per-route limits in the Worker that answer HTTP 429), and the Suggest form adds Cloudflare Turnstile and a stricter limit of its own.
- The Android APK is signed with the project's key. Install it, and the desktop apps, only from the project's [GitHub Releases](https://github.com/poli0981/free-games-itchio-list/releases) page.
If a known vulnerability in a dependency actually affects a deployed component (the website, the Worker, the pipeline or the apps), report it privately as in [§1](#1-reporting-a-vulnerability). If it is just an outdated dependency with no practical impact, a normal issue or pull request is fine.
+42
View File
@@ -0,0 +1,42 @@
# Changelog — 4.5.x
Release notes for 4.5.0, newest first. Every version: [CHANGELOG.md](../../CHANGELOG.md).
## [4.5.0] - 2026-10-27 (Rate limits for the whole site)
The website now limits how many requests one connection can send in a short time, on every page,
cover image and API address — not only on the Suggest form and the verification check. Normal
browsing never comes close to the limits; a script that floods the site gets HTTP 429 ("Too many
requests") and has to wait a minute. Nothing changes in the desktop and Android apps.
### Upgrade notes
- **Nothing to accept again**: no policy changed. The Privacy Policy already says that your IP
address is used, without being stored, to count requests and limit abuse.
- **Desktop and Android apps**: install 4.5.0 over 4.4.3 as usual; they don't use the limited
addresses (4.0–4.1.1 count against the site limit when they read the old catalog files).
### Added
- **Per-connection limits in the Worker**: every request that runs the site's server code counts,
per minute and per address (an IPv6 connection by its /64): 120 for pages, the public API and the
catalog files of the old apps, 600 for cover images (a Games page shows up to 100), 300 for the
admin. The Suggest form (5 a minute) and the verification check (20 a minute) keep their own,
stricter limits on top. Search engines that Cloudflare has verified are not counted.
- **The "Too many requests" answer**: pages get a short page in English or Vietnamese (no scripts),
the API and cover images a JSON `{"error":"rate_limited"}`; both say when to retry
(`Retry-After: 60`) and are never cached. The admin shows "Too many requests from this connection".
- **A first layer in front of the Worker**: the maintainer's runbook (`docs/operations.md`, section 1
step 15) sets up a Cloudflare WAF rate-limiting rule that blocks floods before they reach the site's
code, and covers the static files the Worker never sees.
### Fixed
- **Suggest and verification limits counted other sites too**: a rate limit's namespace is shared by
every Worker of the Cloudflare account, and the two existing limits used the same numbers as other
Workers there, so their visits could use up this site's Suggest or verification allowance. All of
this site's limits now have namespaces of their own.
### Changed
- **Security policy**: says how the site limits requests (EN + VI); no new obligations.
+1 -1
View File
@@ -78,7 +78,7 @@ Những gì repository thực sự làm:
- **Dependabot** ([`.github/dependabot.yml`](../../../.github/dependabot.yml)) được cấu hình để giữ các dependency npm, pip, GitHub Actions và cargo luôn được cập nhật.
- Mọi **GitHub Action của bên thứ ba đều được ghim theo commit SHA đầy đủ**, và mỗi workflow khai báo `permissions` ở mức tối thiểu cần thiết.
- Các thao tác ghi vào repository từ trang quản trị của website đều đi qua một **GitHub App** (commit được xác minh), và trang quản trị nằm sau Cloudflare Access (đăng nhập GitHub hoặc mã PIN dùng một lần qua email) và chỉ dành cho Người duy trì.
- Form Suggest được bảo vệ bằng Cloudflare Turnstile và giới hạn tốc độ.
- Website giới hạn số request mà một địa chỉ có thể gửi trong thời gian ngắn (một quy tắc WAF của Cloudflare, rồi các giới hạn theo từng loại đường dẫn trong Worker, trả về HTTP 429), và form Suggest còn có thêm Cloudflare Turnstile cùng một giới hạn chặt hơn của riêng nó.
- File APK Android được ký bằng khóa của dự án. Chỉ cài APK, cũng như các ứng dụng desktop, từ trang [GitHub Releases](https://github.com/poli0981/free-games-itchio-list/releases) của dự án.
Nếu một lỗ hổng đã biết trong dependency thực sự ảnh hưởng tới một thành phần đang chạy (website, Worker, pipeline hoặc các ứng dụng), hãy báo cáo riêng tư như ở [§1](#1-báo-cáo-lỗ-hổng). Nếu đó chỉ là một dependency lỗi thời, không có tác động thực tế, thì mở issue hoặc pull request thông thường là được.
+47 -2
View File
@@ -15,6 +15,7 @@ secret: secrets are only ever typed into Cloudflare (`wrangler secret put`) or a
| Access app "Admin" | Cloudflare Zero Trust | Login for `/admin` + `/api/admin/*`. |
| Turnstile widget | Cloudflare Turnstile | The verification gate in front of the web app (action `gate`) and anti-spam on `/suggest` (action `suggest`). |
| Transform Rule "Verified bots skip the gate" | Cloudflare → Rules → Transform Rules | Adds `x-fig-known-bot: <GATE_BOT_TOKEN>` to requests from Cloudflare-verified bots, so search engines and link previews skip the gate (section 1, step 14). |
| Rate limits | WAF rule (Security → WAF → Rate limiting rules) + the Worker's `ratelimits` bindings | Two layers of per-IP limits: the WAF rule stops floods before they run (and bill) the Worker; the Worker answers what still gets through with 429 per kind of traffic (section 1, step 15). |
| GitHub App | GitHub → Settings → Developer settings | The Worker's identity for commits (queue approvals, edits, removals). |
| GitHub Actions | [`.github/workflows/`](../.github/workflows) | Data pipeline (`refresh.yml`, `update.yml`, `force_update.yml`), CI, releases, build monitor. |
| Staging Worker `free-games-itchio-list-staging` | `staging.freeitchgames.win` (whole host behind Access) | Optional pre-release checks, deployed by hand from any branch with `npx wrangler deploy --env staging` (`wrangler.jsonc` → `env.staging`). |
@@ -81,8 +82,8 @@ Do these once (and again with the `-staging` names for the staging Worker).
DMARC for poli0981.dev live in that zone's Cloudflare DNS). freeitchgames.win needs no mail setup.
12. **Zone settings**: SSL/TLS Full (strict), Always Use HTTPS, minimum TLS 1.2, TLS 1.3 on; Smart
Tiered Cache; Browser Cache TTL "Respect existing headers"; Rocket Loader and Email Obfuscation
**off** (they rewrite HTML and break the CSP); WAF managed rules on; one rate-limiting rule for
`/api/*`; Bot Fight Mode **off** (it can't be skipped for the apps; the
**off** (they rewrite HTML and break the CSP); WAF managed rules on; the rate-limiting rule of
step 15; Bot Fight Mode **off** (it can't be skipped for the apps; the
verification gate covers the web app instead). Keep HSTS
`preload` only if every subdomain (including `staging`) will always be HTTPS.
- Measured 2026-10-03: TLS 1.0/1.1 are refused, TLS 1.3 and HTTP/3 answer, `http://` redirects
@@ -136,6 +137,46 @@ Do these once (and again with the `-staging` names for the staging Worker).
`SITE_ORIGIN=http://localhost:8787` and a 32+ character `GATE_SECRET` in `webapp/.dev.vars`, then
`npm run build && npx wrangler dev` (on localhost the cookie is `fig_gate`, without `Secure`, and
no bot token is needed).
15. **Rate limits** — two layers, both per client IP (an IPv6 client by its /64 in the Worker).
- **Layer 1, the zone's WAF rule** (Security → WAF → **Rate limiting rules**). It runs before the
Worker, so a request it blocks costs no Worker request, and it is the only limit on static files.
On the Free plan there is **one** rule, counted by IP over **10 seconds**, with a 10-second block;
if a rule already exists (the old one for `/api/*`), edit it rather than adding one. Settings:
- Rule name `Flood guard`. *If incoming requests match* → *Edit expression*:
```
(not cf.client.bot and not starts_with(http.request.uri.path, "/assets/") and not starts_with(http.request.uri.path, "/data/pack/"))
```
(In the builder: *Verified Bot* equals off, *URI Path* does not start with `/assets/`, *URI
Path* does not start with `/data/pack/`, joined with And. If the editor refuses
`starts_with`, use `not http.request.uri.path wildcard r"/assets/*"` and the same for
`/data/pack/*`.) Bundles and packs stay out: they are free static assets with a long cache, and
a first visit loads a few dozen of them at once. Verified bots stay out like at the gate.
- *With the same characteristics*: IP. *When rate exceeds*: **300 requests** per **10 seconds**
(30 a second; a person scrolling the Games list fast loads about 100 covers in that time).
*Then take action*: **Block**, default response (429, Cloudflare's "error 1015" page), for
10 seconds. Deploy.
- Lower it once Security → Events shows what real traffic looks like; don't go below ~150 per
10 s (a Games page asks for up to 100 covers).
- **Layer 2, the Worker** ([`webapp/worker/ratelimit.ts`](../webapp/worker/ratelimit.ts),
`ratelimits` in `wrangler.jsonc`, deployed with every build — nothing to set up). Every request
that runs the Worker counts, per 60 s and per Cloudflare location: `RL_SITE` 120 (app pages, the
public `/api`, the readable `/data` JSON, misses), `RL_IMG` 600 (`/img` covers), `RL_ADMIN` 300
(`/admin` + `/api/admin`); on top, `RL_SUGGEST` 5 (Suggest POSTs) and `RL_GATE` 20 (verification
POSTs). Over a limit: 429 + `Retry-After: 60` — JSON `{"error":"rate_limited"}` for the API,
covers and files (with CORS on `/data`), a small "Too many requests" page (EN/VI, no script) for
app pages. Verified bots (the gate's header) are never counted. Change a limit in
`wrangler.jsonc` (then `npm run cf:types`); keep `period` 60 and a new `namespace_id` per binding in
88xx (staging 89xx) — the ids are account-wide, and the account's other Workers use the low numbers.
The counts are approximate and fail open (a missing binding or a limiter error lets requests
through).
- **Test** after a change (it blocks your own IP for up to a minute):
```sh
seq 150 | xargs -P 30 -I{} curl -s -o /dev/null -w "%{http_code}\n" https://freeitchgames.win/api/health | sort | uniq -c
curl -s https://freeitchgames.win/api/health # {"error":"rate_limited"} → the Worker; "error code: 1015" → the WAF rule
```
About 120 answer 200 and the rest 429. To see the WAF rule fire, send more than its threshold
within 10 s (`seq 400 | xargs -P 60 …`): the extra requests get `error code: 1015` and show up in
Security → Events. Worker 429s show up in Workers Logs (sampled, `head_sampling_rate` 0.1).
## 2. GitHub
@@ -210,6 +251,10 @@ Do these once (and again with the `-staging` names for the staging Worker).
```
`npx wrangler tail` shows the `legacy-data: 410` lines with the requesting `Origin`: an installed
app that sends an unexpected one would show up there.
- **Visitors report "Too many requests" / error 1015**: "error 1015" is the WAF rule (section 1,
step 15) — raise its threshold in the dashboard, effective within a minute. The site's own "Too many
requests" page or `{"error":"rate_limited"}` is the Worker: raise the limit in `wrangler.jsonc` and
deploy. Both reset by themselves (10 s / 60 s).
- **Verification gate misbehaving** (visitors stuck on the check, crawl errors in Search Console):
`npx wrangler secret delete GATE_SECRET` turns it off at once; then look at `npx wrangler tail` and
`curl -s https://freeitchgames.win/api/verify`.
+5 -2
View File
@@ -3,7 +3,7 @@ import { ApiError, SessionExpired } from './api'
import { ERROR_TEXT, errorText } from './texts'
// The sources of every error the admin API answers with.
const SOURCES = import.meta.glob<string>(['../worker/admin.ts', '../worker/http.ts'], {
const SOURCES = import.meta.glob<string>(['../worker/admin.ts', '../worker/http.ts', '../worker/ratelimit.ts'], {
query: '?raw',
import: 'default',
eager: true,
@@ -12,8 +12,11 @@ const SOURCES = import.meta.glob<string>(['../worker/admin.ts', '../worker/http.
describe('admin error texts', () => {
it('has wording for every error code of the admin API', () => {
const codes = new Set(
Object.values(SOURCES).flatMap((src) => [...src.matchAll(/HttpError\(\d{3}, '([a-z_]+)'/g)].map((m) => m[1])),
Object.values(SOURCES).flatMap((src) =>
[...src.matchAll(/HttpError\(\d{3}, '([a-z_]+)'|json\(\{ error: '([a-z_]+)' \}/g)].map((m) => m[1] ?? m[2]),
),
)
expect(codes).toContain('rate_limited')
expect(codes.size).toBeGreaterThan(15)
expect([...codes].filter((c) => !(c in ERROR_TEXT))).toEqual([])
})
+1
View File
@@ -11,6 +11,7 @@ export const ERROR_TEXT: Record<string, string> = {
github_error: 'GitHub refused the change.',
network_error: 'The server could not be reached. Check the connection and try again.',
internal_error: 'Something went wrong on the server.',
rate_limited: 'Too many requests from this connection. Wait a minute, then try again.',
not_found: 'Not found.',
invalid_body: 'The request was malformed.',
invalid_json: 'The request was malformed.',
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "webapp",
"version": "4.4.3",
"version": "4.5.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "webapp",
"version": "4.4.3",
"version": "4.5.0",
"license": "MIT",
"dependencies": {
"@fontsource-variable/geist": "^5.3.0",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "webapp",
"private": true,
"version": "4.4.3",
"version": "4.5.0",
"license": "MIT",
"type": "module",
"engines": {
+1 -1
View File
@@ -1027,7 +1027,7 @@ dependencies = [
[[package]]
name = "free-games-itchio-webapp"
version = "4.4.3"
version = "4.5.0"
dependencies = [
"serde",
"serde_json",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "free-games-itchio-webapp"
version = "4.4.3"
version = "4.5.0"
description = "Itch.io free games catalog viewer"
authors = ["poli0981"]
license = "MIT"
+1 -1
View File
@@ -1,7 +1,7 @@
{
"$schema": "https://schema.tauri.app/config/2",
"productName": "Itch.io Free Games DB",
"version": "4.4.3",
"version": "4.5.0",
"identifier": "com.poli0981.freegamesitchio",
"build": {
"beforeDevCommand": "npm run dev",
+1 -1
View File
@@ -8,7 +8,7 @@ export interface ThirdParty {
export const APP = {
name: 'Free Itch Games',
version: '4.4.3',
version: '4.5.0',
repo: 'https://github.com/poli0981/free-games-itchio-list',
license: 'code MIT · data CC BY 4.0',
} as const
+7 -1
View File
@@ -1,11 +1,14 @@
/* eslint-disable */
// Generated by Wrangler by running `wrangler types --env-file=types.env --strict-vars=false` (hash: bb5186e82d126ae7e5c424393b53d523)
// Generated by Wrangler by running `wrangler types --env-file=types.env --strict-vars=false` (hash: 2c5596f40780529393972ecb04342777)
// Runtime types generated with workerd@1.20261019.1 2026-09-15
interface __BaseEnv_Env {
THUMBS: R2Bucket;
DB: D1Database;
RL_SUGGEST: RateLimit;
RL_GATE: RateLimit;
RL_SITE: RateLimit;
RL_IMG: RateLimit;
RL_ADMIN: RateLimit;
ASSETS: Fetcher;
SITE_ORIGIN: string;
GITHUB_REPO: string;
@@ -27,6 +30,9 @@ declare namespace Cloudflare {
DB: D1Database;
RL_SUGGEST: RateLimit;
RL_GATE: RateLimit;
RL_SITE: RateLimit;
RL_IMG: RateLimit;
RL_ADMIN: RateLimit;
ASSETS: Fetcher;
SITE_ORIGIN: string;
GITHUB_REPO: string;
+64 -3
View File
@@ -1,8 +1,9 @@
/**
* The verification page of the bot gate (gate.ts). Self-contained: inline CSS
* The verification page of the bot gate (gate.ts), and the "too many
* requests" page of the rate limits (ratelimit.ts). Self-contained: inline CSS
* and scripts, system fonts, the logo inline — nothing of the app loads before
* the visitor passes. Built by the Worker, so it sets its own security headers
* (public/_headers only covers static assets).
* the visitor passes. Built by the Worker, so they set their own security
* headers (public/_headers only covers static assets).
*
* Status 200 with noindex: link-preview bots still read the site's og: tags,
* search engines don't index the page. Verified crawlers never see it (gate.ts);
@@ -266,6 +267,66 @@ export function challengePage(request: Request, opts: ChallengeOptions): Respons
})
}
// ── Too many requests ────────────────────────────────────────────────────────
const LIMITED = {
title: { en: 'Too many requests — Free Itch Games', vi: 'Quá nhiều yêu cầu — Free Itch Games' },
heading: { en: 'Too many requests', vi: 'Quá nhiều yêu cầu' },
lead: {
en: 'Your connection sent too many requests in a short time. Wait a minute, then reload the page.',
vi: 'Kết nối của bạn đã gửi quá nhiều yêu cầu trong thời gian ngắn. Hãy đợi một phút rồi tải lại trang.',
},
}
/** No script at all: the page only shows text (Accept-Language picks it, the system theme colours it). */
const LIMITED_CSP = [
"default-src 'none'",
"style-src 'unsafe-inline'",
"img-src 'self'",
"base-uri 'none'",
"form-action 'none'",
"frame-ancestors 'none'",
].join('; ')
function renderRateLimited(lang: Lang): string {
return `<!doctype html>
<html lang="${lang}">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<meta name="robots" content="noindex, nofollow">
<meta name="color-scheme" content="light dark">
<title>${esc(LIMITED.title[lang])}</title>
<link rel="icon" type="image/svg+xml" href="/favicon.svg">
<style>${CSS}</style>
</head>
<body>
<main>
<div class="brand">${LOGO}<span>Free Itch Games</span></div>
<h1>${both(LIMITED.heading)}</h1>
<p class="lead">${both(LIMITED.lead)}</p>
</main>
</body>
</html>
`
}
/** An app page request over its rate limit (429). */
export function rateLimitedPage(request: Request, retryAfter: number): Response {
const lang = pickLanguage(request.headers.get('accept-language'))
return new Response(request.method === 'HEAD' ? null : renderRateLimited(lang), {
status: 429,
headers: {
...PAGE_HEADERS,
'Content-Type': 'text/html; charset=utf-8',
'Cache-Control': 'no-store',
'Retry-After': String(retryAfter),
'X-Robots-Tag': 'noindex, nofollow',
'Content-Security-Policy': LIMITED_CSP,
},
})
}
/** For a client that claims to be a search crawler but did not come through the verified-bot rule. */
export function crawlerUnavailable(request: Request): Response {
return new Response(request.method === 'HEAD' ? null : 'Temporarily unavailable. Please retry later.\n', {
+69
View File
@@ -110,3 +110,72 @@ describe('router, gate off', () => {
expect(res.status).not.toBe(403)
})
})
describe('rate limits', () => {
/** The binding's behaviour for one window: the first `limit` requests per key pass. */
function bucket(limit: number) {
const counts = new Map<string, number>()
return {
limit: vi.fn(async ({ key }: RateLimitOptions) => {
const n = (counts.get(key) ?? 0) + 1
counts.set(key, n)
return { success: n <= limit }
}),
}
}
function limited(over: Record<string, unknown> = {}) {
return env({ RL_SITE: bucket(120), RL_IMG: bucket(600), RL_ADMIN: bucket(300), ...over })
}
const IP = { 'cf-connecting-ip': '198.51.100.20' }
it('lets 120 requests a window through, then answers a burst with 429', async () => {
const e = limited()
const statuses = await Promise.all(Array.from({ length: 150 }, () => run('/api/health', IP, e).then((r) => r.status)))
expect(statuses.filter((s) => s === 200)).toHaveLength(120)
expect(statuses.filter((s) => s === 429)).toHaveLength(30)
// Other clients are not affected.
expect((await run('/api/health', { 'cf-connecting-ip': '198.51.100.21' }, e)).status).toBe(200)
})
it('stops a flooding client before any work: pages, API, data', async () => {
const e = limited({ RL_SITE: bucket(0) })
const page = await run('/games', IP, e)
expect([page.status, page.headers.get('content-type')]).toEqual([429, 'text/html; charset=utf-8'])
expect(page.headers.get('retry-after')).toBe('60')
const suggest = await run('/api/suggest', IP, e)
expect(await suggest.json()).toEqual({ error: 'rate_limited' })
expect((await run('/data/index.json', { ...IP, origin: 'tauri://localhost' }, e)).status).toBe(429)
expect((e as ReturnType<typeof env>).ASSETS.fetch).not.toHaveBeenCalled()
})
it('limits covers on their own budget, before the gate and the edge cache', async () => {
const e = limited({ RL_IMG: bucket(2) })
const statuses = []
for (let i = 0; i < 3; i++) statuses.push((await run('/img/160/abc/original/x.png', IP, e)).status)
expect(statuses).toEqual([403, 403, 429])
expect(cache.match).not.toHaveBeenCalled()
// Pages still work for the same client.
expect((await run('/games', IP, e)).status).toBe(200)
})
it('counts an IPv6 client by its /64', async () => {
const e = limited({ RL_SITE: bucket(1) })
expect((await run('/api/health', { 'cf-connecting-ip': '2001:db8:5:6::1' }, e)).status).toBe(200)
expect((await run('/api/health', { 'cf-connecting-ip': '2001:db8:5:6:ffff::2' }, e)).status).toBe(429)
})
it('never limits verified bots', async () => {
const e = limited({ RL_SITE: bucket(0) })
const res = await run('/games', { ...IP, 'x-fig-known-bot': BOT }, e)
expect(await res.text()).toBe(SHELL)
})
it('limits the admin on its own budget', async () => {
const e = limited({ RL_ADMIN: bucket(0) })
expect((await run('/api/admin/queue', IP, e)).status).toBe(429)
expect((await run('/admin/', IP, e)).status).toBe(429)
expect((await run('/api/health', IP, e)).status).toBe(200)
})
})
+6
View File
@@ -13,6 +13,9 @@
* /admin, /admin/* admin app shell, maintainer only (admin.ts)
* /data/*.json readable catalog JSON, for apps 4.0–4.1.1 only (legacy-data.ts)
*
* Every request is first counted against its per-client rate limit
* (ratelimit.ts; over it → 429).
*
* Cron: one RSS feed per run into the review queue, queue bookkeeping (with
* the pipeline's ingest outcomes), the admin's automatic rules (auto-approve,
* watched itch.io collections — automation.ts), then retention.
@@ -27,6 +30,7 @@ import { json } from './http'
import { handleImg } from './img'
import { handleLegacyData } from './legacy-data'
import { QueueStore } from './queue'
import { rateLimit } from './ratelimit'
import { repoOps } from './repo'
import { handleMiss } from './spa'
import { handleSuggest } from './suggest'
@@ -38,6 +42,8 @@ function queueStore(env: WorkerEnv): QueueStore | null {
export default {
async fetch(request, env: WorkerEnv, ctx): Promise<Response> {
const url = new URL(request.url)
const limited = await rateLimit(request, url, env)
if (limited) return limited
const path = url.pathname
const data = { assets: env.ASSETS, origin: url.origin }
const gate = gateConfig(env, url)
+151
View File
@@ -0,0 +1,151 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { unstable_readConfig } from 'wrangler'
import type { WorkerEnv } from './env'
import { RATE_PERIOD, rateBucket, rateLimit, tooManyRequests } from './ratelimit'
const ORIGIN = 'https://freeitchgames.win'
const BOT = 'b'.repeat(40)
function request(path: string, headers: Record<string, string> = {}, method = 'GET') {
return new Request(new URL(path, ORIGIN), { headers: { 'cf-connecting-ip': '203.0.113.7', ...headers }, method })
}
function limiter(success: boolean) {
return { limit: vi.fn<(options: RateLimitOptions) => Promise<RateLimitOutcome>>(async () => ({ success })) }
}
function env(over: Record<string, unknown> = {}) {
return { GATE_BOT_TOKEN: BOT, ...over } as unknown as WorkerEnv
}
function check(path: string, e: WorkerEnv, headers: Record<string, string> = {}) {
const req = request(path, headers)
return rateLimit(req, new URL(req.url), e)
}
afterEach(() => vi.restoreAllMocks())
describe('rateBucket', () => {
it('counts covers, the maintainer and everything else separately', () => {
expect(rateBucket('/img/160/abc/original/x.png')).toBe('RL_IMG')
for (const path of ['/admin', '/admin/', '/admin/assets/app.js', '/api/admin/queue']) {
expect(rateBucket(path)).toBe('RL_ADMIN')
}
for (const path of ['/', '/games', '/games/x', '/api/verify', '/api/suggest', '/api/health', '/data/index.json', '/administrator', '/x.txt']) {
expect(rateBucket(path)).toBe('RL_SITE')
}
})
})
describe('tooManyRequests', () => {
function answer(path: string, method = 'GET') {
const req = request(path, { 'accept-language': 'vi-VN,vi;q=0.9' }, method)
return tooManyRequests(req, new URL(req.url))
}
it('gives app pages the 429 page: no-store, Retry-After, no script allowed', async () => {
for (const path of ['/', '/games', '/games/some-game', '/admin', '/admin/queue']) {
const res = answer(path)
expect(res.status).toBe(429)
expect(res.headers.get('content-type')).toBe('text/html; charset=utf-8')
expect(res.headers.get('retry-after')).toBe(String(RATE_PERIOD))
expect(res.headers.get('cache-control')).toBe('no-store')
expect(res.headers.get('x-frame-options')).toBe('DENY')
expect(res.headers.get('content-security-policy')).toMatch(/^default-src 'none';/)
expect(res.headers.get('content-security-policy')).not.toContain('script-src')
const html = await res.text()
expect(html).toContain('<html lang="vi">')
expect(html).toContain('Quá nhiều yêu cầu')
expect(html).not.toContain('<script')
}
})
it('answers HEAD without a body', async () => {
const res = answer('/games', 'HEAD')
expect(res.status).toBe(429)
expect(await res.text()).toBe('')
})
it('answers the API, covers and files with JSON', async () => {
for (const path of ['/api/health', '/api/admin/queue', '/img/160/x', '/assets/x.js', '/admin/assets/app.js', '/robots.txt']) {
const res = answer(path)
expect(res.status).toBe(429)
expect(res.headers.get('content-type')).toBe('application/json; charset=utf-8')
expect(res.headers.get('retry-after')).toBe(String(RATE_PERIOD))
expect(res.headers.get('cache-control')).toBe('no-store')
expect(res.headers.get('access-control-allow-origin')).toBeNull()
expect(await res.json()).toEqual({ error: 'rate_limited' })
}
})
it('keeps CORS on /data, so the apps read the 429 instead of a CORS error', async () => {
const res = answer('/data/index.json')
expect([res.status, res.headers.get('access-control-allow-origin')]).toEqual([429, '*'])
expect(await res.json()).toEqual({ error: 'rate_limited' })
})
})
describe('rateLimit', () => {
it('counts each request against its bucket, keyed by the client address', async () => {
const site = limiter(true)
const img = limiter(true)
const e = env({ RL_SITE: site, RL_IMG: img, RL_ADMIN: limiter(true) })
expect(await check('/games', e)).toBeNull()
expect(await check('/img/160/x', e, { 'cf-connecting-ip': '2001:db8:1:2:3:4:5:6' })).toBeNull()
expect(site.limit).toHaveBeenCalledWith({ key: '203.0.113.7' })
expect(img.limit).toHaveBeenCalledWith({ key: '2001:db8:1:2::/64' })
})
it('answers 429 once the bucket says no', async () => {
const res = await check('/api/health', env({ RL_SITE: limiter(false) }))
expect(res?.status).toBe(429)
expect(await res?.json()).toEqual({ error: 'rate_limited' })
})
it("doesn't count verified bots, but does count a wrong bot token", async () => {
const site = limiter(false)
const e = env({ RL_SITE: site })
expect(await check('/games', e, { 'x-fig-known-bot': BOT })).toBeNull()
expect(site.limit).not.toHaveBeenCalled()
expect((await check('/games', e, { 'x-fig-known-bot': 'x'.repeat(40) }))?.status).toBe(429)
})
it('fails open without a binding or when the limiter throws', async () => {
expect(await check('/games', env())).toBeNull()
const error = vi.spyOn(console, 'error').mockImplementation(() => {})
const broken = {
limit: vi.fn(async () => {
throw new Error('down')
}),
}
expect(await check('/games', env({ RL_SITE: broken }))).toBeNull()
expect(error).toHaveBeenCalledOnce()
})
})
describe('wrangler.jsonc', () => {
const NAMES = ['RL_SITE', 'RL_IMG', 'RL_ADMIN']
interface Limit {
name: string
namespace_id: string
simple: { limit: number; period: number }
}
function limits(envName: string | undefined): Limit[] {
return unstable_readConfig({ config: 'wrangler.jsonc', env: envName }, { hideWarnings: true }).ratelimits as Limit[]
}
it.each([undefined, 'staging'])('declares every bucket with a period of RATE_PERIOD (env %s)', (envName) => {
const byName = new Map(limits(envName).map((r) => [r.name, r]))
for (const name of NAMES) expect(byName.get(name)?.simple.period).toBe(RATE_PERIOD)
})
it('gives every binding its own namespace, production and staging apart', () => {
const ids = [undefined, 'staging'].flatMap((envName) => limits(envName).map((r) => r.namespace_id))
expect(ids.length).toBeGreaterThanOrEqual(10)
expect(new Set(ids).size).toBe(ids.length)
// Namespaces are account-wide: other Workers on the account use the low numbers.
expect(ids.filter((id) => !/^8[89]\d\d$/.test(id))).toEqual([])
})
})
+60
View File
@@ -0,0 +1,60 @@
/**
* Per-client request limits for everything that runs the Worker: the second
* layer behind the zone's WAF rate-limiting rule (docs/operations.md §1
* step 15). Static assets (/assets/*, /data/pack/*, …) are served without
* running the Worker, so only the WAF rule limits those — and only the WAF
* saves the Worker request a flood would cost.
*
* One `ratelimits` binding per kind of traffic (wrangler.jsonc), keyed by the
* client's IPv4 address or IPv6 /64 (rateKey):
*
* RL_IMG /img/* covers (a Games page shows up to 100)
* RL_ADMIN /admin, /admin/*, /api/admin/* the maintainer (Access checks who first)
* RL_SITE the rest: app pages, the public /api, the readable /data JSON, misses
*
* Suggest and the gate's POST keep their own, stricter limits on top
* (RL_SUGGEST, RL_GATE). Verified bots (the gate's Transform Rule header) are
* not counted, so a crawl never rate-limits itself out of the index. Counts
* are per Cloudflare location and approximate (the binding's design). Fails
* open: a missing binding or a failing limiter lets the request through.
*/
import type { WorkerEnv } from './env'
import { botPass } from './gate'
import { rateLimitedPage } from './gate-page'
import { json, rateKey } from './http'
/** `period` of every limit in wrangler.jsonc (ratelimit.test.ts checks), sent as Retry-After. */
export const RATE_PERIOD = 60
export type RateBucket = 'RL_SITE' | 'RL_IMG' | 'RL_ADMIN'
export function rateBucket(path: string): RateBucket {
if (path.startsWith('/img/')) return 'RL_IMG'
if (path === '/admin' || path.startsWith('/admin/') || path.startsWith('/api/admin/')) return 'RL_ADMIN'
return 'RL_SITE'
}
/** The 429: JSON for the API, covers, data and files; the "too many requests" page for app pages. */
export function tooManyRequests(request: Request, url: URL): Response {
const path = url.pathname
const retry = { 'Retry-After': String(RATE_PERIOD) }
// CORS on /data, so the desktop/Android apps see the 429, not a CORS error.
if (path.startsWith('/data/')) return json({ error: 'rate_limited' }, 429, { ...retry, 'Access-Control-Allow-Origin': '*' })
const last = path.slice(path.lastIndexOf('/') + 1)
if (/^\/(api|img|assets)\//.test(path) || last.includes('.')) return json({ error: 'rate_limited' }, 429, retry)
return rateLimitedPage(request, RATE_PERIOD)
}
/** Counts the request; null lets it through, else the 429 to answer with. */
export async function rateLimit(request: Request, url: URL, env: WorkerEnv): Promise<Response | null> {
const limiter: Pick<RateLimit, 'limit'> | undefined = env[rateBucket(url.pathname)]
if (!limiter) return null
if (await botPass(request, env.GATE_BOT_TOKEN ?? '')) return null
try {
if ((await limiter.limit({ key: rateKey(request) })).success) return null
} catch (e) {
console.error('rate limit: limiter failed, request let through', e)
return null
}
return tooManyRequests(request, url)
}
+25 -7
View File
@@ -64,12 +64,27 @@
"migrations_dir": "./worker/migrations"
}
],
// Per-IP limits for the Suggest form and the verification gate (an IPv6
// client by its /64). The gate's limit is higher: carrier-grade NAT puts many
// phones behind one address. (Namespace 1002 was the retired ingest API's.)
// Per-IP limits (an IPv6 client by its /64), counted per Cloudflare location;
// over a limit the Worker answers 429 + Retry-After (worker/ratelimit.ts).
// Every request that runs the Worker counts against one of RL_SITE (app
// pages, the public /api, the readable /data JSON, misses), RL_IMG (covers:
// a Games page shows up to 100) or RL_ADMIN (/admin + /api/admin, behind
// Access); verified bots don't count. POSTs to the Suggest form and the
// verification gate also count against their own, stricter RL_SUGGEST /
// RL_GATE (the gate's is higher: carrier-grade NAT puts many phones behind
// one address). Static assets never run the Worker: the zone's WAF
// rate-limiting rule is the first layer for everything (docs/operations.md
// §1 step 15). Keep "period" 60 (the Retry-After). A namespace_id is
// account-wide: two bindings with the same one share their counters, even in
// different Workers. Other Workers on this account use the low numbers
// (1001–1006…), so this one keeps to 88xx (staging 89xx; before 4.5.0 it had
// 1001–1003 / 2001–2003, shared with them).
"ratelimits": [
{ "name": "RL_SUGGEST", "namespace_id": "1001", "simple": { "limit": 5, "period": 60 } },
{ "name": "RL_GATE", "namespace_id": "1003", "simple": { "limit": 20, "period": 60 } }
{ "name": "RL_SUGGEST", "namespace_id": "8801", "simple": { "limit": 5, "period": 60 } },
{ "name": "RL_GATE", "namespace_id": "8803", "simple": { "limit": 20, "period": 60 } },
{ "name": "RL_SITE", "namespace_id": "8804", "simple": { "limit": 120, "period": 60 } },
{ "name": "RL_IMG", "namespace_id": "8805", "simple": { "limit": 600, "period": 60 } },
{ "name": "RL_ADMIN", "namespace_id": "8806", "simple": { "limit": 300, "period": 60 } }
],
// RSS discovery (one feed per run) + queue bookkeeping, worker/index.ts scheduled().
"triggers": { "crons": ["23 */4 * * *"] },
@@ -104,8 +119,11 @@
{ "binding": "DB", "database_name": "freeitchgames-staging", "migrations_dir": "./worker/migrations" }
],
"ratelimits": [
{ "name": "RL_SUGGEST", "namespace_id": "2001", "simple": { "limit": 5, "period": 60 } },
{ "name": "RL_GATE", "namespace_id": "2003", "simple": { "limit": 20, "period": 60 } }
{ "name": "RL_SUGGEST", "namespace_id": "8901", "simple": { "limit": 5, "period": 60 } },
{ "name": "RL_GATE", "namespace_id": "8903", "simple": { "limit": 20, "period": 60 } },
{ "name": "RL_SITE", "namespace_id": "8904", "simple": { "limit": 120, "period": 60 } },
{ "name": "RL_IMG", "namespace_id": "8905", "simple": { "limit": 600, "period": 60 } },
{ "name": "RL_ADMIN", "namespace_id": "8906", "simple": { "limit": 300, "period": 60 } }
],
// No RSS polling from staging: production is the only itch.io client.
"triggers": { "crons": [] },