docs(rate-limits): Worker binding counts per machine; test over one connection (#168)

Measured on production after the 4.5.0 deploy: with a new connection per
request, 1,000 requests in 53 s got no 429 from RL_SITE (120/60 s); over
one keep-alive connection, 429 after ~93. The ratelimits binding counts per
machine of a data centre and syncs late, so the zone's WAF rule is the
limit that holds against floods. Verified once the owner set the rule
(300/10 s): 800 requests in 4.9 s gave 111 x 200, 201 x Worker 429 and
488 x WAF 1015, with the site answering again 10 s later.

Runbook test commands, code comments, CLAUDE.md gotcha 43, the 4.5.0
changelog entry.

[skip-discuss]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 authored and GitHub committed 2026-10-27 06:17:42 +07:00
1 parent 7920011c70
commit 45f64e6c39
5 files changed
+34 -13

No files matched your search

+2 -1
View File
@@ -42,7 +42,7 @@ Local admin without Access: put `DEV_ADMIN_EMAIL=you@example.com` in `webapp/.de
## Worker (`webapp/worker/`)
- `index.ts` routes: `/img/*` (img.ts, behind the gate), `/api/health`, `/api/verify` (gate.ts), `/api/suggest` (suggest.ts), `/api/admin/*` + `/admin*` (admin.ts), the readable `/data/*.json` (legacy-data.ts), everything else that misses an asset → `spa.ts`. `run_worker_first` also lists `/` (otherwise served straight from `dist/index.html`) so the gate sees it, and the five legacy catalog files one by one (never a `!` pattern — gotcha 26). `scheduled()` (cron `23 */4 * * *`), four steps each in its own try/catch: `discover.ts` (one itch.io RSS feed per run, 6 h back-off on 429) → `QueueStore.reconcile` (queued → ingested / failed with the pipeline's outcome, read at head through the GitHub App; else in the catalog → ingested, > 3 days → failed `timeout`) → `automation.ts` (auto-approve + followed collections, ≤ 30 itch.io pages a run, ONE `[auto]` commit, D1 written only after it lands; the run's summary in settings `automation_last_run`) → `purge` (Suggest notes 180 d). The account is on **Workers Paid** (30 s CPU): admin catalog edits parse and rewrite ~3.8 MB in the Worker (≈ 25–30 ms for 100 removals).
- `ratelimit.ts` (since 4.5): every Worker request is counted first, per IP (IPv6 /64, `rateKey`) and per Cloudflare location (approximate, fails open), in one `ratelimits` binding per kind of traffic — `RL_SITE` 120/60 s (pages, public `/api`, legacy `/data`, misses), `RL_IMG` 600, `RL_ADMIN` 300; `RL_SUGGEST` 5 and `RL_GATE` 20 still apply on top to those POSTs. Verified bots (the gate's header) are not counted. Over a limit: 429 + `Retry-After: 60` — JSON `rate_limited` for `/api`, `/img`, files and `/data` (with CORS), the script-less EN/VI page of `gate-page.ts` for app pages. Keep every `period` 60 and a unique `namespace_id` in 88xx (staging 89xx): namespaces are **account-wide** and other Workers on the account (repolens, ticktune…) use 1001–1006, so the old 1001 / 1003 shared their counters (`ratelimit.test.ts` reads `wrangler.jsonc`). Layer 1 is the zone's WAF rate-limiting rule (Free plan: one rule, IP, 10 s; `docs/operations.md` §1 step 15): only it saves the Worker request a flood costs, and only it covers static assets.
- `ratelimit.ts` (since 4.5): every Worker request is counted first, per IP (IPv6 /64, `rateKey`) and per Cloudflare location (approximate, fails open), in one `ratelimits` binding per kind of traffic (counted per machine of the data centre: it holds for one-connection clients, the WAF holds for floods — gotcha 43) — `RL_SITE` 120/60 s (pages, public `/api`, legacy `/data`, misses), `RL_IMG` 600, `RL_ADMIN` 300; `RL_SUGGEST` 5 and `RL_GATE` 20 still apply on top to those POSTs. Verified bots (the gate's header) are not counted. Over a limit: 429 + `Retry-After: 60` — JSON `rate_limited` for `/api`, `/img`, files and `/data` (with CORS), the script-less EN/VI page of `gate-page.ts` for app pages. Keep every `period` 60 and a unique `namespace_id` in 88xx (staging 89xx): namespaces are **account-wide** and other Workers on the account (repolens, ticktune…) use 1001–1006, so the old 1001 / 1003 shared their counters (`ratelimit.test.ts` reads `wrangler.jsonc`). Layer 1 is the zone's WAF rate-limiting rule (Free plan: one rule, IP, 10 s; `docs/operations.md` §1 step 15): only it saves the Worker request a flood costs, and only it covers static assets.
- `img.ts`: `/img/<160|640>/<path on img.itch.zone>` — allow-listed against `data/urls.json`, edge cache → R2 `THUMBS` (`w{w}/<sha256>.webp`) → `cf.image` transform (stored once) → fallback streams the original with a 1-day cache (quota error 9422). The fallback original and a missing upstream image are also kept in `caches.default` for a day (GET only); widths are the exact strings `160` / `640` (a `Map`, no `Number()` aliases); background puts log failures. Upstream errors → controlled 502.
- `spa.ts`: `not_found_handling` is `"none"`, so misses reach the Worker: `/assets/*`, `/data/*` (a pack from an older deploy included — `no-store`, CORS) and file-like paths get a real 404 (`no-store`); app routes get the shell with 200; unknown paths and game slugs not in the catalog get the shell with **404**; with the gate on, a visitor without a pass gets the verification page instead (checked before the shell is fetched, so never a 304). Keep its `ROUTES` in sync with `src/App.tsx` and `src/hooks/useSeoHead.ts`.
- `gate.ts` + `gate-page.ts`: the **verification gate** (web only). App pages and `/img` need a pass: the `__Host-fig_gate` cookie (HMAC of its issue time, HttpOnly, `BOT_GATE_TTL_HOURS`, default 48) set by `POST /api/verify` after a Turnstile check (action `gate`, `RL_GATE`, same-origin); `GET /api/verify` reports `{enabled, valid, via, ttl}`. The page is self-contained (inline CSS/JS with a per-response CSP nonce, EN/VI, the site's og tags, 200 + noindex). Verified bots pass with `x-fig-known-bot: <GATE_BOT_TOKEN>` from a zone Transform Rule; clients that only claim to be a crawler get 503. `/assets`, `/data`, `/api/*`, `/admin*` and the other static files stay open. `turnstile.ts`: siteverify shared with Suggest (hostname + action checked; the test secrets skip both).
@@ -152,6 +152,7 @@ Code **MIT** (`LICENSE`); catalog data **CC BY 4.0** for the maintainer's contri
40. **The Python requirements are hash-locked pip-tools compiles** — `requirements.txt` ← `requirements.in`, `requirements-dev.txt` ← `requirements-dev.in` (`-c requirements.txt`) — installed everywhere with `--require-hashes --only-binary :all:`, so a new release of any dependency, indirect ones included, can't reach the jobs that push to `main`, and no sdist build script runs. After editing a `.in`, recompile with **pip-tools on Python 3.14**: `pip-compile --generate-hashes --allow-unsafe --strip-extras requirements.in`, then the same for `requirements-dev.in`. Dependabot takes the Python version from the `# … autogenerated by pip-compile with Python X.Y` header and supports ≤ 3.14 (gotcha 39); the hashes cover every file of a release, so CI's 3.15 still finds its cp315 wheels. `colorama` is listed in `requirements-dev.in` because pytest needs it on Windows only: a lock compiled on Linux (Dependabot's) would otherwise leave it out and the hash-checked local install would fail. Keep those pip commands in a block scalar (`run: |`): in a plain YAML scalar the `: ` of `:all: -r` makes the whole workflow file invalid — GitHub then records a 0-job failed push run named after the file, and a PR simply shows no check for that workflow.
41. **SonarQube Cloud analyses in CI** (`sonarcloud.yml` + `sonar-project.properties`; project `poli0981_free-games-itchio-list`, public). Automatic Analysis is switched off on sonarcloud.io — switched back on, it makes the CI scan fail. PRs get a "SonarCloud Code Analysis" check from the SonarQube Cloud app; Dependabot and fork PRs get no `SONAR_TOKEN`, so their analysis is skipped with a notice (a Dependabot secret of that name would enable it). Tests run in the `coverage` job without the token; besides the `token` job's presence check, only the `scan` step gets it, in a job that runs no project code. Binary images and the D1 migrations (SQLite, claimed by the Oracle PL/SQL analyzer, which then warned on every run) are excluded from the analysis. Coverage: `pytest --cov` (`[tool.coverage.run]` in `pyproject.toml`) and `npm run test:coverage` (`test.coverage` in `vite.config.ts`; the lcov reporter's `projectRoot` is the repo root, so the scanner resolves the paths); `@vitest/coverage-v8` needs vitest's exact version (Dependabot group `vitest`). The built-in "Sonar way" gate fails a PR under 80 % coverage on new code (from 20 new lines up), so code without unit tests by design is in `sonar.coverage.exclusions` — keep that list in step with what the tests reach. `sonar.projectVersion` comes from `webapp/package.json`: on `main`, "new code" (previous version) starts over at each version bump. `sonar.python.version` stays 3.14 until SonarQube Cloud's analyzer supports 3.15 (it warns "No explicit support for version 3.15"). The free plan keeps only `main` and PRs: a manual run on another branch uploads a branch analysis nobody can open ("not allowed to access data from non main branches"). Mark a false positive on the issue itself. Two of its suggested fixes are wrong here: S2871 wants `localeCompare` in `.sort()`, but chunk names, migration files and `unblocked.json` keep code-unit order (= Python's `sorted()`) — pass `compareCodeUnits` (`worker/text.ts`); S8906 wants `select_one('div.a.b')` for bs4's `class_=[a, b]`, which means *either* class — write a selector list `div.a, div.b`. Trim trailing characters with `trimEnd()` (`worker/text.ts`), not `/x+$/` (S8786: quadratic backtracking on a long run). Known false positive: css:S8776 on Tailwind v4's `@utility … { &::-webkit-scrollbar … }` (the `&` is required for variants).
42. **`overrides.undici` in `webapp/package.json` is a stopgap**: miniflare (wrangler's local runtime) pins `undici` exactly — 7.29.1 in wrangler 4.152, three advisories fixed in 7.30.1, so Dependabot could not raise it — and the override lifts it to `^7.30.1`. Only miniflare uses undici; nothing of it reaches the site, the Worker or the apps. Remove the override once `npm ls undici` without it shows ≥ 7.30.1, and before miniflare moves to undici 8 (the override would hold it on 7).
43. **The Workers rate-limit binding counts per machine, not per data centre.** Measured on production right after the 4.5.0 deploy (CDG): 1,000 requests in 53 s, each `curl` on a new connection, against `RL_SITE` 120/60 s → **no 429 at all**; 200 requests on **one** keep-alive connection → 93 × 200, then 429s. A data centre has many machines, each counts locally and syncs late, so a client that keeps one connection (browsers over HTTP/2, scripts with keep-alive) is held to the limit, while one that opens a new connection per request is spread across machines and barely counted. The zone's WAF rule counts across the data centre: it is the limit that holds against floods. Test the Worker limits over one connection (`curl URL URL …` in one invocation, or `--next`), never with `xargs curl`; under `wrangler dev` (miniflare: one process) every request counts.
## Safe-edit rules
+3 -1
View File
@@ -28,7 +28,9 @@ requests") and has to wait a minute. Nothing changes in the desktop and Android
(`Retry-After: 60`) and are never cached. The admin shows "Too many requests from this connection".
- **A first layer in front of the Worker**: the maintainer's runbook (`docs/operations.md`, section 1
step 15) sets up a Cloudflare WAF rate-limiting rule that blocks floods before they reach the site's
code, and covers the static files the Worker never sees.
code, and covers the static files the Worker never sees. The Worker's own counts are approximate
(Cloudflare counts them per server and syncs them late), so the WAF rule is what holds against a
flood that opens a new connection for every request.
### Fixed
+19 -7
View File
@@ -139,7 +139,9 @@ Do these once (and again with the `-staging` names for the staging Worker).
no bot token is needed).
15. **Rate limits** — two layers, both per client IP (an IPv6 client by its /64 in the Worker).
- **Layer 1, the zone's WAF rule** (Security → WAF → **Rate limiting rules**). It runs before the
Worker, so a request it blocks costs no Worker request, and it is the only limit on static files.
Worker, so a request it blocks costs no Worker request, it is the only limit on static files, and
it counts across the whole data centre — the only limit a flood of new connections can't spread
out of (see layer 2).
On the Free plan there is **one** rule, counted by IP over **10 seconds**, with a 10-second block;
if a rule already exists (the old one for `/api/*`), edit it rather than adding one. Settings:
- Rule name `Flood guard`. *If incoming requests match* → *Edit expression*:
@@ -164,19 +166,29 @@ Do these once (and again with the `-staging` names for the staging Worker).
(`/admin` + `/api/admin`); on top, `RL_SUGGEST` 5 (Suggest POSTs) and `RL_GATE` 20 (verification
POSTs). Over a limit: 429 + `Retry-After: 60` — JSON `{"error":"rate_limited"}` for the API,
covers and files (with CORS on `/data`), a small "Too many requests" page (EN/VI, no script) for
app pages. Verified bots (the gate's header) are never counted. Change a limit in
app pages. Verified bots (the gate's header) are never counted. **These counts are loose**:
each machine of a data centre counts on its own and syncs late, so they hold for a client that
keeps one connection (browsers use one HTTP/2 connection, scripts with keep-alive) but hardly
for one that opens a new connection per request — measured after the 4.5.0 deploy, 1,000 such
requests in a minute got no 429. That is the WAF rule's job. Change a limit in
`wrangler.jsonc` (then `npm run cf:types`); keep `period` 60 and a new `namespace_id` per binding in
88xx (staging 89xx) — the ids are account-wide, and the account's other Workers use the low numbers.
The counts are approximate and fail open (a missing binding or a limiter error lets requests
through).
- **Test** after a change (it blocks your own IP for up to a minute):
- **Test** after a change (it blocks your own IP for up to a minute). The Worker's limits, over
**one** connection (a single `curl` with many URLs reuses it):
```sh
seq 150 | xargs -P 30 -I{} curl -s -o /dev/null -w "%{http_code}\n" https://freeitchgames.win/api/health | sort | uniq -c
curl -s -o /dev/null -w "%{http_code}\n" $(printf 'https://freeitchgames.win/api/health %.0s' $(seq 200)) | sort | uniq -c
curl -s https://freeitchgames.win/api/health # {"error":"rate_limited"} → the Worker; "error code: 1015" → the WAF rule
```
About 120 answer 200 and the rest 429. To see the WAF rule fire, send more than its threshold
within 10 s (`seq 400 | xargs -P 60 …`): the extra requests get `error code: 1015` and show up in
Security → Events. Worker 429s show up in Workers Logs (sampled, `head_sampling_rate` 0.1).
About 120 answer 200 and the rest 429 (fewer pass if you sent requests in the last minute). The
WAF rule, with a new connection per request, more than its threshold within 10 s:
```sh
seq 400 | xargs -P 60 -I{} curl -s -o /dev/null -w "%{http_code}\n" https://freeitchgames.win/api/health | sort | uniq -c
```
Past ~300 the answers turn into 429 `error code: 1015` (Security → Events lists them); without
the rule all 400 answer 200. Worker 429s show up in Workers Logs (sampled,
`head_sampling_rate` 0.1).
## 2. GitHub
+6 -2
View File
@@ -15,8 +15,12 @@
* Suggest and the gate's POST keep their own, stricter limits on top
* (RL_SUGGEST, RL_GATE). Verified bots (the gate's Transform Rule header) are
* not counted, so a crawl never rate-limits itself out of the index. Counts
* are per Cloudflare location and approximate (the binding's design). Fails
* open: a missing binding or a failing limiter lets the request through.
* are approximate (the binding's design): each machine of a data centre
* counts on its own and syncs late, so a client that keeps one connection
* (a browser, a script with keep-alive) is held to the limit, but one that
* opens a new connection per request is barely counted — the WAF rule is the
* limit for floods. Fails open: a missing binding or a failing limiter lets
* the request through.
*/
import type { WorkerEnv } from './env'
import { botPass } from './gate'
+4 -2
View File
@@ -64,8 +64,10 @@
"migrations_dir": "./worker/migrations"
}
],
// Per-IP limits (an IPv6 client by its /64), counted per Cloudflare location;
// over a limit the Worker answers 429 + Retry-After (worker/ratelimit.ts).
// Per-IP limits (an IPv6 client by its /64), counted per machine of each
// Cloudflare data centre (approximate: they hold for a client on one
// connection, not for one that opens a new connection per request); over a
// limit the Worker answers 429 + Retry-After (worker/ratelimit.ts).
// Every request that runs the Worker counts against one of RL_SITE (app
// pages, the public /api, the readable /data JSON, misses), RL_IMG (covers:
// a Games page shows up to 100) or RL_ADMIN (/admin + /api/admin, behind