docs(rate-limits): Worker binding counts per machine; test over one connection (#168)
Measured on production after the 4.5.0 deploy: with a new connection per request, 1,000 requests in 53 s got no 429 from RL_SITE (120/60 s); over one keep-alive connection, 429 after ~93. The ratelimits binding counts per machine of a data centre and syncs late, so the zone's WAF rule is the limit that holds against floods. Verified once the owner set the rule (300/10 s): 800 requests in 4.9 s gave 111 x 200, 201 x Worker 429 and 488 x WAF 1015, with the site answering again 10 s later. Runbook test commands, code comments, CLAUDE.md gotcha 43, the 4.5.0 changelog entry. [skip-discuss] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7920011c70
commit
45f64e6c39
5 files changed
+34
-13
No files matched your search
@@ -42,7 +42,7 @@ Local admin without Access: put `DEV_ADMIN_EMAIL=you@example.com` in `webapp/.de
|
||||
## Worker (`webapp/worker/`)
|
||||
|
||||
- `index.ts` routes: `/img/*` (img.ts, behind the gate), `/api/health`, `/api/verify` (gate.ts), `/api/suggest` (suggest.ts), `/api/admin/*` + `/admin*` (admin.ts), the readable `/data/*.json` (legacy-data.ts), everything else that misses an asset → `spa.ts`. `run_worker_first` also lists `/` (otherwise served straight from `dist/index.html`) so the gate sees it, and the five legacy catalog files one by one (never a `!` pattern — gotcha 26). `scheduled()` (cron `23 */4 * * *`), four steps each in its own try/catch: `discover.ts` (one itch.io RSS feed per run, 6 h back-off on 429) → `QueueStore.reconcile` (queued → ingested / failed with the pipeline's outcome, read at head through the GitHub App; else in the catalog → ingested, > 3 days → failed `timeout`) → `automation.ts` (auto-approve + followed collections, ≤ 30 itch.io pages a run, ONE `[auto]` commit, D1 written only after it lands; the run's summary in settings `automation_last_run`) → `purge` (Suggest notes 180 d). The account is on **Workers Paid** (30 s CPU): admin catalog edits parse and rewrite ~3.8 MB in the Worker (≈ 25–30 ms for 100 removals).
|
||||
- `ratelimit.ts` (since 4.5): every Worker request is counted first, per IP (IPv6 /64, `rateKey`) and per Cloudflare location (approximate, fails open), in one `ratelimits` binding per kind of traffic — `RL_SITE` 120/60 s (pages, public `/api`, legacy `/data`, misses), `RL_IMG` 600, `RL_ADMIN` 300; `RL_SUGGEST` 5 and `RL_GATE` 20 still apply on top to those POSTs. Verified bots (the gate's header) are not counted. Over a limit: 429 + `Retry-After: 60` — JSON `rate_limited` for `/api`, `/img`, files and `/data` (with CORS), the script-less EN/VI page of `gate-page.ts` for app pages. Keep every `period` 60 and a unique `namespace_id` in 88xx (staging 89xx): namespaces are **account-wide** and other Workers on the account (repolens, ticktune…) use 1001–1006, so the old 1001 / 1003 shared their counters (`ratelimit.test.ts` reads `wrangler.jsonc`). Layer 1 is the zone's WAF rate-limiting rule (Free plan: one rule, IP, 10 s; `docs/operations.md` §1 step 15): only it saves the Worker request a flood costs, and only it covers static assets.
|
||||
- `ratelimit.ts` (since 4.5): every Worker request is counted first, per IP (IPv6 /64, `rateKey`) and per Cloudflare location (approximate, fails open), in one `ratelimits` binding per kind of traffic (counted per machine of the data centre: it holds for one-connection clients, the WAF holds for floods — gotcha 43) — `RL_SITE` 120/60 s (pages, public `/api`, legacy `/data`, misses), `RL_IMG` 600, `RL_ADMIN` 300; `RL_SUGGEST` 5 and `RL_GATE` 20 still apply on top to those POSTs. Verified bots (the gate's header) are not counted. Over a limit: 429 + `Retry-After: 60` — JSON `rate_limited` for `/api`, `/img`, files and `/data` (with CORS), the script-less EN/VI page of `gate-page.ts` for app pages. Keep every `period` 60 and a unique `namespace_id` in 88xx (staging 89xx): namespaces are **account-wide** and other Workers on the account (repolens, ticktune…) use 1001–1006, so the old 1001 / 1003 shared their counters (`ratelimit.test.ts` reads `wrangler.jsonc`). Layer 1 is the zone's WAF rate-limiting rule (Free plan: one rule, IP, 10 s; `docs/operations.md` §1 step 15): only it saves the Worker request a flood costs, and only it covers static assets.
|
||||
- `img.ts`: `/img/<160|640>/<path on img.itch.zone>` — allow-listed against `data/urls.json`, edge cache → R2 `THUMBS` (`w{w}/<sha256>.webp`) → `cf.image` transform (stored once) → fallback streams the original with a 1-day cache (quota error 9422). The fallback original and a missing upstream image are also kept in `caches.default` for a day (GET only); widths are the exact strings `160` / `640` (a `Map`, no `Number()` aliases); background puts log failures. Upstream errors → controlled 502.
|
||||
- `spa.ts`: `not_found_handling` is `"none"`, so misses reach the Worker: `/assets/*`, `/data/*` (a pack from an older deploy included — `no-store`, CORS) and file-like paths get a real 404 (`no-store`); app routes get the shell with 200; unknown paths and game slugs not in the catalog get the shell with **404**; with the gate on, a visitor without a pass gets the verification page instead (checked before the shell is fetched, so never a 304). Keep its `ROUTES` in sync with `src/App.tsx` and `src/hooks/useSeoHead.ts`.
|
||||
- `gate.ts` + `gate-page.ts`: the **verification gate** (web only). App pages and `/img` need a pass: the `__Host-fig_gate` cookie (HMAC of its issue time, HttpOnly, `BOT_GATE_TTL_HOURS`, default 48) set by `POST /api/verify` after a Turnstile check (action `gate`, `RL_GATE`, same-origin); `GET /api/verify` reports `{enabled, valid, via, ttl}`. The page is self-contained (inline CSS/JS with a per-response CSP nonce, EN/VI, the site's og tags, 200 + noindex). Verified bots pass with `x-fig-known-bot: <GATE_BOT_TOKEN>` from a zone Transform Rule; clients that only claim to be a crawler get 503. `/assets`, `/data`, `/api/*`, `/admin*` and the other static files stay open. `turnstile.ts`: siteverify shared with Suggest (hostname + action checked; the test secrets skip both).
|
||||
@@ -152,6 +152,7 @@ Code **MIT** (`LICENSE`); catalog data **CC BY 4.0** for the maintainer's contri
|
||||
40. **The Python requirements are hash-locked pip-tools compiles** — `requirements.txt` ← `requirements.in`, `requirements-dev.txt` ← `requirements-dev.in` (`-c requirements.txt`) — installed everywhere with `--require-hashes --only-binary :all:`, so a new release of any dependency, indirect ones included, can't reach the jobs that push to `main`, and no sdist build script runs. After editing a `.in`, recompile with **pip-tools on Python 3.14**: `pip-compile --generate-hashes --allow-unsafe --strip-extras requirements.in`, then the same for `requirements-dev.in`. Dependabot takes the Python version from the `# … autogenerated by pip-compile with Python X.Y` header and supports ≤ 3.14 (gotcha 39); the hashes cover every file of a release, so CI's 3.15 still finds its cp315 wheels. `colorama` is listed in `requirements-dev.in` because pytest needs it on Windows only: a lock compiled on Linux (Dependabot's) would otherwise leave it out and the hash-checked local install would fail. Keep those pip commands in a block scalar (`run: |`): in a plain YAML scalar the `: ` of `:all: -r` makes the whole workflow file invalid — GitHub then records a 0-job failed push run named after the file, and a PR simply shows no check for that workflow.
|
||||
41. **SonarQube Cloud analyses in CI** (`sonarcloud.yml` + `sonar-project.properties`; project `poli0981_free-games-itchio-list`, public). Automatic Analysis is switched off on sonarcloud.io — switched back on, it makes the CI scan fail. PRs get a "SonarCloud Code Analysis" check from the SonarQube Cloud app; Dependabot and fork PRs get no `SONAR_TOKEN`, so their analysis is skipped with a notice (a Dependabot secret of that name would enable it). Tests run in the `coverage` job without the token; besides the `token` job's presence check, only the `scan` step gets it, in a job that runs no project code. Binary images and the D1 migrations (SQLite, claimed by the Oracle PL/SQL analyzer, which then warned on every run) are excluded from the analysis. Coverage: `pytest --cov` (`[tool.coverage.run]` in `pyproject.toml`) and `npm run test:coverage` (`test.coverage` in `vite.config.ts`; the lcov reporter's `projectRoot` is the repo root, so the scanner resolves the paths); `@vitest/coverage-v8` needs vitest's exact version (Dependabot group `vitest`). The built-in "Sonar way" gate fails a PR under 80 % coverage on new code (from 20 new lines up), so code without unit tests by design is in `sonar.coverage.exclusions` — keep that list in step with what the tests reach. `sonar.projectVersion` comes from `webapp/package.json`: on `main`, "new code" (previous version) starts over at each version bump. `sonar.python.version` stays 3.14 until SonarQube Cloud's analyzer supports 3.15 (it warns "No explicit support for version 3.15"). The free plan keeps only `main` and PRs: a manual run on another branch uploads a branch analysis nobody can open ("not allowed to access data from non main branches"). Mark a false positive on the issue itself. Two of its suggested fixes are wrong here: S2871 wants `localeCompare` in `.sort()`, but chunk names, migration files and `unblocked.json` keep code-unit order (= Python's `sorted()`) — pass `compareCodeUnits` (`worker/text.ts`); S8906 wants `select_one('div.a.b')` for bs4's `class_=[a, b]`, which means *either* class — write a selector list `div.a, div.b`. Trim trailing characters with `trimEnd()` (`worker/text.ts`), not `/x+$/` (S8786: quadratic backtracking on a long run). Known false positive: css:S8776 on Tailwind v4's `@utility … { &::-webkit-scrollbar … }` (the `&` is required for variants).
|
||||
42. **`overrides.undici` in `webapp/package.json` is a stopgap**: miniflare (wrangler's local runtime) pins `undici` exactly — 7.29.1 in wrangler 4.152, three advisories fixed in 7.30.1, so Dependabot could not raise it — and the override lifts it to `^7.30.1`. Only miniflare uses undici; nothing of it reaches the site, the Worker or the apps. Remove the override once `npm ls undici` without it shows ≥ 7.30.1, and before miniflare moves to undici 8 (the override would hold it on 7).
|
||||
43. **The Workers rate-limit binding counts per machine, not per data centre.** Measured on production right after the 4.5.0 deploy (CDG): 1,000 requests in 53 s, each `curl` on a new connection, against `RL_SITE` 120/60 s → **no 429 at all**; 200 requests on **one** keep-alive connection → 93 × 200, then 429s. A data centre has many machines, each counts locally and syncs late, so a client that keeps one connection (browsers over HTTP/2, scripts with keep-alive) is held to the limit, while one that opens a new connection per request is spread across machines and barely counted. The zone's WAF rule counts across the data centre: it is the limit that holds against floods. Test the Worker limits over one connection (`curl URL URL …` in one invocation, or `--next`), never with `xargs curl`; under `wrangler dev` (miniflare: one process) every request counts.
|
||||
|
||||
## Safe-edit rules
|
||||
|
||||
|
||||
@@ -28,7 +28,9 @@ requests") and has to wait a minute. Nothing changes in the desktop and Android
|
||||
(`Retry-After: 60`) and are never cached. The admin shows "Too many requests from this connection".
|
||||
- **A first layer in front of the Worker**: the maintainer's runbook (`docs/operations.md`, section 1
|
||||
step 15) sets up a Cloudflare WAF rate-limiting rule that blocks floods before they reach the site's
|
||||
code, and covers the static files the Worker never sees.
|
||||
code, and covers the static files the Worker never sees. The Worker's own counts are approximate
|
||||
(Cloudflare counts them per server and syncs them late), so the WAF rule is what holds against a
|
||||
flood that opens a new connection for every request.
|
||||
|
||||
### Fixed
|
||||
|
||||
|
||||
+19
-7
@@ -139,7 +139,9 @@ Do these once (and again with the `-staging` names for the staging Worker).
|
||||
no bot token is needed).
|
||||
15. **Rate limits** — two layers, both per client IP (an IPv6 client by its /64 in the Worker).
|
||||
- **Layer 1, the zone's WAF rule** (Security → WAF → **Rate limiting rules**). It runs before the
|
||||
Worker, so a request it blocks costs no Worker request, and it is the only limit on static files.
|
||||
Worker, so a request it blocks costs no Worker request, it is the only limit on static files, and
|
||||
it counts across the whole data centre — the only limit a flood of new connections can't spread
|
||||
out of (see layer 2).
|
||||
On the Free plan there is **one** rule, counted by IP over **10 seconds**, with a 10-second block;
|
||||
if a rule already exists (the old one for `/api/*`), edit it rather than adding one. Settings:
|
||||
- Rule name `Flood guard`. *If incoming requests match* → *Edit expression*:
|
||||
@@ -164,19 +166,29 @@ Do these once (and again with the `-staging` names for the staging Worker).
|
||||
(`/admin` + `/api/admin`); on top, `RL_SUGGEST` 5 (Suggest POSTs) and `RL_GATE` 20 (verification
|
||||
POSTs). Over a limit: 429 + `Retry-After: 60` — JSON `{"error":"rate_limited"}` for the API,
|
||||
covers and files (with CORS on `/data`), a small "Too many requests" page (EN/VI, no script) for
|
||||
app pages. Verified bots (the gate's header) are never counted. Change a limit in
|
||||
app pages. Verified bots (the gate's header) are never counted. **These counts are loose**:
|
||||
each machine of a data centre counts on its own and syncs late, so they hold for a client that
|
||||
keeps one connection (browsers use one HTTP/2 connection, scripts with keep-alive) but hardly
|
||||
for one that opens a new connection per request — measured after the 4.5.0 deploy, 1,000 such
|
||||
requests in a minute got no 429. That is the WAF rule's job. Change a limit in
|
||||
`wrangler.jsonc` (then `npm run cf:types`); keep `period` 60 and a new `namespace_id` per binding in
|
||||
88xx (staging 89xx) — the ids are account-wide, and the account's other Workers use the low numbers.
|
||||
The counts are approximate and fail open (a missing binding or a limiter error lets requests
|
||||
through).
|
||||
- **Test** after a change (it blocks your own IP for up to a minute):
|
||||
- **Test** after a change (it blocks your own IP for up to a minute). The Worker's limits, over
|
||||
**one** connection (a single `curl` with many URLs reuses it):
|
||||
```sh
|
||||
seq 150 | xargs -P 30 -I{} curl -s -o /dev/null -w "%{http_code}\n" https://freeitchgames.win/api/health | sort | uniq -c
|
||||
curl -s -o /dev/null -w "%{http_code}\n" $(printf 'https://freeitchgames.win/api/health %.0s' $(seq 200)) | sort | uniq -c
|
||||
curl -s https://freeitchgames.win/api/health # {"error":"rate_limited"} → the Worker; "error code: 1015" → the WAF rule
|
||||
```
|
||||
About 120 answer 200 and the rest 429. To see the WAF rule fire, send more than its threshold
|
||||
within 10 s (`seq 400 | xargs -P 60 …`): the extra requests get `error code: 1015` and show up in
|
||||
Security → Events. Worker 429s show up in Workers Logs (sampled, `head_sampling_rate` 0.1).
|
||||
About 120 answer 200 and the rest 429 (fewer pass if you sent requests in the last minute). The
|
||||
WAF rule, with a new connection per request, more than its threshold within 10 s:
|
||||
```sh
|
||||
seq 400 | xargs -P 60 -I{} curl -s -o /dev/null -w "%{http_code}\n" https://freeitchgames.win/api/health | sort | uniq -c
|
||||
```
|
||||
Past ~300 the answers turn into 429 `error code: 1015` (Security → Events lists them); without
|
||||
the rule all 400 answer 200. Worker 429s show up in Workers Logs (sampled,
|
||||
`head_sampling_rate` 0.1).
|
||||
|
||||
## 2. GitHub
|
||||
|
||||
|
||||
@@ -15,8 +15,12 @@
|
||||
* Suggest and the gate's POST keep their own, stricter limits on top
|
||||
* (RL_SUGGEST, RL_GATE). Verified bots (the gate's Transform Rule header) are
|
||||
* not counted, so a crawl never rate-limits itself out of the index. Counts
|
||||
* are per Cloudflare location and approximate (the binding's design). Fails
|
||||
* open: a missing binding or a failing limiter lets the request through.
|
||||
* are approximate (the binding's design): each machine of a data centre
|
||||
* counts on its own and syncs late, so a client that keeps one connection
|
||||
* (a browser, a script with keep-alive) is held to the limit, but one that
|
||||
* opens a new connection per request is barely counted — the WAF rule is the
|
||||
* limit for floods. Fails open: a missing binding or a failing limiter lets
|
||||
* the request through.
|
||||
*/
|
||||
import type { WorkerEnv } from './env'
|
||||
import { botPass } from './gate'
|
||||
|
||||
@@ -64,8 +64,10 @@
|
||||
"migrations_dir": "./worker/migrations"
|
||||
}
|
||||
],
|
||||
// Per-IP limits (an IPv6 client by its /64), counted per Cloudflare location;
|
||||
// over a limit the Worker answers 429 + Retry-After (worker/ratelimit.ts).
|
||||
// Per-IP limits (an IPv6 client by its /64), counted per machine of each
|
||||
// Cloudflare data centre (approximate: they hold for a client on one
|
||||
// connection, not for one that opens a new connection per request); over a
|
||||
// limit the Worker answers 429 + Retry-After (worker/ratelimit.ts).
|
||||
// Every request that runs the Worker counts against one of RL_SITE (app
|
||||
// pages, the public /api, the readable /data JSON, misses), RL_IMG (covers:
|
||||
// a Games page shows up to 100) or RL_ADMIN (/admin + /api/admin, behind
|
||||
|
||||
Reference in new issue
Block a user