feat(worker): per-surface rate limits behind the WAF rule

The zone's one Free-plan rate-limiting rule (IP, 10 s, block 10 s) stays
the first layer: it refuses a burst before the Worker is invoked and is the
only limit on the prerendered pages. The Worker now adds a second one with
the `ratelimits` bindings in wrangler.jsonc, 60-second windows and one
budget per surface, keyed by IPv4 address or IPv6 /64:

  RL_DATA   /api/data/*                             240/min
  RL_IMG    /img/*                                  600/min
  RL_API    /api/activity, /api/updates/desktop, …  120/min
  RL_HUMAN  /api/human-check                         20/min
  RL_ADMIN  /admin, /admin/*, /api/ingest/*         300/min

Checked after the country block and before every route and the Access
check; over a limit the answer is 429 + Retry-After: 60, no-store, with
CORS only where the route itself sends it (data, api). It fails open
(no binding, no CF-Connecting-IP, a throwing limit()) and logs no refusal.
Verified locally under wrangler dev: each budget refused exactly past its
limit, a second /64 address shared the first one's budget, pages were never
counted.

rate-limit.test.ts holds wrangler.jsonc to the code (binding names, period
60 = Retry-After, account-unique namespace ids) and checks that every
run_worker_first prefix draws on a budget. worker-configuration.d.ts gets
the five Env lines (and the IMG_MINT_PER_TICK value it had missed) without
regenerating the runtime types.

docs/SECURITY_SETUP.md section 3 describes both layers and recommends
widening the WAF rule to the whole zone except /_app/immutable/ at 250 per
10 s, counted from the build (a first visit sends 110-130 requests once
the service worker precaches); CLAUDE.md records the rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 committed 2026-10-27 06:57:27 +07:00
1 parent 277e7f14c6
commit d8d395b568
8 files changed
+616 -42

No files matched your search

+23
View File
@@ -444,6 +444,29 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
it must keep excluding). `geo.ts` FAILS OPEN on an unknown country:
`request.cf` is undefined under Vitest and under `wrangler dev` without
`--remote`, and failing closed there 403s every test.
- **Rate limiting has two layers, and the WAF rule is the first.** The zone's
one Free-plan rate-limiting rule (IP, 10 s, Block 10 s; `docs/SECURITY_SETUP.md`
§3a) refuses a burst before the Worker is invoked, and is the only limit on
the prerendered pages. `worker/lib/rate-limit.ts` is the second: the
`ratelimits` bindings in `wrangler.jsonc`, 60-second windows, one budget per
surface (`RL_DATA`, `RL_IMG`, `RL_API`, `RL_HUMAN`, `RL_ADMIN`), keyed by
IPv4 address or IPv6 /64, checked after the country block and before every
route and the Access check, answering 429 with `Retry-After: 60`. Rules:
- It FAILS OPEN (no binding, no `CF-Connecting-IP`, a throwing `limit()`).
Vitest and `admin/mock` bind no limiter, which is why no other test sees
it; the catalogue must never fail to load because of this layer.
- Counts are per Cloudflare location and approximate: a limit is an abuse
ceiling far above any client (a cold catalogue load is ~16 data requests),
never a quota to tighten.
- Refusals are not logged: Workers Logs has the 429 status, and a line each
would double a flood's log volume.
- A 429 carries CORS only where its route does (`data`, `api`), never on the
human check or the admin surface.
- Every `period` stays 60 (the Retry-After), and a `namespace_id` is unique
in the ACCOUNT - another Worker reusing one shares its counters. A new
`run_worker_first` prefix needs a class, and a new binding needs its line in
`worker-configuration.d.ts`. `rate-limit.test.ts` holds the periods, the
namespaces, the binding names and the prefix coverage.
- `audit_log` and `commit_jobs` are pruned daily after `ADMIN_RETENTION_DAYS`
(wrangler.jsonc, 180). `docs/PRIVACY_POLICY.md` states the number: change
both together. `ingest_queue` and `ingest_decisions` are never pruned — the
+130 -41
View File
@@ -221,55 +221,140 @@ becomes `"pending"`, and `"preloaded"` once it ships in a browser release.
---
## 3. Rate limiting — one rule, and it blocks
## 3. Rate limiting — two layers, the WAF rule first
Two limits, in this order:
1. **The zone's WAF rate-limiting rule** (3a) refuses a burst at the edge,
*before* the Worker runs: a flood it catches costs no Worker invocation, no
subrequest and no R2 read. It is also the only limit on the prerendered
pages, which never reach the Worker.
2. **The Worker's limits** (3b, `web/worker/lib/rate-limit.ts`) give each
surface the Worker serves its own budget over 60 seconds. They stop a
client that paces itself just under the WAF's 10-second threshold, and they
keep holding if the WAF rule is edited or deleted - the same reason
`BLOCKED_COUNTRIES` sits beside the country rule (section 4).
Neither replaces the other. The WAF rule alone cannot tell a human check from
an image, and the Worker alone has already been invoked, and billed, by the
time it says no.
### 3a. The WAF rule — one rule, and it blocks
The Free plan allows exactly **one** rate-limiting rule, and fixes most of it:
counting by IP only, a **10-second** period, a **10-second** mitigation
timeout, expression fields limited to the URI path and verified-bot status,
and cached requests count too. So the decision is not how many rules but what
the single rule covers.
and cached requests count too (no cache exclusion). A refused request gets
HTTP 429 with Cloudflare's error 1015 page. So the decision is not how many
rules but what the single rule covers.
The two public surfaces that cost money are `/img/*` (proxies Steam's CDN) and
`/api/data/*` (proxies GitHub raw). `/admin` (with its API at `/admin/api/*`)
and `/api/ingest/*` are behind Cloudflare Access already. One rule covers both proxies:
**Recommended (since 2026-10-27): the whole zone except the hashed bundles.**
Security → Security rules → Rate limiting rules → the existing rule → Edit
(the Free plan cannot add a second one):
**Security → WAF → Rate limiting rules → Create rule → Edit expression**
| Field | Value |
|---|---|
| Rule name | `site-wide` |
| Expression | `not starts_with(http.request.uri.path, "/_app/immutable/") and not cf.client.bot` |
| Characteristics | IP |
| Period | 10 seconds |
| Requests | **250** |
| Action | Block |
| Duration | 10 seconds |
Why this scope: every page, the API, the image proxy, the icons and the
service worker count, on every hostname of the zone - the rule has no
hostname field, so `media.free-steam-games.win` counts too. `/_app/immutable/*`
is left out because a cold first visit fetches about 115 of those files at
once (the home page alone preloads 53), the browser then keeps them for a
year, and the asset server serves them at no cost. `not cf.client.bot` keeps
verified search crawlers, which read the 10,000 prerendered game pages, out of
the count.
Why 250 per 10 seconds, counted from the build rather than a template: right
after consent and the human check, a first visit sends the page and its icons
(about 6 requests), the service worker's precache of 56 non-hashed URLs (41
pages, 8 `__data.json`, 6 images and the manifest), the catalogue (about 16:
`index.json`, 14 shards, the removed-games list), `/api/activity`,
`/api/human-check` and the dashboard's thumbnails - 110 to 130 requests within
a few seconds. 150 left no room for a second person behind the same address
(a shared office, or a mobile carrier's CGNAT); 250 does. A script faster than
25 requests a second is blocked for 10 seconds, again and again; one slower
than that meets the Worker's per-minute budgets on the paths that cost money.
Block is tolerable because the timeout is fixed at 10 seconds: a false
positive costs a real visitor ten seconds of missing thumbnails, not a
lockout.
**The narrower rule it replaces** is still a valid choice if pages should not
count at all - the Worker limits cover `/api/*` either way:
| Field | Value |
|---|---|
| Rule name | `public-proxies` |
| Expression | `(starts_with(http.request.uri.path, "/img/") or starts_with(http.request.uri.path, "/api/data/")) and not cf.client.bot` |
| Characteristics | IP |
| Period | 10 seconds |
| Requests | **150** |
| Action | Block |
| Duration | 10 seconds |
| Requests | **150** per 10 seconds, IP, Block for 10 seconds |
Why 150 per 10 seconds, from a measured session rather than a template:
opening `/games` fetches `index.json`, 5 shards and roughly 30 thumbnails in
the first couple of seconds, and fast-scrolling the virtualised table adds a
few dozen more. Someone browsing hard stays under ~100 in any 10-second window;
150 leaves headroom for a shared office or CGNAT address that looks like one
client. A scraper pulling faster than 15 requests a second is blocked for 10
seconds, again and again.
Block is tolerable here because the Free plan's timeout is fixed at 10 seconds:
a false positive costs a real visitor ten seconds of missing thumbnails, not a
lockout. `not cf.client.bot` keeps verified search crawlers out of the count.
**Use the WAF rule, not a Worker-side limiter.** The WAF rejects at the edge
*before* the Worker runs, so an abusive burst costs no Worker invocations. A
limiter inside the Worker has already paid for the request by the time it says
no — backwards for cost control.
**Check** (this will get your own IP blocked for 10 seconds, which is the point):
**Check** (blocks your own IP for 10 seconds, which is the point; a static
file, so no Worker invocation is spent):
```bash
for i in $(seq 1 200); do curl -s -o /dev/null -w '%{http_code}\n' https://free-steam-games.win/api/data/data/index.json; done | sort | uniq -c
seq 1 300 | xargs -P 30 -I{} curl -s -o /dev/null -w '%{http_code}\n' https://free-steam-games.win/robots.txt | sort | uniq -c
```
Expected: about 150 × `200`, then `429`s. Watch **Security → Events** for a
week before changing the threshold.
Expected: about 250 × `200`, then `429`s. With `public-proxies` instead, a
static file is never counted: use `/api/data/data/index.json` and expect about
150. Watch **Security → Events** for a week before changing the threshold.
### 3b. The Worker limits — one budget per surface
`ratelimits` bindings in `web/wrangler.jsonc`, checked by
`web/worker/lib/rate-limit.ts` after the country block and before every route,
the admin surface's Access check included:
| Binding | Paths | Requests per IP per minute |
|---|---|---|
| `RL_DATA` | `/api/data/*` | 240 |
| `RL_IMG` | `/img/*` | 600 |
| `RL_API` | `/api/activity`, `/api/updates/desktop`, any other public `/api/*` | 120 |
| `RL_HUMAN` | `/api/human-check` | 20 |
| `RL_ADMIN` | `/admin`, `/admin/*`, `/api/ingest/*` | 300 |
- **Key:** the client's IPv4 address, or its IPv6 /64 (one subscriber usually
holds a whole /64). Counted per Cloudflare location and approximately - a
few requests over the limit can pass - so these are ceilings for abuse,
never quotas. Each sits well above what a person, the apps or the discovery
workflow sends: a cold catalogue load is about 16 data requests, a person
sends one human check a day, discovery a few ingest calls a run.
- **Refusal:** HTTP 429, `{"error":"too many requests"}`, `Retry-After: 60`,
`Cache-Control: no-store`. `/api/data/*` and the public API also carry
`Access-Control-Allow-Origin: *`, so the packaged apps see the 429 instead
of a CORS failure; the human check and the admin surface carry none (they
are same-origin by design).
- **Clients:** a 429 on a pack makes the site and the apps try the plain shard
once, then report a failed load; the human check lets the reader through
(it refuses only on Cloudflare's verdict); discovery stops its run with the
status.
- **Fails open:** no binding, no `CF-Connecting-IP`, or a limiter that throws
lets the request through. The WAF rule is the boundary; this must never be
why the catalogue does not load.
- **Not logged:** Workers Logs already records every invocation with its
status - filter on 429. A console line per refusal would double the log
volume of exactly the floods this is for. The bindings do not appear in the
dashboard.
- **Changing a limit** is a pull request to `web/wrangler.jsonc` (it deploys
with the Worker). Keep every `period` at 60, which is what `Retry-After`
promises (`rate-limit.test.ts` checks), and keep each `namespace_id` unique
within the Cloudflare account: two bindings with one ID share their
counters, across Workers too.
**Check** (after a deploy; spends 130 cheap invocations):
```bash
seq 1 130 | xargs -P 10 -I{} curl -s -o /dev/null -w '%{http_code}\n' https://free-steam-games.win/api/no-such-route | sort | uniq -c
```
Expected: about 120 × `404`, then `429` for the rest of that minute.
---
@@ -823,10 +908,11 @@ Retry.
from `_headers` and may log a harmless message from inside
`challenges.cloudflare.com`; that is not a violation of this site's policy.
**Rate limit (optional, no extra rule).** Each POST costs one siteverify
subrequest. The `public-proxies` expression in section 3 can take
`or http.request.uri.path eq "/api/human-check"`; a person sends a few a day,
far below 150 per 10 seconds.
**Rate limit.** Each POST costs one siteverify subrequest, so the route has
its own Worker budget, `RL_HUMAN`: 20 per IP per minute (section 3b), where a
person sends one a day. A refused check lets the reader through like any other
answer that is not a verdict. The `site-wide` WAF rule (section 3a) counts it
too; the narrower `public-proxies` rule does not.
**Local testing.** `npm run dev` skips the check: its `/api` proxy points at
production, which refuses a localhost token. `npm run preview` proxies the same
@@ -858,7 +944,10 @@ pruned daily after `ADMIN_RETENTION_DAYS` (180), which
`docs/PRIVACY_POLICY.md` states.
**The Free-plan rule budget is now the constraint to watch:** the one rate
limiting rule is used (section 3), and two of the five custom rules are
(section 4). A third custom rule is affordable; a second rate-limiting rule is
not, without upgrading. Configuration Rules are a separate quota: one of ten
is used (section 10).
limiting rule is used (section 3a), and four of the five custom rules are
(as of 2026-10-27: `human-check-pages`, `blocked-countries`,
`media-host-paths`, and Cloudflare's AI Crawl Control rule). A fifth custom
rule is affordable; a second rate-limiting rule is not, without upgrading -
which is why per-surface limits live in the Worker (section 3b), where they
cost no rule. Configuration Rules are a separate quota: one of ten is used
(section 10).
+6 -1
View File
@@ -4,12 +4,17 @@
interface __BaseEnv_Env {
MEDIA: R2Bucket;
DB: D1Database;
RL_DATA: RateLimit;
RL_IMG: RateLimit;
RL_API: RateLimit;
RL_HUMAN: RateLimit;
RL_ADMIN: RateLimit;
IMAGES: ImagesBinding;
ASSETS: Fetcher;
SITE_ORIGIN: "https://free-steam-games.win";
IMG_TRANSFORM: "true";
IMG_TRANSFORM_MONTHLY_CAP: "20000";
IMG_MINT_PER_TICK: "10";
IMG_MINT_PER_TICK: "40";
ADMIN_RETENTION_DAYS: "180";
BLOCKED_COUNTRIES: "CN,RU,AR";
ACCESS_TEAM_DOMAIN: "polished-hall-48fd.cloudflareaccess.com";
+60
View File
@@ -138,6 +138,66 @@ describe("the country block", () => {
});
});
describe("the rate limits", () => {
/** makeEnv() binds no limiter, which is why every other test is unaffected
* (lib/rate-limit.ts fails open). Here each one allows two requests. */
function limitedEnv(extra: Record<string, unknown> = {}) {
const calls: string[] = [];
const counts = new Map<string, number>();
const limiter = (binding: string) => ({
async limit({ key }: { key: string }) {
calls.push(binding);
const n = (counts.get(binding + key) ?? 0) + 1;
counts.set(binding + key, n);
return { success: n <= 2 };
},
});
const bindings = Object.fromEntries(
["RL_DATA", "RL_IMG", "RL_API", "RL_HUMAN", "RL_ADMIN"].map((b) => [b, limiter(b)]),
);
return { env: { ...makeEnv(), ...bindings, ...extra } as unknown as Env, calls };
}
const hit = (limited: Env, path: string, headers: Record<string, string> = {}) =>
worker.fetch(
new Request(`https://free-steam-games.win${path}`, { headers: { "CF-Connecting-IP": "203.0.113.7", ...headers } }),
limited,
ctx,
);
it("answers 429 once a client spends its budget, before the route runs", async () => {
const { env: limited } = limitedEnv();
expect((await hit(limited, "/api/no-such-route")).status).toBe(404);
expect((await hit(limited, "/api/no-such-route")).status).toBe(404);
const res = await hit(limited, "/api/no-such-route");
expect(res.status).toBe(429);
expect(res.headers.get("Retry-After")).toBe("60");
expect(res.headers.get("Access-Control-Allow-Origin")).toBe("*");
});
it("limits the admin surface ahead of the Access check", async () => {
const { env: limited } = limitedEnv();
await hit(limited, "/admin/api/me");
await hit(limited, "/admin/api/me");
const res = await hit(limited, "/admin/api/me", { "Cf-Access-Jwt-Assertion": await token() });
expect(res.status).toBe(429);
// Same-origin by design: no CORS on the admin API, refusal included.
expect(res.headers.get("Access-Control-Allow-Origin")).toBeNull();
});
it("gives the human check its own budget", async () => {
const { env: limited, calls } = limitedEnv();
await hit(limited, "/api/human-check");
expect(calls).toEqual(["RL_HUMAN"]);
});
it("does not count a request the country block already refused", async () => {
const { env: limited, calls } = limitedEnv({ BLOCKED_COUNTRIES: "CN" });
expect((await hit(limited, "/api/activity", { "CF-IPCountry": "CN" })).status).toBe(403);
expect(calls).toEqual([]);
});
});
describe("the human-check route", () => {
// Through the real router: a handler test cannot notice the route being
// shadowed by the /api/* 404 or pulled behind the Access block.
+8
View File
@@ -28,6 +28,7 @@ import { mintImages } from "./lib/img-mint";
import { defaultAdminDeps } from "./lib/deps";
import { isPruneTick, pruneAdminHistory } from "./lib/prune";
import { isBlockedCountry } from "./lib/geo";
import { rateLimited } from "./lib/rate-limit";
export default {
async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
@@ -158,6 +159,13 @@ async function route(
);
}
// Per-IP limits, after the country block so a refused request is not also
// counted, and before the admin block for the same reason as that one: one
// place, ahead of every route (lib/rate-limit.ts). The WAF rate-limiting
// rule runs before the Worker and is the first layer; this is the second.
const limited = await rateLimited(request, pathname, env);
if (limited) return limited;
if (pathname.startsWith("/api/data/")) {
return handleData(request, url, ctx);
}
+223
View File
@@ -0,0 +1,223 @@
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { describe, expect, it, vi } from "vitest";
import {
LIMIT_BINDINGS,
LIMIT_PERIOD_SECONDS,
clientKey,
limitClass,
rateLimited,
tooManyRequests,
} from "./rate-limit";
/** A limiter that counts exactly, unlike Cloudflare's, so tests can be exact. */
function fakeLimiter(limit: number) {
const counts = new Map<string, number>();
const keys: string[] = [];
return {
keys,
async limit({ key }: { key: string }) {
keys.push(key);
const n = (counts.get(key) ?? 0) + 1;
counts.set(key, n);
return { success: n <= limit };
},
};
}
const envWith = (bindings: Record<string, unknown>) => bindings as unknown as Env;
const from = (ip: string | null, path = "/api/data/data/index.json") =>
new Request(`https://free-steam-games.win${path}`, { headers: ip ? { "CF-Connecting-IP": ip } : {} });
/**
* wrangler.jsonc, comments and trailing commas removed. Comments are only
* stripped outside strings: the file's URLs carry `//` inside quotes.
*/
function readWranglerConfig(): Record<string, unknown> {
const text = readFileSync(join(__dirname, "..", "..", "wrangler.jsonc"), "utf8");
let out = "";
for (let i = 0; i < text.length; i++) {
const c = text[i];
if (c === '"') {
let j = i + 1;
while (j < text.length && text[j] !== '"') j += text[j] === "\\" ? 2 : 1;
out += text.slice(i, j + 1);
i = j;
} else if (c === "/" && text[i + 1] === "/") {
while (i < text.length && text[i] !== "\n") i++;
out += "\n";
} else if (c === "/" && text[i + 1] === "*") {
i = text.indexOf("*/", i + 2) + 1;
} else {
out += c;
}
}
return JSON.parse(out.replace(/,(\s*[}\]])/g, "$1")) as Record<string, unknown>;
}
describe("limitClass", () => {
it("gives each surface the Worker serves its own budget", () => {
expect(limitClass("/api/data/data/index.json")).toBe("data");
expect(limitClass("/api/data/p1/data_001.bin")).toBe("data");
expect(limitClass("/img/s/730.jpg")).toBe("img");
expect(limitClass("/api/human-check")).toBe("human");
expect(limitClass("/api/activity")).toBe("api");
expect(limitClass("/api/updates/desktop")).toBe("api");
expect(limitClass("/api/no-such-route")).toBe("api");
});
it("puts the Access-protected surfaces in one generous budget", () => {
for (const path of ["/admin", "/admin/queue", "/admin/assets/app-1.js", "/admin/api/me", "/api/ingest/known"]) {
expect(limitClass(path), path).toBe("admin");
}
});
it("covers nothing the Worker never sees", () => {
expect(limitClass("/")).toBeNull();
expect(limitClass("/games/730")).toBeNull();
expect(limitClass("/_app/immutable/entry/start.js")).toBeNull();
expect(limitClass("/administrator")).toBeNull();
});
it("covers every prefix the Worker runs first for", () => {
// A prefix added to run_worker_first without a budget would be the one
// surface a flood could reach unlimited.
const assets = readWranglerConfig().assets as { run_worker_first: string[] };
for (const prefix of assets.run_worker_first) {
const sample = prefix.replace(/\*$/, "anything");
expect(limitClass(sample), prefix).not.toBeNull();
}
});
});
describe("clientKey", () => {
it("is the IPv4 address", () => {
expect(clientKey(from("203.0.113.7"))).toBe("203.0.113.7");
});
it("is the /64 of an IPv6 address, however it is written", () => {
const key = "2001:db8:a:b::/64";
expect(clientKey(from("2001:db8:a:b:1:2:3:4"))).toBe(key);
expect(clientKey(from("2001:0DB8:000A:000B:ffff:0:0:1"))).toBe(key);
expect(clientKey(from("2001:db8:a:b::1"))).toBe(key);
expect(clientKey(from("2001:db8::1"))).toBe("2001:db8:0:0::/64");
expect(clientKey(from("::1"))).toBe("0:0:0:0::/64");
});
it("unwraps an IPv4-mapped address instead of filing it under ::/64", () => {
// Every mapped address shares its first four groups (all zero), so taking
// the /64 would put every such client on one counter.
expect(clientKey(from("::FFFF:198.51.100.4"))).toBe("198.51.100.4");
});
it("falls back to the raw value for anything it cannot parse", () => {
expect(clientKey(from("1:2:3"))).toBe("1:2:3");
expect(clientKey(from("1::2::3"))).toBe("1::2::3");
});
it("is null without the header (Vitest, a direct origin request)", () => {
expect(clientKey(from(null))).toBeNull();
expect(clientKey(from(" "))).toBeNull();
});
});
describe("tooManyRequests", () => {
it("is a 429 that says when to come back and is never cached", async () => {
const res = tooManyRequests("img");
expect(res.status).toBe(429);
expect(res.headers.get("Retry-After")).toBe(String(LIMIT_PERIOD_SECONDS));
expect(res.headers.get("Cache-Control")).toBe("no-store");
expect(res.headers.get("X-Content-Type-Options")).toBe("nosniff");
expect(await res.json()).toEqual({ error: "too many requests" });
});
it("answers cross-origin only where the route itself does", () => {
// The apps fetch /api/data/* and /api/activity from tauri://localhost;
// without the header they would see a CORS error, not the 429.
expect(tooManyRequests("data").headers.get("Access-Control-Allow-Origin")).toBe("*");
expect(tooManyRequests("api").headers.get("Access-Control-Allow-Origin")).toBe("*");
for (const cls of ["img", "human", "admin"] as const) {
expect(tooManyRequests(cls).headers.get("Access-Control-Allow-Origin"), cls).toBeNull();
}
});
});
describe("rateLimited", () => {
it("lets a client through up to the limit, then refuses it", async () => {
const env = envWith({ RL_DATA: fakeLimiter(2) });
expect(await rateLimited(from("203.0.113.7"), "/api/data/x", env)).toBeNull();
expect(await rateLimited(from("203.0.113.7"), "/api/data/x", env)).toBeNull();
expect((await rateLimited(from("203.0.113.7"), "/api/data/x", env))?.status).toBe(429);
// Another address has its own budget.
expect(await rateLimited(from("203.0.113.8"), "/api/data/x", env)).toBeNull();
});
it("charges one IPv6 /64 as one client", async () => {
const env = envWith({ RL_DATA: fakeLimiter(1) });
expect(await rateLimited(from("2001:db8:a:b::1"), "/api/data/x", env)).toBeNull();
expect((await rateLimited(from("2001:db8:a:b::2"), "/api/data/x", env))?.status).toBe(429);
});
it("keeps each surface's budget apart", async () => {
const data = fakeLimiter(1);
const img = fakeLimiter(1);
const env = envWith({ RL_DATA: data, RL_IMG: img });
expect(await rateLimited(from("203.0.113.7"), "/api/data/x", env)).toBeNull();
expect(await rateLimited(from("203.0.113.7"), "/img/s/730.jpg", env)).toBeNull();
expect(data.keys).toEqual(["203.0.113.7"]);
expect(img.keys).toEqual(["203.0.113.7"]);
});
it("fails open without a binding, without an address, or when the limiter throws", async () => {
expect(await rateLimited(from("203.0.113.7"), "/api/data/x", envWith({}))).toBeNull();
const unused = fakeLimiter(0);
expect(await rateLimited(from(null), "/api/data/x", envWith({ RL_DATA: unused }))).toBeNull();
expect(unused.keys).toEqual([]);
const error = vi.spyOn(console, "error").mockImplementation(() => {});
const broken = envWith({ RL_DATA: { limit: () => Promise.reject(new Error("binding gone")) } });
expect(await rateLimited(from("203.0.113.7"), "/api/data/x", broken)).toBeNull();
expect(error).toHaveBeenCalledOnce();
error.mockRestore();
});
it("does not touch a limiter for a path no class covers", async () => {
const limiter = fakeLimiter(0);
const env = envWith(Object.fromEntries(Object.values(LIMIT_BINDINGS).map((b) => [b, limiter])));
expect(await rateLimited(from("203.0.113.7"), "/games/730", env)).toBeNull();
expect(limiter.keys).toEqual([]);
});
});
describe("wrangler.jsonc ratelimits", () => {
const declared = readWranglerConfig().ratelimits as {
name: string;
namespace_id: string;
simple: { limit: number; period: number };
}[];
it("declares exactly the bindings the Worker reads", () => {
expect(declared.map((r) => r.name).sort()).toEqual(Object.values(LIMIT_BINDINGS).sort());
});
it("uses the period Retry-After promises", () => {
for (const r of declared) expect(r.simple.period, r.name).toBe(LIMIT_PERIOD_SECONDS);
});
it("gives each binding its own namespace, or two budgets would share counters", () => {
const ids = declared.map((r) => r.namespace_id);
expect(new Set(ids).size).toBe(ids.length);
for (const id of ids) expect(id).toMatch(/^[1-9]\d*$/);
});
it("sets limits a person, the apps and the discovery workflow stay well under", () => {
for (const r of declared) {
expect(Number.isInteger(r.simple.limit), r.name).toBe(true);
// A cold catalogue load is ~16 /api/data requests; anything under a few
// loads a minute would refuse a household behind one address.
expect(r.simple.limit, r.name).toBeGreaterThanOrEqual(r.name === "RL_HUMAN" ? 10 : 100);
}
});
});
+139
View File
@@ -0,0 +1,139 @@
/**
* Per-IP request limits: the Worker's half of rate limiting.
*
* The FIRST layer is the zone's WAF rate-limiting rule (docs/SECURITY_SETUP.md
* section 3). It refuses a burst at the edge before this Worker is invoked, so
* a flood it catches costs no invocation, no subrequest and no R2 read. The
* Free plan gives the zone exactly one such rule and fixes its shape: count by
* IP over 10 seconds, block for 10. This layer adds what that rule cannot - a
* 60-second window, so a client pacing itself just under the 10-second
* threshold is still stopped, and a budget per surface, because a human check
* is not an image - and it keeps holding if the WAF rule is edited or deleted.
*
* Like BLOCKED_COUNTRIES (lib/geo.ts) it sees only the four
* `run_worker_first` prefixes. The prerendered pages never reach the Worker;
* the WAF rule is all that limits them.
*
* The limits are `ratelimits` bindings in wrangler.jsonc, one per class below.
* Cloudflare counts per location and approximately (a few requests over the
* limit can pass), so a limit here is a ceiling for abuse, never an accurate
* quota - every number sits well above what a person, the apps or the
* discovery workflow sends.
*
* FAIL OPEN. A missing binding (Vitest, `admin/mock`, a config that lost one),
* a request without CF-Connecting-IP, or a limit() call that throws lets the
* request through: the WAF rule is the boundary, and this must never be the
* reason the catalogue does not load.
*
* Refusals are not logged. Workers Logs already records every invocation with
* its status, and a console line per refused request would double the log
* volume of exactly the floods this exists for. Filter on status 429.
*/
import { HUMAN_CHECK_PATH } from "../../shared/human-check";
import { ADMIN_API_PREFIX } from "../../shared/admin-routes";
import { errorText, jsonError } from "./http";
export type LimitClass = "data" | "img" | "api" | "human" | "admin";
/** The wrangler.jsonc `ratelimits` binding behind each class. */
export const LIMIT_BINDINGS = {
data: "RL_DATA",
img: "RL_IMG",
api: "RL_API",
human: "RL_HUMAN",
admin: "RL_ADMIN",
} as const satisfies Record<LimitClass, keyof Env>;
/**
* Every binding's `period`, and so the Retry-After of a refusal. The binding
* cannot report when its window ends, so this is the honest upper bound.
* rate-limit.test.ts holds wrangler.jsonc to it.
*/
export const LIMIT_PERIOD_SECONDS = 60;
/**
* The classes whose routes answer cross-origin (`/api/data/*`, `/api/activity`),
* so their refusal does too: without the header the packaged apps would see a
* CORS failure where there is a 429. Never `human` or `admin` - those routes
* are same-origin by design (CLAUDE.md), and `/img/*` sends no CORS at all.
*/
const CORS_CLASSES: ReadonlySet<LimitClass> = new Set(["data", "api"]);
/** Which budget a path draws on, or null for a path no limit covers. */
export function limitClass(pathname: string): LimitClass | null {
if (pathname.startsWith("/api/data/")) return "data";
if (pathname.startsWith("/img/")) return "img";
if (pathname === HUMAN_CHECK_PATH) return "human";
if (
pathname.startsWith("/api/ingest/") ||
pathname.startsWith(ADMIN_API_PREFIX) ||
pathname === "/admin" ||
pathname.startsWith("/admin/")
) {
return "admin";
}
if (pathname.startsWith("/api/")) return "api";
return null;
}
/**
* The counter a request is charged to: its IPv4 address, or its IPv6 /64.
*
* An IPv6 subscriber is normally handed a whole /64, so counting full
* addresses would give one client 2^64 fresh budgets. A NAT'd IPv4 address
* can stand for many people (mobile carriers use CGNAT widely), which is why
* the limits are generous rather than why the key is different.
*/
export function clientKey(request: Request): string | null {
const ip = request.headers.get("CF-Connecting-IP")?.trim().toLowerCase();
if (!ip) return null;
if (!ip.includes(":")) return ip;
const mapped = /^::ffff:(\d{1,3}(?:\.\d{1,3}){3})$/.exec(ip);
if (mapped) return mapped[1];
return ipv6Prefix64(ip) ?? ip;
}
/** `2001:db8:a:b:1:2:3:4` -> `2001:db8:a:b::/64`; null if it is not IPv6. */
function ipv6Prefix64(ip: string): string | null {
const halves = ip.split("::");
if (halves.length > 2) return null;
const groups = (s: string | undefined) => (s ? s.split(":") : []);
// A trailing dotted quad (`::ffff:1.2.3.4`, `64:ff9b::1.2.3.4`) is two groups.
const width = (gs: string[]) => gs.reduce((n, g) => n + (g.includes(".") ? 2 : 1), 0);
const left = groups(halves[0]);
const right = groups(halves[1]);
const missing = 8 - width(left) - width(right);
if (halves.length === 1 ? missing !== 0 : missing < 1) return null;
const all = [...left, ...Array<string>(halves.length === 1 ? 0 : missing).fill("0"), ...right];
const prefix = all.slice(0, 4);
if (!prefix.every((g) => /^[0-9a-f]{1,4}$/.test(g))) return null;
return `${prefix.map((g) => g.replace(/^0+(?=.)/, "")).join(":")}::/64`;
}
/** The refusal: JSON like every other Worker error, never cached. */
export function tooManyRequests(cls: LimitClass): Response {
const res = jsonError(429, "too many requests");
res.headers.set("Retry-After", String(LIMIT_PERIOD_SECONDS));
if (CORS_CLASSES.has(cls)) res.headers.set("Access-Control-Allow-Origin", "*");
return res;
}
/**
* Charge the request to its class's budget. Returns the 429 to send when the
* budget is spent, or null to carry on (including every fail-open case).
*/
export async function rateLimited(request: Request, pathname: string, env: Env): Promise<Response | null> {
const cls = limitClass(pathname);
if (!cls) return null;
// Typed as always bound; at runtime it is not, under Vitest and admin/mock.
const limiter: RateLimit | undefined = env[LIMIT_BINDINGS[cls]];
const key = clientKey(request);
if (!limiter || !key) return null;
try {
const { success } = await limiter.limit({ key });
return success ? null : tooManyRequests(cls);
} catch (err) {
console.error("rate-limit: limiter failed, letting the request through", { cls, message: errorText(err) });
return null;
}
}
+27
View File
@@ -94,6 +94,33 @@
"binding": "IMAGES"
},
// Per-IP request limits, the Worker's half of rate limiting
// (worker/lib/rate-limit.ts). The zone's WAF rate-limiting rule is the
// first layer and refuses a burst before this Worker runs at all
// (docs/SECURITY_SETUP.md section 3); the Free plan fixes it at 10 seconds,
// one rule. These add a 60-second window and one budget per surface, and
// still hold if that rule is edited or deleted. They see only the four
// `run_worker_first` prefixes - the prerendered pages are the WAF's alone.
//
// Counts are per Cloudflare location and approximate; the key is the
// client's IPv4 address or IPv6 /64. Over a limit the Worker answers 429
// with Retry-After: 60, so every "period" stays 60 (rate-limit.test.ts
// holds that). A namespace_id must be unique within the Cloudflare account:
// two bindings that share one share counters, across Workers too.
//
// RL_DATA /api/data/* a cold catalogue load is ~16 requests
// RL_IMG /img/* the apps fetch every thumbnail here
// RL_API /api/activity, /api/updates/desktop, any other public /api/*
// RL_HUMAN /api/human-check a person sends one a day
// RL_ADMIN /admin, /admin/*, /api/ingest/* (behind Access as well)
"ratelimits": [
{ "name": "RL_DATA", "namespace_id": "730001", "simple": { "limit": 240, "period": 60 } },
{ "name": "RL_IMG", "namespace_id": "730002", "simple": { "limit": 600, "period": 60 } },
{ "name": "RL_API", "namespace_id": "730003", "simple": { "limit": 120, "period": 60 } },
{ "name": "RL_HUMAN", "namespace_id": "730004", "simple": { "limit": 20, "period": 60 } },
{ "name": "RL_ADMIN", "namespace_id": "730005", "simple": { "limit": 300, "period": 60 } }
],
// Reconcile approvals against the published dataset. Every 15 minutes rather
// than hourly because the gap between approving and seeing it land is the
// one moment a reviewer is actually watching; a tick with nothing approved