Merge pull request #223 from poli0981/feat/worker-rate-limits
v4.5.1 "Speed Bump": per-surface rate limits in the Worker
This commit is contained in:
9 files changed
+656
-42
No files matched your search
@@ -4,6 +4,46 @@ All notable changes to this awesome noob repo will be documented here.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [v4.5.1] – 2026-10-27 (The "Speed Bump" Edition)
|
||||
|
||||
The site's Worker now limits how fast one address can call it, behind
|
||||
Cloudflare's own rate-limiting rule at the edge. Nothing changes for readers,
|
||||
the apps or the pipeline: every limit sits far above what they send. The
|
||||
desktop and Android apps stay at `2.3.0`, with no new build. Repo
|
||||
public-facing version `4.5.0` → `4.5.1`.
|
||||
|
||||
### 🔐 Security
|
||||
|
||||
- **Rate limits in the Worker, one budget per surface.** Per address and per
|
||||
minute: the catalogue API 240 requests (loading the whole catalogue takes
|
||||
about 16), the image proxy 600, the other public API routes 120, the human
|
||||
check 20 (a person sends one a day) and the admin surface 300. Past a limit
|
||||
the Worker answers `429 Too Many Requests` with `Retry-After: 60` until the
|
||||
minute is up; the apps can read that answer, where they would otherwise have
|
||||
seen a network error. An IPv6 address counts by its /64, the block one
|
||||
subscriber is normally given. Cloudflare counts per data centre and
|
||||
approximately, so these are ceilings for abuse, never quotas. The limits are
|
||||
bindings in `web/wrangler.jsonc` and change by pull request.
|
||||
- **Two layers, the edge first.** The zone's rate-limiting rule still refuses
|
||||
a burst before the Worker runs, at no cost, and is the only limit on the
|
||||
pages themselves; the Worker adds the one-minute window the Free plan's
|
||||
10-second rule cannot, and keeps holding if that rule is changed. If the
|
||||
Worker's limiter is ever unavailable, requests go through rather than fail.
|
||||
- The recommended edge rule now covers the whole site except the hashed
|
||||
JavaScript and CSS bundles, at 250 requests per 10 seconds per address. That
|
||||
is about twice what a first visit sends once the offline cache fills in
|
||||
(counted from the build: 110 to 130 requests in a few seconds). Applying it
|
||||
is a Cloudflare dashboard change, described in `docs/SECURITY_SETUP.md` §3a.
|
||||
|
||||
### 📚 Docs
|
||||
|
||||
- `docs/SECURITY_SETUP.md` §3 describes both layers, how to check each one
|
||||
with `curl`, and the Free-plan rule budget as it stands (one of one rate
|
||||
limiting rule, four of five custom rules).
|
||||
- `CLAUDE.md` records the layer's rules: it fails open, is not logged, keeps a
|
||||
60-second period and an account-unique `namespace_id` per binding, and needs
|
||||
a budget for any prefix the Worker runs first for.
|
||||
|
||||
## [v4.5.0] – 2026-10-26 (The "Groundwork" Edition)
|
||||
|
||||
The site and the apps move to SvelteKit 3, the framework every page is
|
||||
|
||||
@@ -444,6 +444,29 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
it must keep excluding). `geo.ts` FAILS OPEN on an unknown country:
|
||||
`request.cf` is undefined under Vitest and under `wrangler dev` without
|
||||
`--remote`, and failing closed there 403s every test.
|
||||
- **Rate limiting has two layers, and the WAF rule is the first.** The zone's
|
||||
one Free-plan rate-limiting rule (IP, 10 s, Block 10 s; `docs/SECURITY_SETUP.md`
|
||||
§3a) refuses a burst before the Worker is invoked, and is the only limit on
|
||||
the prerendered pages. `worker/lib/rate-limit.ts` is the second: the
|
||||
`ratelimits` bindings in `wrangler.jsonc`, 60-second windows, one budget per
|
||||
surface (`RL_DATA`, `RL_IMG`, `RL_API`, `RL_HUMAN`, `RL_ADMIN`), keyed by
|
||||
IPv4 address or IPv6 /64, checked after the country block and before every
|
||||
route and the Access check, answering 429 with `Retry-After: 60`. Rules:
|
||||
- It FAILS OPEN (no binding, no `CF-Connecting-IP`, a throwing `limit()`).
|
||||
Vitest and `admin/mock` bind no limiter, which is why no other test sees
|
||||
it; the catalogue must never fail to load because of this layer.
|
||||
- Counts are per Cloudflare location and approximate: a limit is an abuse
|
||||
ceiling far above any client (a cold catalogue load is ~16 data requests),
|
||||
never a quota to tighten.
|
||||
- Refusals are not logged: Workers Logs has the 429 status, and a line each
|
||||
would double a flood's log volume.
|
||||
- A 429 carries CORS only where its route does (`data`, `api`), never on the
|
||||
human check or the admin surface.
|
||||
- Every `period` stays 60 (the Retry-After), and a `namespace_id` is unique
|
||||
in the ACCOUNT - another Worker reusing one shares its counters. A new
|
||||
`run_worker_first` prefix needs a class, and a new binding needs its line in
|
||||
`worker-configuration.d.ts`. `rate-limit.test.ts` holds the periods, the
|
||||
namespaces, the binding names and the prefix coverage.
|
||||
- `audit_log` and `commit_jobs` are pruned daily after `ADMIN_RETENTION_DAYS`
|
||||
(wrangler.jsonc, 180). `docs/PRIVACY_POLICY.md` states the number: change
|
||||
both together. `ingest_queue` and `ingest_decisions` are never pruned — the
|
||||
|
||||
+130
-41
@@ -221,55 +221,140 @@ becomes `"pending"`, and `"preloaded"` once it ships in a browser release.
|
||||
|
||||
---
|
||||
|
||||
## 3. Rate limiting — one rule, and it blocks
|
||||
## 3. Rate limiting — two layers, the WAF rule first
|
||||
|
||||
Two limits, in this order:
|
||||
|
||||
1. **The zone's WAF rate-limiting rule** (3a) refuses a burst at the edge,
|
||||
*before* the Worker runs: a flood it catches costs no Worker invocation, no
|
||||
subrequest and no R2 read. It is also the only limit on the prerendered
|
||||
pages, which never reach the Worker.
|
||||
2. **The Worker's limits** (3b, `web/worker/lib/rate-limit.ts`) give each
|
||||
surface the Worker serves its own budget over 60 seconds. They stop a
|
||||
client that paces itself just under the WAF's 10-second threshold, and they
|
||||
keep holding if the WAF rule is edited or deleted - the same reason
|
||||
`BLOCKED_COUNTRIES` sits beside the country rule (section 4).
|
||||
|
||||
Neither replaces the other. The WAF rule alone cannot tell a human check from
|
||||
an image, and the Worker alone has already been invoked, and billed, by the
|
||||
time it says no.
|
||||
|
||||
### 3a. The WAF rule — one rule, and it blocks
|
||||
|
||||
The Free plan allows exactly **one** rate-limiting rule, and fixes most of it:
|
||||
counting by IP only, a **10-second** period, a **10-second** mitigation
|
||||
timeout, expression fields limited to the URI path and verified-bot status,
|
||||
and cached requests count too. So the decision is not how many rules but what
|
||||
the single rule covers.
|
||||
and cached requests count too (no cache exclusion). A refused request gets
|
||||
HTTP 429 with Cloudflare's error 1015 page. So the decision is not how many
|
||||
rules but what the single rule covers.
|
||||
|
||||
The two public surfaces that cost money are `/img/*` (proxies Steam's CDN) and
|
||||
`/api/data/*` (proxies GitHub raw). `/admin` (with its API at `/admin/api/*`)
|
||||
and `/api/ingest/*` are behind Cloudflare Access already. One rule covers both proxies:
|
||||
**Recommended (since 2026-10-27): the whole zone except the hashed bundles.**
|
||||
Security → Security rules → Rate limiting rules → the existing rule → Edit
|
||||
(the Free plan cannot add a second one):
|
||||
|
||||
**Security → WAF → Rate limiting rules → Create rule → Edit expression**
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Rule name | `site-wide` |
|
||||
| Expression | `not starts_with(http.request.uri.path, "/_app/immutable/") and not cf.client.bot` |
|
||||
| Characteristics | IP |
|
||||
| Period | 10 seconds |
|
||||
| Requests | **250** |
|
||||
| Action | Block |
|
||||
| Duration | 10 seconds |
|
||||
|
||||
Why this scope: every page, the API, the image proxy, the icons and the
|
||||
service worker count, on every hostname of the zone - the rule has no
|
||||
hostname field, so `media.free-steam-games.win` counts too. `/_app/immutable/*`
|
||||
is left out because a cold first visit fetches about 115 of those files at
|
||||
once (the home page alone preloads 53), the browser then keeps them for a
|
||||
year, and the asset server serves them at no cost. `not cf.client.bot` keeps
|
||||
verified search crawlers, which read the 10,000 prerendered game pages, out of
|
||||
the count.
|
||||
|
||||
Why 250 per 10 seconds, counted from the build rather than a template: right
|
||||
after consent and the human check, a first visit sends the page and its icons
|
||||
(about 6 requests), the service worker's precache of 56 non-hashed URLs (41
|
||||
pages, 8 `__data.json`, 6 images and the manifest), the catalogue (about 16:
|
||||
`index.json`, 14 shards, the removed-games list), `/api/activity`,
|
||||
`/api/human-check` and the dashboard's thumbnails - 110 to 130 requests within
|
||||
a few seconds. 150 left no room for a second person behind the same address
|
||||
(a shared office, or a mobile carrier's CGNAT); 250 does. A script faster than
|
||||
25 requests a second is blocked for 10 seconds, again and again; one slower
|
||||
than that meets the Worker's per-minute budgets on the paths that cost money.
|
||||
|
||||
Block is tolerable because the timeout is fixed at 10 seconds: a false
|
||||
positive costs a real visitor ten seconds of missing thumbnails, not a
|
||||
lockout.
|
||||
|
||||
**The narrower rule it replaces** is still a valid choice if pages should not
|
||||
count at all - the Worker limits cover `/api/*` either way:
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Rule name | `public-proxies` |
|
||||
| Expression | `(starts_with(http.request.uri.path, "/img/") or starts_with(http.request.uri.path, "/api/data/")) and not cf.client.bot` |
|
||||
| Characteristics | IP |
|
||||
| Period | 10 seconds |
|
||||
| Requests | **150** |
|
||||
| Action | Block |
|
||||
| Duration | 10 seconds |
|
||||
| Requests | **150** per 10 seconds, IP, Block for 10 seconds |
|
||||
|
||||
Why 150 per 10 seconds, from a measured session rather than a template:
|
||||
opening `/games` fetches `index.json`, 5 shards and roughly 30 thumbnails in
|
||||
the first couple of seconds, and fast-scrolling the virtualised table adds a
|
||||
few dozen more. Someone browsing hard stays under ~100 in any 10-second window;
|
||||
150 leaves headroom for a shared office or CGNAT address that looks like one
|
||||
client. A scraper pulling faster than 15 requests a second is blocked for 10
|
||||
seconds, again and again.
|
||||
|
||||
Block is tolerable here because the Free plan's timeout is fixed at 10 seconds:
|
||||
a false positive costs a real visitor ten seconds of missing thumbnails, not a
|
||||
lockout. `not cf.client.bot` keeps verified search crawlers out of the count.
|
||||
|
||||
**Use the WAF rule, not a Worker-side limiter.** The WAF rejects at the edge
|
||||
*before* the Worker runs, so an abusive burst costs no Worker invocations. A
|
||||
limiter inside the Worker has already paid for the request by the time it says
|
||||
no — backwards for cost control.
|
||||
|
||||
**Check** (this will get your own IP blocked for 10 seconds, which is the point):
|
||||
**Check** (blocks your own IP for 10 seconds, which is the point; a static
|
||||
file, so no Worker invocation is spent):
|
||||
|
||||
```bash
|
||||
for i in $(seq 1 200); do curl -s -o /dev/null -w '%{http_code}\n' https://free-steam-games.win/api/data/data/index.json; done | sort | uniq -c
|
||||
seq 1 300 | xargs -P 30 -I{} curl -s -o /dev/null -w '%{http_code}\n' https://free-steam-games.win/robots.txt | sort | uniq -c
|
||||
```
|
||||
|
||||
Expected: about 150 × `200`, then `429`s. Watch **Security → Events** for a
|
||||
week before changing the threshold.
|
||||
Expected: about 250 × `200`, then `429`s. With `public-proxies` instead, a
|
||||
static file is never counted: use `/api/data/data/index.json` and expect about
|
||||
150. Watch **Security → Events** for a week before changing the threshold.
|
||||
|
||||
### 3b. The Worker limits — one budget per surface
|
||||
|
||||
`ratelimits` bindings in `web/wrangler.jsonc`, checked by
|
||||
`web/worker/lib/rate-limit.ts` after the country block and before every route,
|
||||
the admin surface's Access check included:
|
||||
|
||||
| Binding | Paths | Requests per IP per minute |
|
||||
|---|---|---|
|
||||
| `RL_DATA` | `/api/data/*` | 240 |
|
||||
| `RL_IMG` | `/img/*` | 600 |
|
||||
| `RL_API` | `/api/activity`, `/api/updates/desktop`, any other public `/api/*` | 120 |
|
||||
| `RL_HUMAN` | `/api/human-check` | 20 |
|
||||
| `RL_ADMIN` | `/admin`, `/admin/*`, `/api/ingest/*` | 300 |
|
||||
|
||||
- **Key:** the client's IPv4 address, or its IPv6 /64 (one subscriber usually
|
||||
holds a whole /64). Counted per Cloudflare location and approximately - a
|
||||
few requests over the limit can pass - so these are ceilings for abuse,
|
||||
never quotas. Each sits well above what a person, the apps or the discovery
|
||||
workflow sends: a cold catalogue load is about 16 data requests, a person
|
||||
sends one human check a day, discovery a few ingest calls a run.
|
||||
- **Refusal:** HTTP 429, `{"error":"too many requests"}`, `Retry-After: 60`,
|
||||
`Cache-Control: no-store`. `/api/data/*` and the public API also carry
|
||||
`Access-Control-Allow-Origin: *`, so the packaged apps see the 429 instead
|
||||
of a CORS failure; the human check and the admin surface carry none (they
|
||||
are same-origin by design).
|
||||
- **Clients:** a 429 on a pack makes the site and the apps try the plain shard
|
||||
once, then report a failed load; the human check lets the reader through
|
||||
(it refuses only on Cloudflare's verdict); discovery stops its run with the
|
||||
status.
|
||||
- **Fails open:** no binding, no `CF-Connecting-IP`, or a limiter that throws
|
||||
lets the request through. The WAF rule is the boundary; this must never be
|
||||
why the catalogue does not load.
|
||||
- **Not logged:** Workers Logs already records every invocation with its
|
||||
status - filter on 429. A console line per refusal would double the log
|
||||
volume of exactly the floods this is for. The bindings do not appear in the
|
||||
dashboard.
|
||||
- **Changing a limit** is a pull request to `web/wrangler.jsonc` (it deploys
|
||||
with the Worker). Keep every `period` at 60, which is what `Retry-After`
|
||||
promises (`rate-limit.test.ts` checks), and keep each `namespace_id` unique
|
||||
within the Cloudflare account: two bindings with one ID share their
|
||||
counters, across Workers too.
|
||||
|
||||
**Check** (after a deploy; spends 130 cheap invocations):
|
||||
|
||||
```bash
|
||||
seq 1 130 | xargs -P 10 -I{} curl -s -o /dev/null -w '%{http_code}\n' https://free-steam-games.win/api/no-such-route | sort | uniq -c
|
||||
```
|
||||
|
||||
Expected: about 120 × `404`, then `429` for the rest of that minute.
|
||||
|
||||
---
|
||||
|
||||
@@ -823,10 +908,11 @@ Retry.
|
||||
from `_headers` and may log a harmless message from inside
|
||||
`challenges.cloudflare.com`; that is not a violation of this site's policy.
|
||||
|
||||
**Rate limit (optional, no extra rule).** Each POST costs one siteverify
|
||||
subrequest. The `public-proxies` expression in section 3 can take
|
||||
`or http.request.uri.path eq "/api/human-check"`; a person sends a few a day,
|
||||
far below 150 per 10 seconds.
|
||||
**Rate limit.** Each POST costs one siteverify subrequest, so the route has
|
||||
its own Worker budget, `RL_HUMAN`: 20 per IP per minute (section 3b), where a
|
||||
person sends one a day. A refused check lets the reader through like any other
|
||||
answer that is not a verdict. The `site-wide` WAF rule (section 3a) counts it
|
||||
too; the narrower `public-proxies` rule does not.
|
||||
|
||||
**Local testing.** `npm run dev` skips the check: its `/api` proxy points at
|
||||
production, which refuses a localhost token. `npm run preview` proxies the same
|
||||
@@ -858,7 +944,10 @@ pruned daily after `ADMIN_RETENTION_DAYS` (180), which
|
||||
`docs/PRIVACY_POLICY.md` states.
|
||||
|
||||
**The Free-plan rule budget is now the constraint to watch:** the one rate
|
||||
limiting rule is used (section 3), and two of the five custom rules are
|
||||
(section 4). A third custom rule is affordable; a second rate-limiting rule is
|
||||
not, without upgrading. Configuration Rules are a separate quota: one of ten
|
||||
is used (section 10).
|
||||
limiting rule is used (section 3a), and four of the five custom rules are
|
||||
(as of 2026-10-27: `human-check-pages`, `blocked-countries`,
|
||||
`media-host-paths`, and Cloudflare's AI Crawl Control rule). A fifth custom
|
||||
rule is affordable; a second rate-limiting rule is not, without upgrading -
|
||||
which is why per-surface limits live in the Worker (section 3b), where they
|
||||
cost no rule. Configuration Rules are a separate quota: one of ten is used
|
||||
(section 10).
|
||||
Vendored
+6
-1
@@ -4,12 +4,17 @@
|
||||
interface __BaseEnv_Env {
|
||||
MEDIA: R2Bucket;
|
||||
DB: D1Database;
|
||||
RL_DATA: RateLimit;
|
||||
RL_IMG: RateLimit;
|
||||
RL_API: RateLimit;
|
||||
RL_HUMAN: RateLimit;
|
||||
RL_ADMIN: RateLimit;
|
||||
IMAGES: ImagesBinding;
|
||||
ASSETS: Fetcher;
|
||||
SITE_ORIGIN: "https://free-steam-games.win";
|
||||
IMG_TRANSFORM: "true";
|
||||
IMG_TRANSFORM_MONTHLY_CAP: "20000";
|
||||
IMG_MINT_PER_TICK: "10";
|
||||
IMG_MINT_PER_TICK: "40";
|
||||
ADMIN_RETENTION_DAYS: "180";
|
||||
BLOCKED_COUNTRIES: "CN,RU,AR";
|
||||
ACCESS_TEAM_DOMAIN: "polished-hall-48fd.cloudflareaccess.com";
|
||||
|
||||
@@ -138,6 +138,66 @@ describe("the country block", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("the rate limits", () => {
|
||||
/** makeEnv() binds no limiter, which is why every other test is unaffected
|
||||
* (lib/rate-limit.ts fails open). Here each one allows two requests. */
|
||||
function limitedEnv(extra: Record<string, unknown> = {}) {
|
||||
const calls: string[] = [];
|
||||
const counts = new Map<string, number>();
|
||||
const limiter = (binding: string) => ({
|
||||
async limit({ key }: { key: string }) {
|
||||
calls.push(binding);
|
||||
const n = (counts.get(binding + key) ?? 0) + 1;
|
||||
counts.set(binding + key, n);
|
||||
return { success: n <= 2 };
|
||||
},
|
||||
});
|
||||
const bindings = Object.fromEntries(
|
||||
["RL_DATA", "RL_IMG", "RL_API", "RL_HUMAN", "RL_ADMIN"].map((b) => [b, limiter(b)]),
|
||||
);
|
||||
return { env: { ...makeEnv(), ...bindings, ...extra } as unknown as Env, calls };
|
||||
}
|
||||
|
||||
const hit = (limited: Env, path: string, headers: Record<string, string> = {}) =>
|
||||
worker.fetch(
|
||||
new Request(`https://free-steam-games.win${path}`, { headers: { "CF-Connecting-IP": "203.0.113.7", ...headers } }),
|
||||
limited,
|
||||
ctx,
|
||||
);
|
||||
|
||||
it("answers 429 once a client spends its budget, before the route runs", async () => {
|
||||
const { env: limited } = limitedEnv();
|
||||
expect((await hit(limited, "/api/no-such-route")).status).toBe(404);
|
||||
expect((await hit(limited, "/api/no-such-route")).status).toBe(404);
|
||||
const res = await hit(limited, "/api/no-such-route");
|
||||
expect(res.status).toBe(429);
|
||||
expect(res.headers.get("Retry-After")).toBe("60");
|
||||
expect(res.headers.get("Access-Control-Allow-Origin")).toBe("*");
|
||||
});
|
||||
|
||||
it("limits the admin surface ahead of the Access check", async () => {
|
||||
const { env: limited } = limitedEnv();
|
||||
await hit(limited, "/admin/api/me");
|
||||
await hit(limited, "/admin/api/me");
|
||||
const res = await hit(limited, "/admin/api/me", { "Cf-Access-Jwt-Assertion": await token() });
|
||||
expect(res.status).toBe(429);
|
||||
// Same-origin by design: no CORS on the admin API, refusal included.
|
||||
expect(res.headers.get("Access-Control-Allow-Origin")).toBeNull();
|
||||
});
|
||||
|
||||
it("gives the human check its own budget", async () => {
|
||||
const { env: limited, calls } = limitedEnv();
|
||||
await hit(limited, "/api/human-check");
|
||||
expect(calls).toEqual(["RL_HUMAN"]);
|
||||
});
|
||||
|
||||
it("does not count a request the country block already refused", async () => {
|
||||
const { env: limited, calls } = limitedEnv({ BLOCKED_COUNTRIES: "CN" });
|
||||
expect((await hit(limited, "/api/activity", { "CF-IPCountry": "CN" })).status).toBe(403);
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the human-check route", () => {
|
||||
// Through the real router: a handler test cannot notice the route being
|
||||
// shadowed by the /api/* 404 or pulled behind the Access block.
|
||||
|
||||
@@ -28,6 +28,7 @@ import { mintImages } from "./lib/img-mint";
|
||||
import { defaultAdminDeps } from "./lib/deps";
|
||||
import { isPruneTick, pruneAdminHistory } from "./lib/prune";
|
||||
import { isBlockedCountry } from "./lib/geo";
|
||||
import { rateLimited } from "./lib/rate-limit";
|
||||
|
||||
export default {
|
||||
async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
|
||||
@@ -158,6 +159,13 @@ async function route(
|
||||
);
|
||||
}
|
||||
|
||||
// Per-IP limits, after the country block so a refused request is not also
|
||||
// counted, and before the admin block for the same reason as that one: one
|
||||
// place, ahead of every route (lib/rate-limit.ts). The WAF rate-limiting
|
||||
// rule runs before the Worker and is the first layer; this is the second.
|
||||
const limited = await rateLimited(request, pathname, env);
|
||||
if (limited) return limited;
|
||||
|
||||
if (pathname.startsWith("/api/data/")) {
|
||||
return handleData(request, url, ctx);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
LIMIT_BINDINGS,
|
||||
LIMIT_PERIOD_SECONDS,
|
||||
clientKey,
|
||||
limitClass,
|
||||
rateLimited,
|
||||
tooManyRequests,
|
||||
} from "./rate-limit";
|
||||
|
||||
/** A limiter that counts exactly, unlike Cloudflare's, so tests can be exact. */
|
||||
function fakeLimiter(limit: number) {
|
||||
const counts = new Map<string, number>();
|
||||
const keys: string[] = [];
|
||||
return {
|
||||
keys,
|
||||
async limit({ key }: { key: string }) {
|
||||
keys.push(key);
|
||||
const n = (counts.get(key) ?? 0) + 1;
|
||||
counts.set(key, n);
|
||||
return { success: n <= limit };
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const envWith = (bindings: Record<string, unknown>) => bindings as unknown as Env;
|
||||
|
||||
const from = (ip: string | null, path = "/api/data/data/index.json") =>
|
||||
new Request(`https://free-steam-games.win${path}`, { headers: ip ? { "CF-Connecting-IP": ip } : {} });
|
||||
|
||||
/**
|
||||
* wrangler.jsonc, comments and trailing commas removed. Comments are only
|
||||
* stripped outside strings: the file's URLs carry `//` inside quotes.
|
||||
*/
|
||||
function readWranglerConfig(): Record<string, unknown> {
|
||||
const text = readFileSync(join(__dirname, "..", "..", "wrangler.jsonc"), "utf8");
|
||||
let out = "";
|
||||
for (let i = 0; i < text.length; i++) {
|
||||
const c = text[i];
|
||||
if (c === '"') {
|
||||
let j = i + 1;
|
||||
while (j < text.length && text[j] !== '"') j += text[j] === "\\" ? 2 : 1;
|
||||
out += text.slice(i, j + 1);
|
||||
i = j;
|
||||
} else if (c === "/" && text[i + 1] === "/") {
|
||||
while (i < text.length && text[i] !== "\n") i++;
|
||||
out += "\n";
|
||||
} else if (c === "/" && text[i + 1] === "*") {
|
||||
i = text.indexOf("*/", i + 2) + 1;
|
||||
} else {
|
||||
out += c;
|
||||
}
|
||||
}
|
||||
return JSON.parse(out.replace(/,(\s*[}\]])/g, "$1")) as Record<string, unknown>;
|
||||
}
|
||||
|
||||
describe("limitClass", () => {
|
||||
it("gives each surface the Worker serves its own budget", () => {
|
||||
expect(limitClass("/api/data/data/index.json")).toBe("data");
|
||||
expect(limitClass("/api/data/p1/data_001.bin")).toBe("data");
|
||||
expect(limitClass("/img/s/730.jpg")).toBe("img");
|
||||
expect(limitClass("/api/human-check")).toBe("human");
|
||||
expect(limitClass("/api/activity")).toBe("api");
|
||||
expect(limitClass("/api/updates/desktop")).toBe("api");
|
||||
expect(limitClass("/api/no-such-route")).toBe("api");
|
||||
});
|
||||
|
||||
it("puts the Access-protected surfaces in one generous budget", () => {
|
||||
for (const path of ["/admin", "/admin/queue", "/admin/assets/app-1.js", "/admin/api/me", "/api/ingest/known"]) {
|
||||
expect(limitClass(path), path).toBe("admin");
|
||||
}
|
||||
});
|
||||
|
||||
it("covers nothing the Worker never sees", () => {
|
||||
expect(limitClass("/")).toBeNull();
|
||||
expect(limitClass("/games/730")).toBeNull();
|
||||
expect(limitClass("/_app/immutable/entry/start.js")).toBeNull();
|
||||
expect(limitClass("/administrator")).toBeNull();
|
||||
});
|
||||
|
||||
it("covers every prefix the Worker runs first for", () => {
|
||||
// A prefix added to run_worker_first without a budget would be the one
|
||||
// surface a flood could reach unlimited.
|
||||
const assets = readWranglerConfig().assets as { run_worker_first: string[] };
|
||||
for (const prefix of assets.run_worker_first) {
|
||||
const sample = prefix.replace(/\*$/, "anything");
|
||||
expect(limitClass(sample), prefix).not.toBeNull();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("clientKey", () => {
|
||||
it("is the IPv4 address", () => {
|
||||
expect(clientKey(from("203.0.113.7"))).toBe("203.0.113.7");
|
||||
});
|
||||
|
||||
it("is the /64 of an IPv6 address, however it is written", () => {
|
||||
const key = "2001:db8:a:b::/64";
|
||||
expect(clientKey(from("2001:db8:a:b:1:2:3:4"))).toBe(key);
|
||||
expect(clientKey(from("2001:0DB8:000A:000B:ffff:0:0:1"))).toBe(key);
|
||||
expect(clientKey(from("2001:db8:a:b::1"))).toBe(key);
|
||||
expect(clientKey(from("2001:db8::1"))).toBe("2001:db8:0:0::/64");
|
||||
expect(clientKey(from("::1"))).toBe("0:0:0:0::/64");
|
||||
});
|
||||
|
||||
it("unwraps an IPv4-mapped address instead of filing it under ::/64", () => {
|
||||
// Every mapped address shares its first four groups (all zero), so taking
|
||||
// the /64 would put every such client on one counter.
|
||||
expect(clientKey(from("::FFFF:198.51.100.4"))).toBe("198.51.100.4");
|
||||
});
|
||||
|
||||
it("falls back to the raw value for anything it cannot parse", () => {
|
||||
expect(clientKey(from("1:2:3"))).toBe("1:2:3");
|
||||
expect(clientKey(from("1::2::3"))).toBe("1::2::3");
|
||||
});
|
||||
|
||||
it("is null without the header (Vitest, a direct origin request)", () => {
|
||||
expect(clientKey(from(null))).toBeNull();
|
||||
expect(clientKey(from(" "))).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("tooManyRequests", () => {
|
||||
it("is a 429 that says when to come back and is never cached", async () => {
|
||||
const res = tooManyRequests("img");
|
||||
expect(res.status).toBe(429);
|
||||
expect(res.headers.get("Retry-After")).toBe(String(LIMIT_PERIOD_SECONDS));
|
||||
expect(res.headers.get("Cache-Control")).toBe("no-store");
|
||||
expect(res.headers.get("X-Content-Type-Options")).toBe("nosniff");
|
||||
expect(await res.json()).toEqual({ error: "too many requests" });
|
||||
});
|
||||
|
||||
it("answers cross-origin only where the route itself does", () => {
|
||||
// The apps fetch /api/data/* and /api/activity from tauri://localhost;
|
||||
// without the header they would see a CORS error, not the 429.
|
||||
expect(tooManyRequests("data").headers.get("Access-Control-Allow-Origin")).toBe("*");
|
||||
expect(tooManyRequests("api").headers.get("Access-Control-Allow-Origin")).toBe("*");
|
||||
for (const cls of ["img", "human", "admin"] as const) {
|
||||
expect(tooManyRequests(cls).headers.get("Access-Control-Allow-Origin"), cls).toBeNull();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("rateLimited", () => {
|
||||
it("lets a client through up to the limit, then refuses it", async () => {
|
||||
const env = envWith({ RL_DATA: fakeLimiter(2) });
|
||||
expect(await rateLimited(from("203.0.113.7"), "/api/data/x", env)).toBeNull();
|
||||
expect(await rateLimited(from("203.0.113.7"), "/api/data/x", env)).toBeNull();
|
||||
expect((await rateLimited(from("203.0.113.7"), "/api/data/x", env))?.status).toBe(429);
|
||||
// Another address has its own budget.
|
||||
expect(await rateLimited(from("203.0.113.8"), "/api/data/x", env)).toBeNull();
|
||||
});
|
||||
|
||||
it("charges one IPv6 /64 as one client", async () => {
|
||||
const env = envWith({ RL_DATA: fakeLimiter(1) });
|
||||
expect(await rateLimited(from("2001:db8:a:b::1"), "/api/data/x", env)).toBeNull();
|
||||
expect((await rateLimited(from("2001:db8:a:b::2"), "/api/data/x", env))?.status).toBe(429);
|
||||
});
|
||||
|
||||
it("keeps each surface's budget apart", async () => {
|
||||
const data = fakeLimiter(1);
|
||||
const img = fakeLimiter(1);
|
||||
const env = envWith({ RL_DATA: data, RL_IMG: img });
|
||||
expect(await rateLimited(from("203.0.113.7"), "/api/data/x", env)).toBeNull();
|
||||
expect(await rateLimited(from("203.0.113.7"), "/img/s/730.jpg", env)).toBeNull();
|
||||
expect(data.keys).toEqual(["203.0.113.7"]);
|
||||
expect(img.keys).toEqual(["203.0.113.7"]);
|
||||
});
|
||||
|
||||
it("fails open without a binding, without an address, or when the limiter throws", async () => {
|
||||
expect(await rateLimited(from("203.0.113.7"), "/api/data/x", envWith({}))).toBeNull();
|
||||
|
||||
const unused = fakeLimiter(0);
|
||||
expect(await rateLimited(from(null), "/api/data/x", envWith({ RL_DATA: unused }))).toBeNull();
|
||||
expect(unused.keys).toEqual([]);
|
||||
|
||||
const error = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
const broken = envWith({ RL_DATA: { limit: () => Promise.reject(new Error("binding gone")) } });
|
||||
expect(await rateLimited(from("203.0.113.7"), "/api/data/x", broken)).toBeNull();
|
||||
expect(error).toHaveBeenCalledOnce();
|
||||
error.mockRestore();
|
||||
});
|
||||
|
||||
it("does not touch a limiter for a path no class covers", async () => {
|
||||
const limiter = fakeLimiter(0);
|
||||
const env = envWith(Object.fromEntries(Object.values(LIMIT_BINDINGS).map((b) => [b, limiter])));
|
||||
expect(await rateLimited(from("203.0.113.7"), "/games/730", env)).toBeNull();
|
||||
expect(limiter.keys).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("wrangler.jsonc ratelimits", () => {
|
||||
const declared = readWranglerConfig().ratelimits as {
|
||||
name: string;
|
||||
namespace_id: string;
|
||||
simple: { limit: number; period: number };
|
||||
}[];
|
||||
|
||||
it("declares exactly the bindings the Worker reads", () => {
|
||||
expect(declared.map((r) => r.name).sort()).toEqual(Object.values(LIMIT_BINDINGS).sort());
|
||||
});
|
||||
|
||||
it("uses the period Retry-After promises", () => {
|
||||
for (const r of declared) expect(r.simple.period, r.name).toBe(LIMIT_PERIOD_SECONDS);
|
||||
});
|
||||
|
||||
it("gives each binding its own namespace, or two budgets would share counters", () => {
|
||||
const ids = declared.map((r) => r.namespace_id);
|
||||
expect(new Set(ids).size).toBe(ids.length);
|
||||
for (const id of ids) expect(id).toMatch(/^[1-9]\d*$/);
|
||||
});
|
||||
|
||||
it("sets limits a person, the apps and the discovery workflow stay well under", () => {
|
||||
for (const r of declared) {
|
||||
expect(Number.isInteger(r.simple.limit), r.name).toBe(true);
|
||||
// A cold catalogue load is ~16 /api/data requests; anything under a few
|
||||
// loads a minute would refuse a household behind one address.
|
||||
expect(r.simple.limit, r.name).toBeGreaterThanOrEqual(r.name === "RL_HUMAN" ? 10 : 100);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,139 @@
|
||||
/**
|
||||
* Per-IP request limits: the Worker's half of rate limiting.
|
||||
*
|
||||
* The FIRST layer is the zone's WAF rate-limiting rule (docs/SECURITY_SETUP.md
|
||||
* section 3). It refuses a burst at the edge before this Worker is invoked, so
|
||||
* a flood it catches costs no invocation, no subrequest and no R2 read. The
|
||||
* Free plan gives the zone exactly one such rule and fixes its shape: count by
|
||||
* IP over 10 seconds, block for 10. This layer adds what that rule cannot - a
|
||||
* 60-second window, so a client pacing itself just under the 10-second
|
||||
* threshold is still stopped, and a budget per surface, because a human check
|
||||
* is not an image - and it keeps holding if the WAF rule is edited or deleted.
|
||||
*
|
||||
* Like BLOCKED_COUNTRIES (lib/geo.ts) it sees only the four
|
||||
* `run_worker_first` prefixes. The prerendered pages never reach the Worker;
|
||||
* the WAF rule is all that limits them.
|
||||
*
|
||||
* The limits are `ratelimits` bindings in wrangler.jsonc, one per class below.
|
||||
* Cloudflare counts per location and approximately (a few requests over the
|
||||
* limit can pass), so a limit here is a ceiling for abuse, never an accurate
|
||||
* quota - every number sits well above what a person, the apps or the
|
||||
* discovery workflow sends.
|
||||
*
|
||||
* FAIL OPEN. A missing binding (Vitest, `admin/mock`, a config that lost one),
|
||||
* a request without CF-Connecting-IP, or a limit() call that throws lets the
|
||||
* request through: the WAF rule is the boundary, and this must never be the
|
||||
* reason the catalogue does not load.
|
||||
*
|
||||
* Refusals are not logged. Workers Logs already records every invocation with
|
||||
* its status, and a console line per refused request would double the log
|
||||
* volume of exactly the floods this exists for. Filter on status 429.
|
||||
*/
|
||||
import { HUMAN_CHECK_PATH } from "../../shared/human-check";
|
||||
import { ADMIN_API_PREFIX } from "../../shared/admin-routes";
|
||||
import { errorText, jsonError } from "./http";
|
||||
|
||||
export type LimitClass = "data" | "img" | "api" | "human" | "admin";
|
||||
|
||||
/** The wrangler.jsonc `ratelimits` binding behind each class. */
|
||||
export const LIMIT_BINDINGS = {
|
||||
data: "RL_DATA",
|
||||
img: "RL_IMG",
|
||||
api: "RL_API",
|
||||
human: "RL_HUMAN",
|
||||
admin: "RL_ADMIN",
|
||||
} as const satisfies Record<LimitClass, keyof Env>;
|
||||
|
||||
/**
|
||||
* Every binding's `period`, and so the Retry-After of a refusal. The binding
|
||||
* cannot report when its window ends, so this is the honest upper bound.
|
||||
* rate-limit.test.ts holds wrangler.jsonc to it.
|
||||
*/
|
||||
export const LIMIT_PERIOD_SECONDS = 60;
|
||||
|
||||
/**
|
||||
* The classes whose routes answer cross-origin (`/api/data/*`, `/api/activity`),
|
||||
* so their refusal does too: without the header the packaged apps would see a
|
||||
* CORS failure where there is a 429. Never `human` or `admin` - those routes
|
||||
* are same-origin by design (CLAUDE.md), and `/img/*` sends no CORS at all.
|
||||
*/
|
||||
const CORS_CLASSES: ReadonlySet<LimitClass> = new Set(["data", "api"]);
|
||||
|
||||
/** Which budget a path draws on, or null for a path no limit covers. */
|
||||
export function limitClass(pathname: string): LimitClass | null {
|
||||
if (pathname.startsWith("/api/data/")) return "data";
|
||||
if (pathname.startsWith("/img/")) return "img";
|
||||
if (pathname === HUMAN_CHECK_PATH) return "human";
|
||||
if (
|
||||
pathname.startsWith("/api/ingest/") ||
|
||||
pathname.startsWith(ADMIN_API_PREFIX) ||
|
||||
pathname === "/admin" ||
|
||||
pathname.startsWith("/admin/")
|
||||
) {
|
||||
return "admin";
|
||||
}
|
||||
if (pathname.startsWith("/api/")) return "api";
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The counter a request is charged to: its IPv4 address, or its IPv6 /64.
|
||||
*
|
||||
* An IPv6 subscriber is normally handed a whole /64, so counting full
|
||||
* addresses would give one client 2^64 fresh budgets. A NAT'd IPv4 address
|
||||
* can stand for many people (mobile carriers use CGNAT widely), which is why
|
||||
* the limits are generous rather than why the key is different.
|
||||
*/
|
||||
export function clientKey(request: Request): string | null {
|
||||
const ip = request.headers.get("CF-Connecting-IP")?.trim().toLowerCase();
|
||||
if (!ip) return null;
|
||||
if (!ip.includes(":")) return ip;
|
||||
const mapped = /^::ffff:(\d{1,3}(?:\.\d{1,3}){3})$/.exec(ip);
|
||||
if (mapped) return mapped[1];
|
||||
return ipv6Prefix64(ip) ?? ip;
|
||||
}
|
||||
|
||||
/** `2001:db8:a:b:1:2:3:4` -> `2001:db8:a:b::/64`; null if it is not IPv6. */
|
||||
function ipv6Prefix64(ip: string): string | null {
|
||||
const halves = ip.split("::");
|
||||
if (halves.length > 2) return null;
|
||||
const groups = (s: string | undefined) => (s ? s.split(":") : []);
|
||||
// A trailing dotted quad (`::ffff:1.2.3.4`, `64:ff9b::1.2.3.4`) is two groups.
|
||||
const width = (gs: string[]) => gs.reduce((n, g) => n + (g.includes(".") ? 2 : 1), 0);
|
||||
const left = groups(halves[0]);
|
||||
const right = groups(halves[1]);
|
||||
const missing = 8 - width(left) - width(right);
|
||||
if (halves.length === 1 ? missing !== 0 : missing < 1) return null;
|
||||
const all = [...left, ...Array<string>(halves.length === 1 ? 0 : missing).fill("0"), ...right];
|
||||
const prefix = all.slice(0, 4);
|
||||
if (!prefix.every((g) => /^[0-9a-f]{1,4}$/.test(g))) return null;
|
||||
return `${prefix.map((g) => g.replace(/^0+(?=.)/, "")).join(":")}::/64`;
|
||||
}
|
||||
|
||||
/** The refusal: JSON like every other Worker error, never cached. */
|
||||
export function tooManyRequests(cls: LimitClass): Response {
|
||||
const res = jsonError(429, "too many requests");
|
||||
res.headers.set("Retry-After", String(LIMIT_PERIOD_SECONDS));
|
||||
if (CORS_CLASSES.has(cls)) res.headers.set("Access-Control-Allow-Origin", "*");
|
||||
return res;
|
||||
}
|
||||
|
||||
/**
|
||||
* Charge the request to its class's budget. Returns the 429 to send when the
|
||||
* budget is spent, or null to carry on (including every fail-open case).
|
||||
*/
|
||||
export async function rateLimited(request: Request, pathname: string, env: Env): Promise<Response | null> {
|
||||
const cls = limitClass(pathname);
|
||||
if (!cls) return null;
|
||||
// Typed as always bound; at runtime it is not, under Vitest and admin/mock.
|
||||
const limiter: RateLimit | undefined = env[LIMIT_BINDINGS[cls]];
|
||||
const key = clientKey(request);
|
||||
if (!limiter || !key) return null;
|
||||
try {
|
||||
const { success } = await limiter.limit({ key });
|
||||
return success ? null : tooManyRequests(cls);
|
||||
} catch (err) {
|
||||
console.error("rate-limit: limiter failed, letting the request through", { cls, message: errorText(err) });
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -94,6 +94,33 @@
|
||||
"binding": "IMAGES"
|
||||
},
|
||||
|
||||
// Per-IP request limits, the Worker's half of rate limiting
|
||||
// (worker/lib/rate-limit.ts). The zone's WAF rate-limiting rule is the
|
||||
// first layer and refuses a burst before this Worker runs at all
|
||||
// (docs/SECURITY_SETUP.md section 3); the Free plan fixes it at 10 seconds,
|
||||
// one rule. These add a 60-second window and one budget per surface, and
|
||||
// still hold if that rule is edited or deleted. They see only the four
|
||||
// `run_worker_first` prefixes - the prerendered pages are the WAF's alone.
|
||||
//
|
||||
// Counts are per Cloudflare location and approximate; the key is the
|
||||
// client's IPv4 address or IPv6 /64. Over a limit the Worker answers 429
|
||||
// with Retry-After: 60, so every "period" stays 60 (rate-limit.test.ts
|
||||
// holds that). A namespace_id must be unique within the Cloudflare account:
|
||||
// two bindings that share one share counters, across Workers too.
|
||||
//
|
||||
// RL_DATA /api/data/* a cold catalogue load is ~16 requests
|
||||
// RL_IMG /img/* the apps fetch every thumbnail here
|
||||
// RL_API /api/activity, /api/updates/desktop, any other public /api/*
|
||||
// RL_HUMAN /api/human-check a person sends one a day
|
||||
// RL_ADMIN /admin, /admin/*, /api/ingest/* (behind Access as well)
|
||||
"ratelimits": [
|
||||
{ "name": "RL_DATA", "namespace_id": "730001", "simple": { "limit": 240, "period": 60 } },
|
||||
{ "name": "RL_IMG", "namespace_id": "730002", "simple": { "limit": 600, "period": 60 } },
|
||||
{ "name": "RL_API", "namespace_id": "730003", "simple": { "limit": 120, "period": 60 } },
|
||||
{ "name": "RL_HUMAN", "namespace_id": "730004", "simple": { "limit": 20, "period": 60 } },
|
||||
{ "name": "RL_ADMIN", "namespace_id": "730005", "simple": { "limit": 300, "period": 60 } }
|
||||
],
|
||||
|
||||
// Reconcile approvals against the published dataset. Every 15 minutes rather
|
||||
// than hourly because the gap between approving and seeing it land is the
|
||||
// one moment a reviewer is actually watching; a tick with nothing approved
|
||||
|
||||
Reference in new issue
Block a user