Site-wide rate limit: 429 for every request the Worker sees (#106)

* feat(security): site-wide rate limit for every request the Worker sees

Only the /api groups were rate-limited; pages, /media, /ext-img and the
search index had no limit at all (a 200-request burst against production
answered 200 every time). A new RL_SITE binding (100 requests per 10 s per
client, IPv6 per /64) is checked by the Worker entry right after the IP
denylist. Over it: the 429 page, no-store, Retry-After: 10, its countdown
rewritten to 10 s. The admin host is exempt (behind Access, RL_ADMIN per
account). A throwing limiter fails open without skipping the Turnstile gate.

- tests: worker spam/keys/exemptions/fail-open; Retry-After constants are
  checked against the periods in wrangler.jsonc
- smoke: a fourth phase spams the built Worker until the limit answers
- scripts/probe-ratelimit.mjs: tell the Worker's 429 from the WAF's 1015
- docs 06/07/02/09/00: three layers (WAF → RL_SITE → /api groups) and the
  Free plan's real WAF limits; CHANGELOG 2.1.0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(sonar): project version 2.1.0 for the release

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(worker): stay a 429 if the rate-limit page can't be built

Review follow-ups on the site-wide limit:
- If fetching or rewriting the 429 page threw, the error reached the edge
  fail-open path, which hands the request to the Astro app and skips the
  Turnstile gate. It now answers a plain-text 429 (no-store, Retry-After).
- Drop Last-Modified and Content-Length with the ETag, as the gate does:
  the countdown rewrite only kept the length by luck ("60" vs "10").
- Docs: the WAF rule can't filter by host on Free, so it counts admin.*
  too; 09 lists the worker entry as a 429 source and the probe checks;
  CHANGELOG says what was unlimited before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 authored and GitHub committed 2026-10-27 07:34:15 +07:00
1 parent bf66593c7d
commit 846e5ec17d
18 files changed
+355 -42

No files matched your search

+22 -1
View File
@@ -7,6 +7,26 @@ and are not listed here.
## [Unreleased]
## [2.1.0] — 2026-10-27
### Security
- **Site-wide rate limit.** Every request that reaches the Worker — pages, `/media`, `/ext-img`,
the search index and the API — now spends one budget per client: 100 requests per 10 seconds
(IPv6 counted per /64), a new `RL_SITE` binding in `wrangler.jsonc`. Over it, visitors get the
site's 429 page with `Retry-After: 10` and a 10-second countdown; the stricter per-endpoint
API limits still apply underneath. Before, only the API groups were limited — pages, images,
the search index and the widgets endpoint were not. The admin host is exempt (Cloudflare Access
stands in front of it), and a failing limiter lets requests through without skipping the
Turnstile gate.
- Setup docs describe the zone WAF rate-limiting rule as the outer layer above it, with the Free
plan's real limits (path and verified-bot fields, 10-second periods and blocks).
### Added
- `npm run probe:ratelimit` spams a live URL and tells the Worker's 429s apart from the WAF's.
The CI smoke test now spams the built Worker until the site-wide limit answers.
### Changed
- Faster first paint on phones: every page carries its CSS inline (about 9 KB brotli) instead of
@@ -109,6 +129,7 @@ The public launch is tagged `v1-launch`; the 1.x changes after it were not tagge
- **2026-10** — Custom Worker entry in front of every page; Workers Rate Limiting; Web Analytics;
topic email aliases; R2 media; the Turnstile gate; the admin at `admin.poli0981.dev`.
[Unreleased]: https://github.com/poli0981/poli0981-dev/compare/v2.0.0...HEAD
[Unreleased]: https://github.com/poli0981/poli0981-dev/compare/v2.1.0...HEAD
[2.1.0]: https://github.com/poli0981/poli0981-dev/compare/v2.0.0...v2.1.0
[2.0.0]: https://github.com/poli0981/poli0981-dev/releases/tag/v2.0.0
[1.x]: https://github.com/poli0981/poli0981-dev/releases/tag/v1-launch
+1
View File
@@ -65,6 +65,7 @@ identity (`dev@localhost`); saving content needs a `GITHUB_CONTENT_TOKEN` in `.d
| `npm run lint` · `npm run format:check` · `npm run knip` | ESLint · Prettier · unused files/exports |
| `npm test` · `npm run test:coverage` | `node:test` suites; coverage writes `coverage/lcov.info` (80% line floor) |
| `npm run smoke` | Runs the built Worker with `wrangler dev` and probes pages, headers, the gate and the denylist |
| `npm run probe:ratelimit -- [url] [requests] [parallel]` | Spams a live URL and counts 429s from the Worker's site-wide limit vs. the WAF rule |
| `npm run new:post` | Scaffold a post or story from `docs/templates/` |
| `npm run check-exif` · `npm run strip-exif` | Refuse / remove GPS and camera metadata in images |
| `npm run gen-notices` | Regenerate `THIRD-PARTY-NOTICES.md` (CI checks it) |
+1
View File
@@ -18,6 +18,7 @@
"build": "astro build && node scripts/gen-og.mjs && pagefind --site dist/client && node scripts/check-csp-hashes.mjs",
"check-csp": "node scripts/check-csp-hashes.mjs",
"smoke": "node scripts/smoke-worker.mjs",
"probe:ratelimit": "node scripts/probe-ratelimit.mjs",
"test": "node --import ./tests/support/register.mjs --test \"tests/**/*.test.ts\"",
"pretest:coverage": "node -e \"require('node:fs').mkdirSync('coverage', { recursive: true })\"",
"test:coverage": "node --import ./tests/support/register.mjs --test --experimental-test-coverage --test-coverage-include=\"src/**/*.ts\" --test-coverage-include=\"workers/widgets/src/**/*.ts\" --test-coverage-exclude=\"**/*.d.ts\" --test-coverage-exclude=\"src/lib/easter.ts\" --test-coverage-exclude=\"src/content.config.ts\" --test-coverage-lines=80 --test-reporter=spec --test-reporter-destination=stdout --test-reporter=lcov --test-reporter-destination=coverage/lcov.info \"tests/**/*.test.ts\"",
+1 -1
View File
@@ -46,7 +46,7 @@
| D9 | API ngoài | Không weather/map; widget YouTube + Steam + GitHub qua cron worker |
| D11 | Email | Google Workspace (MX Google), một hộp thư + alias theo chủ đề (`07` §11). Không Cloudflare Email Routing |
| D12 | Chống bot | **Cổng Turnstile** trước mọi trang, vé 48 giờ (`06` §3b); bot đã xác thực đi thẳng nhờ Transform Rule `x-verified-bot` |
| D13 | Rate limit | Workers Rate Limiting bindings theo nhóm `/api` (`06` §2) thay bộ đếm KV |
| D13 | Rate limit | WAF rule (edge) → Workers Rate Limiting: RL_SITE toàn site 100/10s → bindings theo nhóm `/api` (`06` §2, `07` §6) thay bộ đếm KV |
| D14 | Analytics | Cloudflare Web Analytics, **snippet thủ công** chỉ ở bản build production (`07` §12) |
| D15 | Admin | `admin.poli0981.dev` sau Cloudflare Access + Worker tự kiểm JWT (≤ 24 giờ, email phải có trong `ADMIN_EMAILS` — thiếu là từ chối); token GitHub chỉ ghi repo content; ảnh → Images binding → R2; mọi thao tác ghi audit log (`06` §9) |
| D10 | Data services | **KV** (cache widget + ảnh widget mirror cho `/ext-img`, denylist một khoá, audit log admin) + **R2** `poli0981-media` cho ảnh upload qua admin (từ 10/2026; AVIF/WebP sinh một lần lúc upload, phục vụ qua `/media`, cache 1 năm). Không D1. Ảnh build-time cũ vẫn ở `src/assets/` |
+3 -2
View File
@@ -11,6 +11,7 @@ Khách ──HTTPS──> Cloudflare edge (WAF free, Bot Fight, DDoS auto, Trans
src/worker.ts — entry của Worker (assets.run_worker_first: mọi trang đều qua đây)
├── www → apex (301)
├── denylist IP (KV, một khoá "denylist", CIDR + hạn) → trang 403
├── rate limit toàn site RL_SITE 100/10s/IP (trừ admin.*) → trang 429
├── tách host: admin.poli0981.dev chỉ /admin*, /api/admin*, /media/*
├── /media/* → R2 poli0981-media (ảnh upload qua admin, cache 1 năm)
├── /ext-img/* → KV widgets:img:* (ảnh widget đã mirror; không gọi Google/Steam)
@@ -54,7 +55,7 @@ poli0981-dev/
├── scripts/ # gen-og, gen-notices, check-exif/strip-exif, check-csp-hashes,
│ # check-install-scripts, smoke-worker, new-post
├── src/
│ ├── worker.ts # entry: denylist, host, /media, /ext-img, cổng, assets
│ ├── worker.ts # entry: denylist, rate limit site, host, /media, /ext-img, cổng, assets
│ ├── middleware.ts # admin JWT + CSRF, rate limit /api, trang lỗi
│ ├── assets/ # ảnh qua pipeline Astro (gallery/, covers/)
│ ├── components/ # .astro + islands .svelte
@@ -94,7 +95,7 @@ poli0981-dev/
"assets": { "binding": "ASSETS", "directory": "./dist", "not_found_handling": "404-page",
"html_handling": "auto-trailing-slash", "run_worker_first": ["/*", "!/_astro/*", "…"] },
"kv_namespaces": [{ "binding": "KV", "id": "…" }],
"ratelimits": [ RL_REPORT 5/60s · RL_GATE 30/60s · RL_ADMIN 120/60s · RL_API 120/60s ],
"ratelimits": [ RL_SITE 100/10s · RL_REPORT 5/60s · RL_GATE 30/60s · RL_ADMIN 120/60s · RL_API 120/60s ],
"vars": { "GATE_MODE": "on", "ACCESS_TEAM_DOMAIN": "…", "ACCESS_AUD": "…", "REPORT_REPO": "poli0981/site-reports" },
"r2_buckets": [{ "binding": "MEDIA", "bucket_name": "poli0981-media" }],
"images": { "binding": "IMAGES" },
+7 -2
View File
@@ -23,7 +23,11 @@ request → src/worker.ts (mọi route, cả trang tĩnh)
1. denylist: MỘT khoá KV "denylist" = [{cidr, reason?, until?, added?}]
đọc tối đa 1 lần/phút/isolate (+ cacheTtl 60 s ở colo); IP hoặc dải CIDR (IPv4/IPv6),
mục hết hạn bị bỏ qua → khớp ⇒ 403 (trang 09 §2, no-store) · KV lỗi ⇒ bỏ qua (fail-open, có log)
2. app Astro — middleware, nếu path bắt đầu /api/<group>:
2. rate limit toàn site: RL_SITE 100/10s, khoá "site:<ip>" (IPv6 gộp /64), cho MỌI request
tới Worker — trang, /media, /ext-img, /pagefind, /api — trừ host admin.* (đã có Access)
vượt ⇒ 429 + Retry-After: 10, đếm ngược 10 s · binding lỗi ⇒ bỏ qua (fail-open, có log,
không bỏ qua cổng Turnstile)
3. app Astro — middleware, nếu path bắt đầu /api/<group>:
Workers Rate Limiting binding theo nhóm (src/lib/ratelimit.ts), khoá "<group>:<ip>"
(IPv6 gộp theo /64):
report → RL_REPORT 5/60s · gate → RL_GATE 30/60s · admin → RL_ADMIN 120/60s
@@ -32,7 +36,8 @@ request → src/worker.ts (mọi route, cả trang tĩnh)
```
- Một khoá thay vì `denylist:<ip>`: tra cứu tốn như nhau dù IP nào hỏi (một lần đọc/phút/isolate) và chặn được cả dải. Quản lý ở **admin → Bảo mật** (`/admin/security/`): thêm kèm lý do + thời hạn (1 giờ … 30 ngày, hoặc đến khi gỡ), gỡ, có audit log. Không cho chặn dải chứa IP của chính admin — danh sách áp cho mọi host, kể cả admin. Thay đổi có hiệu lực trong khoảng một phút.
- Binding rate limit đếm **theo từng colo** và chỉ cho `period` 10 hoặc 60 s → lớp mềm; lớp cứng là zone rate-limit rule (07 §6). `namespace_id` của mỗi binding chỉ cần duy nhất trong account (981001–981004).
- Binding rate limit đếm **theo từng colo**, gần đúng (burst có thể lọt quá ngưỡng một chút) và chỉ cho `period` 10 hoặc 60 s → lớp mềm; lớp cứng là zone rate-limit rule (07 §6). `namespace_id` của mỗi binding chỉ cần duy nhất trong account (981001–981005). `Retry-After` phải bằng `period` (hằng số trong `src/lib/ratelimit.ts`, test đối chiếu với `wrangler.jsonc`).
- Ba lớp, ngoài vào trong: **WAF rule** (edge, chặn trước khi Worker chạy — không tốn request Worker, trang chặn 1015 của Cloudflare) → **RL_SITE** (ngưỡng thấp hơn WAF để khách thật gặp trang 429 "đẹp" trước) → **nhóm /api** (chặt nhất). Một lượt xem trang tốn 1–20 request tới Worker (file tĩnh `/_astro/*` không qua Worker), nên 100/10s chỉ chặn flood và giữ F5.
- Chặn quốc gia/ASN hay danh sách rất dài: dùng WAF custom rule / IP List (07 §7).
## 3. Form duy nhất: /api/report (bug)
+16 -7
View File
@@ -33,13 +33,22 @@ Zone → Security:
- **WAF → Managed rules**: bật Cloudflare Managed Ruleset (free ruleset).
- Settings → Security Level: Medium · Browser Integrity Check: ON.
## 6. Rate limiting rule (free có 1 rule) — lớp ngoài cho /api/*
## 6. Rate limiting rule (free có 1 rule) — lớp ngoài cho toàn site
Security → WAF → Rate limiting rules → Create:
- Name: `api-outer-limit`
- If: `URI Path starts with /api/`
- Rate: **20 requests / 10 seconds** per IP → Action: **Block**, duration 1 phút (mitigation timeout theo option gói free).
(Lớp trong Worker vẫn đếm 5/phút để trả trang 429 đẹp — rule này chỉ hứng flood thô.)
Gói Free: **1 rule**, biểu thức chỉ dùng được trường **URI Path** và **Verified Bot**, đếm theo **IP**, chu kỳ **10 s**, thời gian chặn **10 s**, không đổi được trang chặn (Cloudflare trả **429**, trang "Error 1015"). Rule chặn ở edge, trước Worker ⇒ flood không tốn request Worker.
Lớp trong Worker (06 §2) đếm thấp hơn — RL_SITE 100/10s cho mọi request tới Worker — để khách thật gặp trang 429 của site trước; rule này chỉ hứng flood thô, nên ngưỡng phải **cao hơn** 100.
Security → Security rules → Rate limiting rules (đang có rule mẫu "Leaked credential check" chiếm slot duy nhất — site không có form mật khẩu nào trên zone, admin đăng nhập qua Access ở `*.cloudflareaccess.com`, nên rule đó không bao giờ khớp: sửa nó thành rule dưới, hoặc xoá rồi tạo mới):
- Name: `site-outer-limit`
- Expression (Edit expression):
`(not starts_with(http.request.uri.path, "/_astro/") and not cf.client.bot)`
— bỏ file tĩnh đã hash (lần tải đầu một trang có thể kéo vài chục file) và bot đã xác thực (Googlebot…; bot AI đã bị chặn ở §5).
- Characteristics: IP · Rate: **150 requests / 10 seconds** · Action: **Block** · Duration: **10 seconds**.
- Gói Free không lọc được theo hostname, nên rule này đếm cả `admin.*` (RL_SITE thì miễn admin). Thư viện ảnh admin tải thumbnail lười (`loading="lazy"`), mỗi màn hình vài chục request — dưới 150.
- Khi bị tấn công: hạ xuống 60–80/10s hoặc đổi Action sang **Managed Challenge**; xong thì trả về.
Kiểm tra: `npm run probe:ratelimit -- https://poli0981.dev/legal/privacy/ 400` — dòng `worker 429` là RL_SITE (trang của site), `waf 429` là rule này (trang 1015). Rule chưa bật ⇒ chỉ thấy `worker 429`.
## 7. Custom rules (free có ~5 rule) — chặn IP/dải/quốc gia khi cần
@@ -134,7 +143,7 @@ Account → Analytics & Logs → **Web Analytics** → site `poli0981.dev` ở c
- [ ] `dig poli0981.dev` ra IP Cloudflare; DNSSEC = Success
- [ ] https:// bắt buộc, chứng chỉ hợp lệ, HSTS header có mặt
- [ ] Bot Fight ON · Managed WAF ON · (AI crawlers block ON nếu chọn)
- [ ] Rate limit rule active — test bằng script bắn 30 req/10s vào /api/ → bị block
- [ ] Rate limit rule active — `npm run probe:ratelimit` (§6) thấy `worker 429` rồi `waf 429`
- [ ] Turnstile hoạt động trên form report (site key đúng domain)
- [ ] Custom domain trỏ Worker, `www` redirect về apex
- [ ] Cron widgets chạy (Workers → widgets → Logs thấy execution)
+5 -4
View File
@@ -17,10 +17,10 @@ Nhắc lại quyết định D8: Custom Errors của Cloudflare cần zone plan
| **404** `404.astro` · `en/404.astro` | route không tồn tại | "Trang này không phát ra tín hiệu." / "This page makes no sound." | + ô search (mở Pagefind — đúng index của ngôn ngữ trang), gợi ý 3 bài mới cùng ngôn ngữ. Assets config `not_found_handling: "404-page"` trả `404.html` gần nhất theo đường dẫn: `/en/…` nhận bản tiếng Anh (`en/404.html`, xem 02 §4), mọi đường dẫn khác nhận bản tiếng Việt |
| **500** `500.astro` | exception khi render on-demand / api | "Có gì đó vỡ trong im lặng." / "Something broke, quietly." | Astro tự dùng 500.astro khi SSR lỗi; api route bọc try/catch trả JSON lỗi gọn. Mã tham chiếu = **Cloudflare Ray ID** của request, worker đóng dấu vào `#ref` (`src/lib/error-ref.ts`) — trùng field `ray` trong Workers Logs, ReportDialog tự đính vào báo lỗi |
| **403** worker entry | IP/dải nằm trong denylist (KV `denylist`, `06` §2) | "Cửa này đang khoá." / "This door is locked." | Kèm link `mailto:contact@` nếu nghĩ là nhầm. Không nêu lý do cụ thể |
| **429** middleware render | vượt rate-limit app (5 req/phút/IP trên /api) | "Chậm lại một nhịp." / "Take a breath." | Header `Retry-After: 60`; hiện đồng hồ đếm ngược 60s (vanilla JS inline) |
| **429** worker entry · middleware | vượt rate-limit toàn site (RL_SITE 100/10s/IP, worker entry) hoặc của một nhóm /api (middleware, `06` §2) | "Chậm lại một nhịp." / "Take a breath." | Header `Retry-After`: 10 (toàn site) / 60 (nhóm /api); đồng hồ đếm ngược đọc số giây trong `#cd` — mặc định 60, worker ghi đè thành 10. Vượt rule WAF (`07` §6) thì gặp trang 1015 của Cloudflare thay vì trang này |
| **/offline** | service worker fallback | "Không có mạng — nhưng vẫn có thứ để làm." / "You're offline." | Xem §3–4 |
403 (worker entry) và 429 (middleware) dùng template Astro đã **prerender sẵn thành static asset** (`/403/` `/429/` là trang thật): chỉ `fetch` asset đó qua binding ASSETS rồi trả với status tương ứng + `no-store` — không nhân đôi markup (đường `astro dev` dùng `rewrite` cũng được bọc lại status).
403 (worker entry) và 429 (worker entry + middleware) dùng template Astro đã **prerender sẵn thành static asset** (`/403/` `/429/` là trang thật): chỉ `fetch` asset đó qua binding ASSETS rồi trả với status tương ứng + `no-store` — không nhân đôi markup (đường `astro dev` dùng `rewrite` cũng được bọc lại status).
Trang xác minh của cổng Turnstile (06 §3b) **không** phải trang lỗi: nó là chính trang được yêu cầu (status 200, `<head>` thật) với `<body>` thay bằng thẻ xác minh. `/offline/` đứng ngoài cổng để service worker luôn precache được.
@@ -45,7 +45,8 @@ Canvas ≤ 6KB JS vanilla nằm ngay trong trang offline: đầu lâu pixel nh
| 404 asset | mở `/anh-khong-co.png` | trang 404 (not_found_handling) |
| 500 | tạm ném throw trong 1 api route ở preview | trang/JSON 500 + mã tham chiếu |
| 403 | thêm IP nhà vào denylist KV preview → mở site | trang 403 custom |
| 429 | bắn 6 POST /api/report trong 1 phút | request 6 nhận trang 429 + Retry-After |
| 429 lớp zone | bắn 30 req/10s | Cloudflare block (trang mặc định) — xác nhận rule sống |
| 429 | bắn 6 POST /api/report trong 1 phút | request 6 nhận trang 429 + Retry-After: 60 |
| 429 toàn site | `npm run probe:ratelimit` (> 100 req/10s) | trang 429, Retry-After: 10, đếm ngược 10 (dòng `worker 429`) |
| 429 lớp zone | `npm run probe:ratelimit -- <url> 400 40` (> 150 req/10s) | trang 1015 của Cloudflare (dòng `waf 429`) — xác nhận rule sống |
| Offline | DevTools → Network Offline → điều hướng | trang /offline + game chạy |
| Mobile | lặp các case trên ở viewport 390px | không vỡ layout |
+53
View File
@@ -0,0 +1,53 @@
// Spam a URL from this machine and report who answered 429: the Worker's site-wide limit
// (RL_SITE — the site's own 429 page, Retry-After: 10) or the zone WAF rate-limiting rule
// (Cloudflare's "Error 1015" page, before the Worker runs). Docs 06 §2 and 07 §6.
//
// npm run probe:ratelimit -- [url] [requests] [parallel]
// defaults: https://poli0981.dev/legal/privacy/ 300 20
//
// The default page skips the Turnstile gate, so a 200 is the real page. Every request spends
// this address's budget: expect 429s from the site for the next ~10 seconds.
const url = process.argv[2] ?? "https://poli0981.dev/legal/privacy/";
const total = Number(process.argv[3] ?? 300);
const parallel = Number(process.argv[4] ?? 20);
/** "worker 429" | "waf 429" | the status code as text. */
async function probe() {
const response = await fetch(url, {
redirect: "manual",
headers: { "user-agent": "poli0981-ratelimit-probe" },
signal: AbortSignal.timeout(15000),
});
const body = await response.text();
if (response.status !== 429) return String(response.status);
// The site's 429 page carries the countdown span; the WAF's block page never does.
return body.includes('id="cd"') ? "worker 429" : "waf 429";
}
const results = [];
const firstAt = {};
const started = Date.now();
let next = 0;
await Promise.all(
Array.from({ length: parallel }, async () => {
while (next < total) {
const index = next++;
const kind = await probe().catch((error) => `error (${error.name})`);
results.push(kind);
firstAt[kind] = Math.min(firstAt[kind] ?? Infinity, index + 1);
}
}),
);
const seconds = ((Date.now() - started) / 1000).toFixed(1);
console.log(`${total} requests to ${url} in ${seconds}s (${parallel} in parallel)`);
const counts = Object.groupBy(results, (kind) => kind);
for (const [kind, hits] of Object.entries(counts)) {
console.log(
` ${kind.padEnd(12)} ${String(hits.length).padStart(5)} first at #${firstAt[kind]}`,
);
}
if (!counts["worker 429"] && !counts["waf 429"]) {
console.log("no 429 at all — the limits were not reached (more requests or more parallel?)");
}
+21 -1
View File
@@ -8,7 +8,8 @@
// with Cloudflare's always-pass test secret, so the whole flow — gate page, exemptions,
// /api/gate → pass cookie → real page — is exercised end to end (siteverify is a real
// network call; the test secret accepts any token). Phase 3 lists the client's address in
// the KV denylist and expects 403s everywhere, pages included.
// the KV denylist and expects 403s everywhere, pages included. Phase 4 spams a page until the
// site-wide rate limit answers with the 429 page.
//
// Usage: npm run build && npm run smoke (works on Linux, macOS and Windows)
import { execFileSync, spawn } from "node:child_process";
@@ -384,6 +385,25 @@ try {
rmSync(tmp, { recursive: true, force: true });
}
console.log("— phase 4: spam until the site-wide rate limit answers");
// Last of all: the simulated counters outlive a `wrangler dev` restart, so a spent budget
// would turn the next phase into 429s. RL_SITE allows 100 per 10 s; keep going until a 429
// arrives — a burst can straddle two windows.
await withWorker(8791, {}, async (get) => {
let limited = null;
for (let i = 0; i < 400 && !limited; i++) {
const response = await get("/legal/privacy/");
if (response.status === 429) limited = response;
else await response.body?.cancel();
}
const html = limited ? await limited.text() : "";
expect("spam → 429 from the site-wide limit", limited !== null);
expect("429 sends Retry-After: 10", limited?.headers.get("retry-after") === "10");
expect("429 is no-store", limited?.headers.get("cache-control") === "no-store");
expect("429 has nosniff", limited?.headers.get("x-content-type-options") === "nosniff");
expect("429 countdown starts at 10", html.includes('<span id="cd">10</span>'));
});
if (failures.length) {
console.error(`\nsmoke test failed (${failures.length}): ${failures.join(", ")}`);
process.exit(1);
+1 -1
View File
@@ -4,7 +4,7 @@ sonar.organization=poli0981
sonar.projectKey=poli0981_poli0981-dev
sonar.projectName=poli0981-dev
# Bumped with each release tag; "previous version" new-code periods start from it.
sonar.projectVersion=2.0.0
sonar.projectVersion=2.1.0
sonar.sourceEncoding=UTF-8
# .github is analyzed too: the workflow rules (pinned actions, no npx/install scripts) are
+4
View File
@@ -5,8 +5,12 @@ import { env } from "cloudflare:workers";
// RL_API. /api/widgets is never limited: its response is identical for every visitor
// and edge-cached (api/widgets.ts), and the home page fetches it on every view.
/** The /api groups' window (wrangler.jsonc `ratelimits`), sent as Retry-After. */
export const RATE_LIMIT_WINDOW_SECONDS = 60;
/** RL_SITE's window: the site-wide limit src/worker.ts applies before any route. */
export const SITE_RATE_LIMIT_SECONDS = 10;
type RateLimiter = { limit(options: { key: string }): Promise<{ success: boolean }> };
const BINDING_FOR_GROUP: Record<string, string> = {
+2 -1
View File
@@ -6,14 +6,15 @@ import ErrorLayout from "@/layouts/ErrorLayout.astro";
<p class="code">429</p>
<h1>Chậm lại một nhịp.</h1>
<p class="echo" lang="en">Take a breath.</p>
{/* 60 s = the /api groups' window; the Worker rewrites it for the site-wide limit (10 s). */}
<p class="code">Thử lại sau / Retry in <span id="cd">60</span>s</p>
{
/* Processed (not `is:inline`): body-level, so the <meta> CSP governs it and only a
bundled script gets a hash. Deferring costs nothing for a 1s-tick countdown. */
}
<script>
let n = 60;
const el = document.getElementById("cd");
let n = Number(el?.textContent) || 60;
const t = setInterval(() => {
n -= 1;
if (el) el.textContent = String(n);
+57 -1
View File
@@ -8,6 +8,7 @@ import { renderGate } from "./lib/gate/render";
import { routeHosts } from "./lib/hosts";
import { log, requestFields } from "./lib/log";
import { serveMedia } from "./lib/media/serve";
import { clientSubject, SITE_RATE_LIMIT_SECONDS } from "./lib/ratelimit";
import { withSecurityHeaders } from "./lib/security-headers";
import { versionInfo } from "./lib/version";
@@ -54,6 +55,61 @@ async function denied(request: Request, env: Env): Promise<Response | null> {
return withSecurityHeaders(response);
}
/**
* The site-wide rate limit (RL_SITE, docs 06 §2): one budget per client for every request
* the Worker sees — pages, /media, /ext-img, /pagefind and the API, whose groups then apply
* their own stricter limits in the middleware. Over it → the 429 page with Retry-After and
* its countdown set to the window. The admin host is exempt: Cloudflare Access stands in
* front of it, its API is limited per account (RL_ADMIN) and the media library loads many
* thumbnails at once. Fails OPEN like the denylist, and never throws: the edge fail-open
* below would skip the gate too. The zone WAF rule (docs 07 §6) is the hard limit.
*/
async function rateLimited(request: Request, url: URL, env: Env): Promise<Response | null> {
const ip = request.headers.get("cf-connecting-ip");
// Typed as always bound, but unit tests (and any env without the binding) have none.
const limiter = env.RL_SITE as RateLimit | undefined;
if (!ip || !limiter || url.hostname.startsWith("admin.")) return null;
try {
if ((await limiter.limit({ key: `site:${clientSubject(ip)}` })).success) return null;
} catch (error) {
log.error("ratelimit.unavailable", { ...requestFields(request), error });
return null;
}
log.warn("ratelimit.exceeded", { ...requestFields(request), group: "site" });
const seconds = String(SITE_RATE_LIMIT_SECONDS);
try {
const page = await env.ASSETS.fetch(new URL("/429/", request.url));
const response = new Response(page.body, { status: 429, headers: page.headers });
response.headers.set("Cache-Control", "no-store");
response.headers.set("Retry-After", seconds);
// The body is rewritten below.
for (const header of ["ETag", "Last-Modified", "Content-Length"]) {
response.headers.delete(header);
}
const stamped = new HTMLRewriter()
.on("#cd", {
element(el) {
el.setInnerContent(seconds);
},
})
.transform(response);
return withSecurityHeaders(stamped);
} catch (error) {
// Still a 429 — falling through to the edge fail-open would serve the page ungated.
log.error("ratelimit.page_failed", { ...requestFields(request), error });
return withSecurityHeaders(
new Response("Too many requests", {
status: 429,
headers: {
"content-type": "text/plain; charset=utf-8",
"cache-control": "no-store",
"retry-after": seconds,
},
}),
);
}
}
/**
* The Turnstile gate (docs 06 §3b): unless the request carries a valid 48-hour pass, a
* page answers with the gate instead of its content. Returns the response to send, or
@@ -103,7 +159,7 @@ export default {
return Response.redirect(url.toString(), 301);
}
const blocked = await denied(request, env);
const blocked = (await denied(request, env)) ?? (await rateLimited(request, url, env));
if (blocked) return blocked;
const hostRoute = routeHosts(url);
+18 -1
View File
@@ -1,10 +1,17 @@
// The small libraries: rate limiting, Turnstile, headers, bodies, logs, version, denylist,
// formatting, links, SEO, RSS, OG, i18n and nav.
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import { afterEach, describe, it } from "node:test";
import { setWorkerEnv } from "./support/stubs/cloudflare-workers.ts";
import { captureLogs, FakeKV, FakeRateLimiter, jsonResponse, mockFetch } from "./support/fakes.ts";
import { allowRequest, apiGroup, clientSubject } from "../src/lib/ratelimit.ts";
import {
allowRequest,
apiGroup,
clientSubject,
RATE_LIMIT_WINDOW_SECONDS,
SITE_RATE_LIMIT_SECONDS,
} from "../src/lib/ratelimit.ts";
import { verifyTurnstile } from "../src/lib/turnstile-server.ts";
import { applySecurityHeaders, withSecurityHeaders } from "../src/lib/security-headers.ts";
import {
@@ -84,6 +91,16 @@ describe("rate limiting", () => {
setWorkerEnv({});
assert.equal(await allowRequest("gate", "a"), true);
});
it("sends a Retry-After equal to each binding's period in wrangler.jsonc", async () => {
const config = await readFile(new URL("../wrangler.jsonc", import.meta.url), "utf8");
const period = (name: string) =>
Number(new RegExp(`"name": "${name}"[^}]*"period": (\\d+)`).exec(config)?.[1]);
assert.equal(period("RL_SITE"), SITE_RATE_LIMIT_SECONDS);
for (const name of ["RL_REPORT", "RL_GATE", "RL_ADMIN", "RL_API"]) {
assert.equal(period(name), RATE_LIMIT_WINDOW_SECONDS, name);
}
});
});
describe("Turnstile siteverify", () => {
+23 -19
View File
@@ -306,12 +306,15 @@ export function fakeAssets(pages: Record<string, AssetPage>) {
export class FakeRateLimiter {
readonly counts = new Map<string, number>();
readonly limitPerKey: number;
/** When set, every call throws it (the binding failing). */
failWith: Error | null = null;
constructor(limitPerKey: number) {
this.limitPerKey = limitPerKey;
}
async limit({ key }: { key: string }): Promise<{ success: boolean }> {
if (this.failWith) throw this.failWith;
const count = (this.counts.get(key) ?? 0) + 1;
this.counts.set(key, count);
return { success: count <= this.limitPerKey };
@@ -387,10 +390,10 @@ interface FakeElement {
}
/**
* Just enough of workerd's HTMLRewriter for src/lib/gate/render.ts and src/lib/error-ref.ts:
* the selectors they use (`html`, `head script[src]`, `link[rel="modulepreload"]`,
* `script[type="application/ld+json"]`, `body`, `#ref`) applied to the whole document with
* string surgery.
* Just enough of workerd's HTMLRewriter for src/lib/gate/render.ts, src/lib/error-ref.ts and
* the 429 countdown in src/worker.ts: the selectors they use (`html`, `head script[src]`,
* `link[rel="modulepreload"]`, `script[type="application/ld+json"]`, `body`, `#<id>`)
* applied to the whole document with string surgery.
*/
export class FakeHTMLRewriter {
private readonly handlers: [string, ElementHandlers][] = [];
@@ -459,6 +462,22 @@ function applySelector(
return out + html.slice(copied);
};
if (selector.startsWith("#")) {
const id = html.indexOf(` id="${selector.slice(1)}"`);
if (id === -1) return html;
const open = html.indexOf(">", id) + 1;
const close = html.indexOf("</", open);
let inner = html.slice(open, close);
handler({
getAttribute: () => null,
remove() {},
setInnerContent: (content) => {
inner = content.replaceAll("&", "&amp;").replaceAll("<", "&lt;").replaceAll(">", "&gt;");
},
});
return html.slice(0, open) + inner + html.slice(close);
}
switch (selector) {
case "html": {
const tag = /<html[^>]*>/.exec(html)?.[0] ?? "";
@@ -474,21 +493,6 @@ function applySelector(
return replaceTags(/<link\b[^>]*rel="modulepreload"[^>]*>/g);
case 'script[type="application/ld+json"]':
return replaceScripts((tag) => attribute(tag, "type") === "application/ld+json");
case "#ref": {
const id = html.indexOf(' id="ref"');
if (id === -1) return html;
const open = html.indexOf(">", id) + 1;
const close = html.indexOf("</", open);
let inner = html.slice(open, close);
handler({
getAttribute: () => null,
remove() {},
setInnerContent: (content) => {
inner = content.replaceAll("&", "&amp;").replaceAll("<", "&lt;").replaceAll(">", "&gt;");
},
});
return html.slice(0, open) + inner + html.slice(close);
}
case "body":
return html.replace(
/(<body[^>]*>)[\s\S]*(<\/body>)/,
+113
View File
@@ -11,6 +11,7 @@ import {
fakeContext,
FakeKV,
FakeR2,
FakeRateLimiter,
installCaches,
installHTMLRewriter,
type AssetPage,
@@ -25,6 +26,15 @@ const PAGES: Record<string, AssetPage> = {
"/": { body: HOME },
"/en/": { body: HOME.replace('lang="vi"', 'lang="en"') },
"/403/": { status: 200, body: "<html><body>403 page</body></html>" },
"/429/": {
body: `<html><body><p>429 page, retry in <span id="cd">60</span>s</p></body></html>`,
headers: {
"content-type": "text/html; charset=utf-8",
etag: '"asset-etag"',
"last-modified": "Mon, 26 Oct 2026 00:00:00 GMT",
"content-length": "72",
},
},
"/404.html": { body: `<html lang="vi"><body>404 page</body></html>` },
"/en/404.html": { body: `<html lang="en"><body>404 page</body></html>` },
"/data.json": { body: "{}", headers: { "content-type": "application/json" } },
@@ -259,6 +269,109 @@ describe("worker: IP denylist", () => {
});
});
describe("worker: site-wide rate limit", () => {
it("answers spam with the 429 page, Retry-After and its countdown set to the window", async () => {
const limiter = new FakeRateLimiter(3);
const statuses: number[] = [];
for (let i = 0; i < 5; i++) {
statuses.push((await run("https://poli0981.dev/", {}, { RL_SITE: limiter })).response.status);
}
assert.deepEqual(statuses, [200, 200, 200, 429, 429]);
const { env } = makeEnv({ RL_SITE: limiter });
const { result, logs } = await captureLogs(() =>
fetchWorker(
request("https://poli0981.dev/"),
env,
fakeContext() as unknown as ExecutionContext,
),
);
assert.equal(result.status, 429);
assert.equal(result.headers.get("retry-after"), "10");
assert.equal(result.headers.get("cache-control"), "no-store");
for (const header of ["etag", "last-modified", "content-length"]) {
assert.equal(result.headers.get(header), null, header);
}
assert.equal(result.headers.get("x-content-type-options"), "nosniff");
assert.match(await result.text(), /429 page, retry in <span id="cd">10<\/span>s/);
assert.equal(logs[0]?.event, "ratelimit.exceeded");
assert.equal(logs[0]?.group, "site");
});
it("spends one budget per client on every route, IPv6 by its /64", async () => {
const limiter = new FakeRateLimiter(100);
const from = (ip: string) => ({ headers: { "cf-connecting-ip": ip } });
await run("https://poli0981.dev/", from("2001:db8:0:1::1"), { RL_SITE: limiter });
await run("https://poli0981.dev/blog/", from("2001:db8:0:1::2"), { RL_SITE: limiter });
await run("https://poli0981.dev/api/health", {}, { RL_SITE: limiter });
await run("https://poli0981.dev/media/v1/x/webp-800", {}, { RL_SITE: limiter });
await run("https://poli0981.dev/pagefind/pagefind.js", {}, { RL_SITE: limiter });
assert.deepEqual(Object.fromEntries(limiter.counts), {
"site:2001:db8:0:1::/64": 2,
"site:198.51.100.7": 3,
});
});
it("is checked before the gate, and a limited client never sees the page", async () => {
const limiter = new FakeRateLimiter(0);
const { response } = await run("https://poli0981.dev/", {}, { ...gateOn, RL_SITE: limiter });
assert.equal(response.status, 429);
assert.doesNotMatch(await response.text(), /real page|data-human-check/);
});
it("leaves the admin host and requests without a client address alone", async () => {
const limiter = new FakeRateLimiter(0);
const admin = await run("https://admin.poli0981.dev/admin/", {}, { RL_SITE: limiter });
assert.match(await admin.response.text(), /from the Astro handler/);
const { env } = makeEnv({ RL_SITE: limiter });
const anonymous = await fetchWorker(
new Request("https://poli0981.dev/"),
env,
fakeContext() as unknown as ExecutionContext,
);
assert.equal(anonymous.status, 200);
assert.equal(limiter.counts.size, 0);
});
it("fails open when the limiter breaks — without skipping the gate", async () => {
const limiter = new FakeRateLimiter(100);
limiter.failWith = new Error("rate limiter down");
const { env } = makeEnv({ ...gateOn, RL_SITE: limiter });
const { result, logs } = await captureLogs(() =>
fetchWorker(
request("https://poli0981.dev/"),
env,
fakeContext() as unknown as ExecutionContext,
),
);
assert.equal(result.status, 200);
assert.match(await result.text(), /data-human-check/);
assert.equal(logs[0]?.event, "ratelimit.unavailable");
});
it("stays a 429 when the page can't be fetched — never the ungated page", async () => {
const { env } = makeEnv({ ...gateOn, RL_SITE: new FakeRateLimiter(0) });
(env as unknown as { ASSETS: { fetch: () => never } }).ASSETS.fetch = () => {
throw new Error("assets down");
};
const { result, logs } = await captureLogs(() =>
fetchWorker(
request("https://poli0981.dev/"),
env,
fakeContext() as unknown as ExecutionContext,
),
);
assert.equal(result.status, 429);
assert.equal(result.headers.get("retry-after"), "10");
assert.equal(result.headers.get("cache-control"), "no-store");
assert.equal(await result.text(), "Too many requests");
assert.deepEqual(
logs.map((line) => line.event),
["ratelimit.exceeded", "ratelimit.page_failed"],
);
});
});
describe("worker: Turnstile gate", () => {
it("answers a page view without a pass with the gate", async () => {
const { response } = await run("https://poli0981.dev/", {}, gateOn);
+7 -1
View File
@@ -64,8 +64,14 @@
// Workers Rate Limiting (docs 06 §2) — replaces the KV read-then-write counter. The
// period must be 10 or 60 s; counters are per Cloudflare location, so these are
// per-colo soft limits (the zone WAF rule in docs 07 §6 stays the hard outer limit).
// namespace_id only has to be unique within the account.
// namespace_id only has to be unique within the account. A period here must match its
// Retry-After in src/lib/ratelimit.ts (tests/lib.test.ts checks).
"ratelimits": [
// Site-wide, per client, for every request the Worker sees (src/worker.ts): pages,
// /media, /ext-img, /pagefind and the API. A page view costs 1–20 of these (static
// assets never reach the Worker), so 100 per 10 s only stops floods and F5-spam; the
// /api groups below are the strict ones. The WAF rule sits above it (docs 07 §6).
{ "name": "RL_SITE", "namespace_id": "981005", "simple": { "limit": 100, "period": 10 } },
{ "name": "RL_REPORT", "namespace_id": "981001", "simple": { "limit": 5, "period": 60 } },
// Generous: one carrier-grade-NAT address can front many phones on VN mobile networks.
{ "name": "RL_GATE", "namespace_id": "981002", "simple": { "limit": 30, "period": 60 } },