Files
poli0981andClaude Fable 5.2 0e0e442de5 chore: maintenance mode, Qodana analysis; 4.6.1 (#188) [skip-discuss]
The project moves to maintenance mode (stable, feature-complete): docs, templates and badges (EN + VI). Qodana analysis in CI for the TypeScript / JavaScript, Python and Rust projects, with the findings of its first reports fixed. Version 4.6.1.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 21:44:32 +07:00

25 KiB
Raw Permalink Blame History

Operations runbook (maintainer)

How freeitchgames.win is wired together and how to set it up from scratch. Nothing in this file is secret: secrets are only ever typed into Cloudflare (wrangler secret put) or a local, gitignored webapp/.dev.vars.

Architecture in one screen

Piece Where What it does
Worker free-games-itchio-list Cloudflare Workers, custom domain freeitchgames.win Serves the built app + the catalog packs /data/pack/* (static assets), the readable /data/*.json to apps 4.0–4.1.1 only (legacy-data.ts), /img (resized covers), the verification gate (/api/verify), /api/suggest, /api/admin/*, /admin, and runs the RSS cron. Config: webapp/wrangler.jsonc.
Workers Builds Cloudflare dashboard → the Worker → Settings → Build Builds and deploys main on every push (data commits included).
R2 freeitchgames-thumbs Cloudflare R2 Resized WebP covers (w160/…, w640/…).
D1 freeitchgames Cloudflare D1 Review queue, feed state, audit log, automation settings, followed collections (webapp/worker/migrations/; the idempotency table of the retired ingest API stays, empty).
Access app "Admin" Cloudflare Zero Trust Login for /admin + /api/admin/*.
Turnstile widget Cloudflare Turnstile The verification gate in front of the web app (action gate) and anti-spam on /suggest (action suggest).
Transform Rule "Verified bots skip the gate" Cloudflare → Rules → Transform Rules Adds x-fig-known-bot: <GATE_BOT_TOKEN> to requests from Cloudflare-verified bots, so search engines and link previews skip the gate (section 1, step 14).
Rate limits WAF rule (Security → WAF → Rate limiting rules) + the Worker's ratelimits bindings Two layers of per-IP limits: the WAF rule stops floods before they run (and bill) the Worker; the Worker answers what still gets through with 429 per kind of traffic (section 1, step 15).
GitHub App GitHub → Settings → Developer settings The Worker's identity for commits (queue approvals, edits, removals).
GitHub Actions .github/workflows/ Data pipeline (refresh.yml, update.yml, force_update.yml), CI, releases, build monitor.
Staging Worker free-games-itchio-list-staging staging.freeitchgames.win (whole host behind Access) Optional pre-release checks, deployed by hand from any branch with npx wrangler deploy --env staging (wrangler.jsonc → env.staging).

1. Cloudflare

Do these once (and again with the -staging names for the staging Worker).

  1. Build settings (Worker → Settings → Build): root directory webapp; build command npm run build; deploy command npx wrangler deploy; build variable NODE_VERSION=26 (no VITE_CF_BEACON_TOKEN: the zone injects Web Analytics, step 10). The staging Worker, if you use it, is deployed by hand (npx wrangler deploy --env staging); turn its builds for non-production branches off (otherwise every push to main also gets a preview build there).

  2. Domains & routes (Worker → Settings → Domains & Routes): keep the custom domains freeitchgames.win and www.freeitchgames.win; delete any *freeitchgames.win/* route; a Redirect Rule sends www to the apex. workers.dev and preview URLs stay off (wrangler.jsonc sets them; those hostnames would bypass Access).

  3. R2: enable R2 for the account in the dashboard first (R2 Object Storage; the free tier still asks for a payment method), then cd webapp && npx wrangler r2 bucket create freeitchgames-thumbs (and freeitchgames-thumbs-staging). A deploy fails while its bucket is missing.

  4. Images → Transformations: enable for the zone; allowed source origins: img.itch.zone only.

  5. D1: production freeitchgames exists (created 2026-09-18 in APAC with npx wrangler d1 create freeitchgames; its id is in wrangler.jsonc; 0001_init.sql applied, 0002_automation.sql (4.4: ingest outcomes, settings, followed collections) applied before the 4.4.0 merge). Staging: cd webapp && npx wrangler d1 create freeitchgames-staging, put the printed id in env.staging.d1_databases, then npx wrangler d1 migrations apply freeitchgames-staging --remote --env staging. Apply every new migration by hand (npx wrangler d1 migrations apply freeitchgames --remote) before deploying code that needs it.

  6. Zero Trust (free plan is enough):

    • Settings → Authentication: add GitHub (OAuth App callback https://<team>.cloudflareaccess.com/cdn-cgi/access/callback) and One-time PIN.
    • Access → Applications → Self-hosted "Admin": freeitchgames.win/admin, freeitchgames.win/admin/*, freeitchgames.win/api/admin/*; policy Allow → your email(s); session 24 h. Copy its AUD tag.
    • 4.4.1 removed the browser extension's ingest API: delete the old self-hosted "Ingest" app (freeitchgames.win/api/ingest) and its service token (Access → Service credentials) if they still exist.
    • Staging: one self-hosted app covering all of staging.freeitchgames.win.
  7. Turnstile: add a widget (Managed) for freeitchgames.win and staging.freeitchgames.win. The same widget serves the verification gate (action gate) and the Suggest form (action suggest); the Worker checks the token, the hostname and the action, so a token from one can't be used for the other.

  8. Fill in wrangler.jsonc vars (not secret) and commit: ACCESS_TEAM_DOMAIN (https://<team>.cloudflareaccess.com, no trailing slash), ACCESS_AUD_ADMIN, GH_APP_ID, GH_APP_INSTALLATION_ID, TURNSTILE_SITEKEY — in both the top level and env.staging. Then npm run cf:types. The production values are filled in (2026-09-18). Change them in the file, not only in the dashboard: every deploy — so every data commit — replaces the Worker's plain-text variables with the file's values. Secrets are not affected.

  9. Secrets (cd webapp; add --env staging for staging):

    # The Worker converts GitHub's PKCS#1 key itself, so the downloaded .pem works as is:
    npx wrangler secret put GH_APP_PRIVATE_KEY < app.private-key.pem
    npx wrangler secret put ADMIN_EMAILS        # comma-separated
    npx wrangler secret put TURNSTILE_SECRET
    npx wrangler secret put GATE_BOT_TOKEN      # the verification gate, step 14
    openssl rand -base64 48 | npx wrangler secret put GATE_SECRET   # last: this switches the gate on
    

    Until a feature's settings exist it answers 503 (the gate simply stays off); the public site works without any of them.

  10. Web Analytics: the site uses the automatic setup — the zone injects the beacon into every HTML response (checked on 2026-09-24), so the build needs no token. Don't also set VITE_CF_BEACON_TOKEN: every page view would count twice. The CSPs of the app, the admin page and the verification page allow the beacon.

  11. Contact email: the policies publish contact@, legal@, privacy@, security@ and takedown@poli0981.dev — aliases of the maintainer's Google Workspace mailbox (MX / SPF / DKIM / DMARC for poli0981.dev live in that zone's Cloudflare DNS). freeitchgames.win needs no mail setup.

  12. Zone settings: SSL/TLS Full (strict), Always Use HTTPS, minimum TLS 1.2, TLS 1.3 on; Smart Tiered Cache; Browser Cache TTL "Respect existing headers"; Rocket Loader and Email Obfuscation off (they rewrite HTML and break the CSP); WAF managed rules on; the rate-limiting rule of step 15; Bot Fight Mode off (it can't be skipped for the apps; the verification gate covers the web app instead). Keep HSTS preload only if every subdomain (including staging) will always be HTTPS.

    • Measured 2026-10-03: TLS 1.0/1.1 are refused, TLS 1.3 and HTTP/3 answer, http:// redirects (301), and the zone sends Strict-Transport-Security: max-age=31536000; includeSubDomains; preload. The domain is not on the browsers' HSTS preload list (hstspreload.org); submitting it is optional and slow to undo.
    • Smart Tiered Cache is free, so leave it on, but it barely matters here: static assets (the app and /data/pack/*) have Cloudflare's own tiered asset cache, and /img uses the Worker's Cache API (per data centre, never tiered) plus R2. Only cacheable Worker fetch() subrequests use it — in practice the img.itch.zone fallback fetch. No Region Hint is needed for api.github.com (it shows up as an "origin" because the Worker calls it; those calls aren't cacheable).
    • Don't turn on Workers Cache (cache.enabled in wrangler.jsonc): it bills every static-asset request as a Worker request and would serve cached /img responses without running the verification gate.
    • Compression is automatic (zstd / Brotli / gzip, picked per browser) for JSON and the app files. The .bin packs are deflated at build time and served as application/octet-stream, which Cloudflare does not compress again.
  13. Notifications: enable the Workers Builds failure notification (independent of deploy-status.yml).

  14. Verification gate (webapp/worker/gate.ts). A visitor must pass a Turnstile check before the app shell and /img are served; the pass is an HttpOnly cookie good for BOT_GATE_TTL_HOURS (1–168, default 48; lowering it also shortens passes already given out). /data, /assets, robots, sitemap, the social image, /api/* and /admin* stay open, and the desktop/Android apps are not affected. BOT_GATE is "on" in wrangler.jsonc, but the gate stays off (fails open) until both secrets below exist. In this order:

    1. Check the Workers plan: / now runs the Worker too (on the Free plan, Worker-first paths answer 429 once the day's 100,000 requests are used up; /img already uses most of them).
    2. Generate the bot token (openssl rand -hex 32), keep it in your password manager, and npx wrangler secret put GATE_BOT_TOKEN it.
    3. Rules → Transform Rules → Modify Request Header → create "Verified bots skip the gate": custom filter (cf.client.bot) — or (cf.client.bot and not cf.verified_bot_category in {"AI Crawler"}) to keep AI crawlers out — then Set static header x-fig-known-bot to the token.
    4. Check that the header reaches the Worker: add a temporary rule (ip.src eq <your IP>) that sets the same header, then curl -s https://freeitchgames.win/api/verify must say "via":"bot". Delete the temporary rule. If it doesn't say so, stop here and don't set GATE_SECRET.
    5. openssl rand -base64 48 | npx wrangler secret put GATE_SECRET — the gate is on.
    6. Smoke test: curl -sI https://freeitchgames.win/ shows x-robots-tag: noindex, nofollow and cache-control: no-store (the verification page); /img/160/x answers 403; /data/pack/manifest-v1.json answers 200 with access-control-allow-origin: * (the catalog checks in section 4 cover the rest); curl -sI -A 'Googlebot/2.1' https://freeitchgames.win/ answers 503 (only verified bots, via the rule, get through); a real browser passes and lands on the page it opened; Search Console → URL Inspection → Test live URL renders the real app. Watch Security → Events and Search Console for a week.

    Off switch: npx wrangler secret delete GATE_SECRET turns the gate off at once, and data-commit deploys don't bring it back; to keep it off, set BOT_GATE to "off" in wrangler.jsonc. Staging: set TURNSTILE_SITEKEY and BOT_GATE: "on" under env.staging and add both secrets with --env staging. Locally: put the Turnstile test keys, SITE_ORIGIN=http://localhost:8787 and a 32+ character GATE_SECRET in webapp/.dev.vars, then npm run build && npx wrangler dev (on localhost the cookie is fig_gate, without Secure, and no bot token is needed).

  15. Rate limits — two layers, both per client IP (an IPv6 client by its /64 in the Worker).

    • Layer 1, the zone's WAF rule (Security → WAF → Rate limiting rules). It runs before the Worker, so a request it blocks costs no Worker request, it is the only limit on static files, and it counts across the whole data centre — the only limit a flood of new connections can't spread out of (see layer 2). On the Free plan there is one rule, counted by IP over 10 seconds, with a 10-second block; if a rule already exists (the old one for /api/*), edit it rather than adding one. Settings:
      • Rule name Flood guard. If incoming requests match → Edit expression:
        (not cf.client.bot and not starts_with(http.request.uri.path, "/assets/") and not starts_with(http.request.uri.path, "/data/pack/"))
        
        (In the builder: Verified Bot equals off, URI Path does not start with /assets/, URI Path does not start with /data/pack/, joined with And. If the editor refuses starts_with, use not http.request.uri.path wildcard r"/assets/*" and the same for /data/pack/*.) Bundles and packs stay out: they are free static assets with a long cache, and a first visit loads a few dozen of them at once. Verified bots stay out like at the gate.
      • With the same characteristics: IP. When rate exceeds: 300 requests per 10 seconds (30 a second; a person scrolling the Games list fast loads about 100 covers in that time). Then take action: Block, default response (429, Cloudflare's "error 1015" page), for 10 seconds. Deploy.
      • Lower it once Security → Events shows what real traffic looks like; don't go below ~150 per 10 s (a Games page asks for up to 100 covers).
    • Layer 2, the Worker (webapp/worker/ratelimit.ts, ratelimits in wrangler.jsonc, deployed with every build — nothing to set up). Every request that runs the Worker counts, per 60 s and per Cloudflare location: RL_SITE 120 (app pages, the public /api, the readable /data JSON, misses), RL_IMG 600 (/img covers), RL_ADMIN 300 (/admin + /api/admin); on top, RL_SUGGEST 5 (Suggest POSTs) and RL_GATE 20 (verification POSTs). Over a limit: 429 + Retry-After: 60 — JSON {"error":"rate_limited"} for the API, covers and files (with CORS on /data), a small "Too many requests" page (EN/VI, no script) for app pages. Verified bots (the gate's header) are never counted. These counts are loose: each machine of a data centre counts on its own and syncs late, so they hold for a client that keeps one connection (browsers use one HTTP/2 connection, scripts with keep-alive) but hardly for one that opens a new connection per request — measured after the 4.5.0 deploy, 1,000 such requests in a minute got no 429. That is the WAF rule's job. Change a limit in wrangler.jsonc (then npm run cf:types); keep period 60 and a new namespace_id per binding in 88xx (staging 89xx) — the ids are account-wide, and the account's other Workers use the low numbers. The counts are approximate and fail open (a missing binding or a limiter error lets requests through).
    • Test after a change (it blocks your own IP for up to a minute). The Worker's limits, over one connection (a single curl with many URLs reuses it):
      curl -s -o /dev/null -w "%{http_code}\n" $(printf 'https://freeitchgames.win/api/health %.0s' $(seq 200)) | sort | uniq -c
      curl -s https://freeitchgames.win/api/health   # {"error":"rate_limited"} → the Worker; "error code: 1015" → the WAF rule
      
      About 120 answer 200 and the rest 429 (fewer pass if you sent requests in the last minute). The WAF rule needs more than 300 requests within 10 s — faster than xargs curl gets from a home connection (~20 a second). Node's fetch keeps a pool of connections and does it:
      node -e "let n=0,t={};const go=async()=>{while(n++<800){const r=await fetch('https://freeitchgames.win/api/health');const b=await r.text();const k=r.status==200?'200':b.includes('1015')?'WAF 1015':b.includes('rate_limited')?'Worker 429':r.status;t[k]=(t[k]||0)+1}};Promise.all(Array.from({length:100},go)).then(()=>console.log(t))"
      
      Measured on 2026-10-27 with the rule above: 800 requests in 4.9 s → 111 × 200, 201 × Worker 429, 488 × WAF 1015 (from 1.4 s on), and the site answered normally again 10 s later. Security → Events lists the WAF blocks; Worker 429s show up in Workers Logs (sampled, head_sampling_rate 0.1).

2. GitHub

  1. GitHub App (Settings → Developer settings → GitHub Apps → New): name e.g. freeitchgames-admin; webhook off; repository permissions Contents: Read and write, Metadata: Read; install it on this repository only. Note the App ID and the installation id (the number in the installation's URL); generate a private key (see step 9 above).
  2. Repository settings: homepage https://freeitchgames.win; private vulnerability reporting on; Dependabot security updates on; CodeQL default setup and secret scanning if you want them (the policies don't claim them). Rulesets: target the default branch (block deletion and force pushes); protect tags v*. Merge PRs with squash or a merge commit — never fast-forward.
  3. Secrets: keep DISCORD_CI_WEBHOOK, the release webhooks and the Android keystore secrets. GH_TOKEN, TELEGRAM_BOT_TOKEN and the three DISCUSSION_* secrets, which no workflow used any more, were deleted on 2026-10-16.
  4. Signing key: the old key 03F965C2E2DB5C6B was exposed; sign with the replacement key.
  5. Qodana (JetBrains, https://qodana.cloud): one Qodana Cloud project per linter, because a project takes one linter — itch-typescript (qodana-js), itch-python (qodana-python) and itch-rust (qodana-rust, Early Access). Each project token is a repository secret: QODANA_TOKEN_TS_JS, QODANA_TOKEN_PYTHON, QODANA_TOKEN_RUST (Settings → Secrets and variables → Actions). qodana.yml runs the three jobs; the settings are webapp/qodana.yaml, qodana.yaml and webapp/src-tauri/qodana.yaml (each with a quality gate of 0 problems: a finding fails its job — fix it, or switch the inspection off there with the reason). The three linters need an Ultimate licence: when the trial or the subscription ends, the scans fail with a licence error — renew it, or delete qodana.yml and the three files (nothing else depends on them). A new token: Qodana Cloud → the project → Settings, then update the secret.

3. Releasing a version

  1. Bump the version everywhere it lives: webapp/src/lib/about.ts (APP.version), webapp/package.json + package-lock.json (npm version X.Y.Z --no-git-tag-version), webapp/src-tauri/tauri.conf.json, webapp/src-tauri/Cargo.toml + Cargo.lock, and the version badges of README.md / README.vi.md. Write the notes in changelog/<major>.x/ver<major>.<minor>.x.md and add the row to CHANGELOG.md.
  2. Run the accessibility smoke test (ACCESSIBILITY.md §4) on the preview or the merged site: the main tasks with the keyboard only, a screen reader spot check (Narrator, NVDA or VoiceOver), 200 % zoom and a narrow window, reduced motion, and a Windows contrast theme. If a change made a known limitation go away (or added one), update §6 of the statement and its Vietnamese mirror.
  3. Open a PR; when CI is green, merge it with squash or a merge commit — never fast-forward. A push that touches the changelog posts to Discussions → General unless the head commit says [skip-discuss] (the release announcement usually covers it).
  4. Apply any new D1 migration and set any new secret before the merge deploys code that needs them.
  5. Tag from main: git fetch origin && git tag -s vX.Y.Z origin/main -m "…", then git push origin vX.Y.Z. release_desktop.yml opens one draft release and the desktop and Android jobs upload into it (12 assets for 4.x). Never create the release by hand first.
  6. When every asset is there, edit the title and notes (from the changelog), then gh release edit vX.Y.Z --draft=false — that fires the Discord and Discussions announcements.

4. Routine

  • Maintenance mode (since 4.6.1): the project is stable. The work left is the weekly dependency sweep, security fixes, runtime updates and the items below; feature requests and code pull requests are closed with a pointer to README → Project status. Issues stay open for the data only (add, fix, remove a game), accessibility barriers and things that broke.
  • Mass-change alert (Discord, "mass_change=true"): itch.io probably changed its page markup. Look at a few flagged games on itch.io. If the parser is wrong, fix scripts/scraper.py; if the removals are real, dispatch Refresh catalog with allow_mass_removal.
  • Removal / takedown request: /admin → Catalog → the game → Remove… (the reason is public; the presets get a translated label on the Removed page). Several games: tick up to 100 search results, or paste their URLs under Remove by list — one commit either way. A wrongly removed game: Catalog → Removed → Restore.
  • Review queue: /admin → Review queue. "Approve with override" is needed for games that were removed before. Selections of any size go out in batches of 200 (one commit each).
  • Adding a list of games: /admin → Add games. Paste any text (or load a file, or read an itch.io collection), press Check to see what Add would do, then Add. Recent batches follows each game through the ingest (added / waiting / dropped with the reason); the ingest runs right after each commit and handles about 650 games per 45-minute run, the rest waits in the queue.
  • Automatic rules: /admin → Settings. Auto-approve (on by default for RSS and Suggest, after 24 h) queues pending items without a flag that nobody rejected; it only considers items found after the moment it was switched on ("Include older pending items" lifts that), at most 100 per cron run. Followed collections are read in turn (≤ 30 pages per run, one a second); their new free games are queued directly, removed or flagged ones go to the review queue. Settings also shows what the last cron run did (Workers logs are sampled), and a collection whose page could not be read (status 422: itch.io changed its markup) adds nothing until worker/collections.ts is fixed.
  • Reviewing new games: /admin → New games (added in the last 7 / 30 days): set 18+ and the virus check for up to 100 games at once. Activity lists every admin action and automatic run with its commit.
  • Build failed (Discord from deploy-status.yml): open the build log linked in the message; the site keeps serving the previous deployment meanwhile.
  • New D1 migration: add webapp/worker/migrations/000N_*.sql (never edit applied ones), apply it to staging and production by hand, then deploy.
  • Catalog delivery (after a release that touches it, or when the site shows no games):
    B=https://freeitchgames.win/data
    curl -sI $B/pack/manifest-v1.json                    # 200, application/json, max-age=60, ACAO *
    H=$(curl -s $B/pack/manifest-v1.json | node -p "JSON.parse(require('fs').readFileSync(0)).chunks[0].main")
    curl -sI -H 'Accept-Encoding: br, gzip' $B/pack/$H   # 200, octet-stream, no content-encoding, immutable
    curl -sI $B/pack/0000000000000000.bin                # 404, no-store (answered by the Worker)
    curl -sI $B/index.json                               # 410, vary: Origin (browsers and scripts)
    curl -sI -H 'Origin: tauri://localhost' $B/index.json   # 200 (apps 4.0–4.1.1)
    
    npx wrangler tail shows the legacy-data: 410 lines with the requesting Origin: an installed app that sends an unexpected one would show up there.
  • Visitors report "Too many requests" / error 1015: "error 1015" is the WAF rule (section 1, step 15) — raise its threshold in the dashboard, effective within a minute. The site's own "Too many requests" page or {"error":"rate_limited"} is the Worker: raise the limit in wrangler.jsonc and deploy. Both reset by themselves (10 s / 60 s).
  • Verification gate misbehaving (visitors stuck on the check, crawl errors in Search Console): npx wrangler secret delete GATE_SECRET turns it off at once; then look at npx wrangler tail and curl -s https://freeitchgames.win/api/verify.