Licenses - Code stays MIT (LICENSE: 2025-2026 poli0981 (SkullMute)). The catalog data (data_game/LICENSE.md) and the documentation are CC BY 4.0 (full legal code in LICENSES/CC-BY-4.0.txt); creators' descriptions, cover images, names and trademarks are excluded. NOTICE.md: license map and non-affiliation with itch.io. Policies (English + Vietnamese mirrors, dated 2026-09-18, applying from 4.0.0) - Terms of Use now cover the website and the public data, with the content-removal / copyright process, prohibited uses (bulk scraping, /img abuse, /admin and ingest access, Suggest spam), 18+ opt-in and Vietnamese law with the consumer-rights reservation. - Privacy Policy rewritten for Cloudflare as processor: exact browser storage, cookies, Turnstile, D1 retention (Suggest notes 180 days, idempotency 7 days), rights under Law No. 91/2025/QH15 and the GDPR. - EULA covers only the desktop and Android apps (also the installer RTF). - Disclaimer, Security Policy (private reporting, scope, safe harbor) and Code of Conduct (Contributor Covenant 2.1) rewritten; PAT, GPG and Telegram-bot material removed. Docs and templates - README / CONTRIBUTING (EN + VI), webapp/README, TAURI.md, dev_env (EN + VI), THIRD_PARTY, ACKNOWLEDGEMENTS; new docs/operations.md (Cloudflare + GitHub setup, v4 release order, routine tasks). - CLAUDE.md rewritten for the v4 architecture; CHANGELOG [4.0.0]. - Issue templates: config.yml (Suggest page + private security reports), "Add games" points to /suggest, "Remove a game" gained a rights-holder section; PR templates updated; CODEOWNERS. - Dependabot npm version updates resume with the v4 merge (limit 5). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
21 KiB
Repo notes for Claude
Quick context so future Claude sessions don't have to re-derive it.
What this repo is
A curated, auto-updating catalog of free itch.io games, published at https://freeitchgames.win (read-only for everyone; the old github.io URL only redirects). Three layers share one JSON catalog:
- Python pipeline — the data layer (GitHub Actions).
data_game/game_info_NNN.json— chunked records, max 500 per file, 2,681 games as of Sept 2026. Serialization is fixed:json.dumps(..., ensure_ascii=False, indent=4), no trailing newline (json_io.dumps) — the Worker port (webapp/worker/catalog.ts) emits identical bytes (tests/fixtures/golden/).data_game/index.json— chunk manifest (total + per-file counts).data_game/count_history.json— date-keyed{date,total}series ("games over time" chart).scripts/deleted_games.json— removal log (public "Removed" page).scripts/temp_link.json— ingest queue.scripts/— scanners never write the catalog:update_info.py(queued URLs) andrefresh.py(rotating 1/7-per-day check;--fullre-scrape) emit a URL-keyed patch (patch.py);apply_patch.pyapplies it to the latestmainand runsvalidate.py;bash/commit_push.shdoes fetch → reset → apply → commit → push with retries. Shared:scraper.py(session,Pacer+ 429 back-off, parsing),data_store.py(chunks stay in place, rebalance only past thresholds, write-if-changed),json_io.py,canonical.py(canonical itch.io URL; vectors intests/fixtures/url_vectors.json, shared with the Worker). Pipeline bookkeeping lives inscripts/state/(not published). Tests:pytest(network blocked). Lint:ruff check scripts tests webapp/scripts,ruff format,vulture(config in rootpyproject.toml). Local env:python -m venv .venv && .venv/Scripts/pip install -r requirements-dev.txt(Windows: run withPYTHONUTF8=1).
- Cloudflare Worker + static assets —
webapp/worker/+webapp/wrangler.jsonc, deployed by Cloudflare Workers Builds frommain(root dirwebapp, buildnpm run build, deploynpx wrangler deploy; no deploy workflow). The catalog JSON is bundled into the site at build time (vite-plugins/catalog-data.ts→dist/data/*, plusdata/urls.jsonandsitemap.xml), so every data commit rebuilds the site. - Web app + Tauri apps —
webapp/src/(React 19.3 + TypeScript 6.0 + Vite 8/rolldown + Tailwind + shadcn/ui + React Router 8), and the Tauri 2 desktop + Android read-only viewers inwebapp/src-tauri/(same code; they fetchhttps://freeitchgames.win/dataand load covers straight from img.itch.zone). Pluswebapp/admin/— the maintainer-only admin app (second Vite entry, served at/admin/).
Adding games: the public Suggest page (/suggest, Turnstile), the maintainer's browser extension (POST /api/ingest, Access service token; repo poli0981/itch-f2p-extension), and RSS discovery (Worker cron) all feed a D1 review queue; approving in /admin commits the URLs to scripts/temp_link.json as a GitHub App, which triggers update.yml.
How to work in webapp/
cd webapp
npm ci
npm run dev # Vite on http://localhost:5173 (proxies /api and /img to wrangler dev on :8787)
npx wrangler dev # the Worker + built assets (run `npm run build` first); .dev.vars holds local secrets
npm run build # → webapp/dist/ (public app + /admin/ + /data + sitemap); Tauri builds use the same dir
npm test # vitest (worker/, src/lib); worker/test/* D1 tests spawn a local workerd via getPlatformProxy
npm run lint && npm run knip && npx tsc -b
npm run cf:types # after editing wrangler.jsonc (ignores .dev.vars via types.env; CI runs it with --check)
npx wrangler d1 migrations apply freeitchgames --local # local D1 (use --remote by hand for production)
npm run tauri:dev # native window; needs Rust toolchain
npm run tauri:build # native installers
npm run tauri icon -- path/to/source-1024.png # regenerate src-tauri/icons/
npm run tauri -- android init # generate src-tauri/gen/android (gitignored)
npm run tauri -- android build --apk --target aarch64 # arm64-v8a release APK (unsigned; CI post-signs)
Local admin without Access: put DEV_ADMIN_EMAIL=you@example.com in webapp/.dev.vars — honoured only when the host is localhost. Turnstile test keys work locally (TURNSTILE_SITEKEY=1x00000000000000000000AA, TURNSTILE_SECRET=1x0000000000000000000000000000000AA, SITE_ORIGIN=http://localhost:<port> in .dev.vars). Android details live in webapp/TAURI.md → Android APK. Path alias: @/* → webapp/src/*.
Worker (webapp/worker/)
index.tsroutes:/img/*(img.ts),/api/health,/api/suggest(suggest.ts),/api/ingest(ingest.ts),/api/admin/*+/admin*(admin.ts), everything else that misses an asset →spa.ts.scheduled()(cron23 */4 * * *):discover.ts(one itch.io RSS feed per run, 6 h back-off on 429) →QueueStore.reconcile(queued → ingested / failed after 3 days) →purge(idempotency 7 d, Suggest notes 180 d).img.ts:/img/<160|640>/<path on img.itch.zone>— allow-listed againstdata/urls.json, edge cache → R2THUMBS(w{w}/<sha256>.webp) →cf.imagetransform (stored once) → fallback streams the original with a 1-day cache (quota error 9422). Upstream errors → controlled 502.spa.ts:not_found_handlingis"none", so misses reach the Worker:/assets/*,/data/*and file-like paths get a real 404 (no-store); app routes get the shell with 200; unknown paths and game slugs not in the catalog get the shell with 404. Keep itsROUTESin sync withsrc/App.tsxandsrc/hooks/useSeoHead.ts.access.ts: verifiesCf-Access-Jwt-Assertionwith jose (issuer + AUD). Maintainer = email in secretADMIN_EMAILS+ same-origin writes; ingest = service-tokencommon_name.admin.ts: API (queue list / approve / reject / add / edit the 3 fields / delete with reason + R2 cleanup / restore) and the/adminshell with its own CSP. The maintainer's email goes only to the D1 audit log, never into a commit (the repo is public).repo.ts+github.ts+catalog.ts: writes as a GitHub App (installation token, cached per isolate) with GraphQLcreateCommitOnBranch+expectedHeadOid(verified commits;commitWithRetryrebuilds from the new head on a race). Reads use the raw media type (no base64/UTF-8 mojibake).catalog.tsportsdata_store.pybyte-for-byte (golden fixtures).queue.ts: URL classification (catalog → blocklist → pending → previously deleted) + the D1 store (worker/migrations/, additive only — never edit an applied migration).data.ts: catalog facts read from the site's own/dataviaenv.ASSETS, cached per isolate (loaded values only — see gotchas).- Configuration: non-secret settings are
varsinwrangler.jsonc(empty until setup); secrets (GH_APP_PRIVATE_KEYPKCS#8,ADMIN_EMAILS,TURNSTILE_SECRET) viawrangler secret put, typed inworker/env.tsas optional. A feature whose settings are missing answers 503; the public site needs none of them.env.staging= Workerfree-games-itchio-list-stagingonstaging.freeitchgames.win(no cron, own R2/D1/rate-limit namespaces).
Web app structure (webapp/src/)
routes/*.tsx— one file per route (/,/games,/games/:slug,/charts,/removed(/deletedredirects),/suggest,/settings,/about,/errors/:code);App.tsxwires them;components/app-router.tsxpicks BrowserRouter on the web, HashRouter in Tauri.lib/legacy.tsrewrites v3#/xlinks to/x(crafted hashes go home) and deletes v3 PAT/GPG storage keys.- Data:
lib/config.ts(DATA_BASE=/dataon the web,https://freeitchgames.win/datain Tauri),lib/data/*,hooks/useGames.ts. react-query cache persisted to IndexedDB (webapp.query-cache, 7 days);CACHE_BUSTER = ${APP.version}:data-v1.gcTimemust stay >= persistmaxAge. components/data-table/*(TanStack Table v8 + Virtual; row ids are URLs),components/charts/(16 charts, 4 lazy tabs, no barrelindex.ts),components/game-thumb.tsx(lib/thumbnail.ts→/img/160|640/...on the web, the original URL in Tauri),components/legal-gate.tsx(links flaggedinGateinlib/about.ts; re-prompts whenLEGAL_VERSIONinstores/prefs.tschanges).hooks/useSeoHead.tssets canonical + robots per route (index.html has no static canonical/og:url on purpose).lib/i18n/—en.tsis the source of truth;vi.tsisRecord<MessageKey, string>and must never be static-imported (lazy chunk).stores/— Zustand:prefs(webapp.prefs: language, density, legal acceptance…),theme(webapp.theme).lib/about.ts— keepTHIRD_PARTYup to date when adding npm deps;LEGAL_LINKS;APP.version.- Code split (
vite.config.tsrolldowncodeSplitting.groups):vendor-react(priority 100 — must claim React first),vendor-query,vendor-ui. There is deliberately no vendor-charts group (it pulled React/clsx in and put Recharts on the first load); Recharts stays in the lazy chart chunks. Initial load ≈ 195 KB gzip. Web buildtargetis Safari/iOS 15.4 (Radix class static blocks otherwise break iOS 15); Tauri uses the default.
Editable vs read-only fields
Each game has 20 fields (plus optional added_at / updated_at, appended by the pipeline — schema changes are additive only, installed v3 apps may still read the raw JSON). Only 3 are maintainer-editable: safe_virus (?|Yes|No|Caution), notes, nsfw (Yes|No) — via /admin. The other 17 are scraper output; no scrape (not even --full) overwrites the 3 editable fields.
GitHub Actions
| Workflow | Trigger | Notes |
|---|---|---|
update.yml |
Push to main touching scripts/temp_link.json + daily 01:23 UTC + workflow_dispatch (url) |
Ingest queue → patch → commit_push.sh. Workflow-level concurrency: ingest with queue: max (the default queue replaces a pending run, which would drop its dispatched url). A dispatched url is processed first and written to temp_link.json (queue_add) until it reaches a final outcome. Failed fetches count as an attempt only ≥ 6 h apart (last_attempt; 3 → given up), and a 429 counts as none. GitHub-App pushes (extension, admin Worker) trigger it; its own GITHUB_TOKEN push does not (no loop). Keep the filename. |
refresh.yml |
Daily 02:47 UTC + workflow_dispatch (budget, url, allow_mass_removal) + workflow_call |
scan (read-only, 55-min step timeout, patch artifact uploaded if: always()) → apply (!cancelled(): a failed / timed-out scan's checkpoint is applied, a cancelled run's is not) → notify. Workflow-level concurrency: refresh-pipeline + queue: max runs one whole scan → apply at a time (also for force-update's call; the group name must differ from the caller's or GitHub cancels it as a deadlock). Removal needs the same 404/410 or paid strike seen again ≥ 20 h after the first sighting (strike_at). Checkpoints never carry removals. Mass-change guard: a run whose removals or new strikes exceed max(10, 10 %) of the games it checked withholds all its removals and marks them — and, on a strike spike, all its strikes — withheld_at; later runs keep withholding those (they take at most half a run's budget) until a healthy check clears them, and alert (mass_change) — that pattern means itch.io markup changed; dispatch with allow_mass_removal only after checking on itch.io. apply_patch.py re-checks the same per-run limit and max(25, 5 %) of the catalog unless ALLOW_MASS_REMOVAL=1. |
force_update.yml |
workflow_dispatch ONLY (url, budget, allow_mass_removal) |
queue: max. Calls refresh.yml with full: true: re-scrapes every scraper field for one URL or the next batch (rotates on full timestamps). Always preserves safe_virus / notes / nsfw. Canonical tool for repairing mojibake. |
notify-failure.yml |
workflow_call |
Discord (DISCORD_CI_WEBHOOK) on failure / cancelled (job-level timeouts end as cancelled) / rate-limited / mass change / failed Cloudflare build. |
python-ci.yml / webapp-ci.yml |
PRs (+ pushes for Python) | ruff + format + vulture + pytest + validate.py / npm run cf:types -- --check + lint + knip + vitest + build + wrangler deploy --dry-run --env="". |
deploy-status.yml |
Hourly + manual | Not a deploy. bash/cf_build_status.sh walks main's first-parent history to the latest finished Workers Builds: free-games-itchio-list check run (Cloudflare posts it only when a build ends; branch commits carry preview builds under the same name) and alerts once per failed build (cache key per check-run id, saved after the Discord post). Scheduled because GitHub never starts check_run workflows for GITHUB_TOKEN pushes (the data commits). |
release_desktop.yml |
Tag v* push, or manual |
create-release makes the single draft (id passed to tauri-action v1 releaseId); release builds use no npm/cargo caches (cache-poisoning). Win + macOS aarch64 + macOS x86_64 (cross-compile) + Linux (ubuntu-22.04 for a low glibc floor). Uploads .dmg / .app.tar.gz / .pkg (macOS), .msi / .exe (Windows), .deb / .AppImage (Linux). macOS .pkg --identifier reads tauri.conf.json. |
release_android.yml |
Tag v* push, or manual |
Tauri Android build → arm64-v8a APK, post-signed with zipalign+apksigner (gen/ stays gitignored). Tag → attaches to the same draft Release; manual dispatch → workflow artifact. Secrets: ANDROID_KEYSTORE_BASE64 / ANDROID_KEYSTORE_PASSWORD / ANDROID_KEY_ALIAS. |
announce-*.yml, notify-release-pipeline.yml |
Release published / pipeline runs | Reusable workflows from poli0981/.github, SHA-pinned — but at that pin they still check out their notifier script from that repo's main. |
All workflows: permissions: {} at the top, per-job grants, actions pinned by commit SHA.
Release / tag process
git fetch origin
git tag -s vX.Y.Z origin/main -m "summary"
git push origin vX.Y.Z
# release_desktop.yml's create-release job opens ONE draft; desktop + Android jobs upload into it.
# When every asset is there, edit the notes and publish (this fires the announcements):
gh release edit vX.Y.Z --draft=false
Draft-first: never gh release create by hand before CI — tauri-action v1 refuses to upload into a published release, and announcements would go out before installers exist. Merge PRs into main with squash or a merge commit, never by fast-forward pushing a branch: its commits' preview-build check runs would then sit on main's own line and confuse deploy-status.yml. Tag from main only. Versions live in webapp/src/lib/about.ts (APP.version), webapp/package.json, webapp/src-tauri/tauri.conf.json and Cargo.toml (+ Cargo.lock) — keep them equal. GPG signing is wired through gpg-agent (commit.gpgsign=true, so git tag -s and git commit both sign; the pinentry must be answered — in unattended sessions commits fail rather than going unsigned). The historical key 03F965C2E2DB5C6B was exposed in the browser extension's storage (Sept 2026) and is being replaced — use the maintainer's current key.
Licenses & policies
Code MIT (LICENSE); catalog data CC BY 4.0 for the maintainer's contribution — excluding creators' descriptions, cover images, names/trademarks (data_game/LICENSE.md, LICENSES/CC-BY-4.0.txt); docs CC BY 4.0; map + non-affiliation in NOTICE.md. Policies (EN in docs/, VI mirrors in docs/i18n/vi/, English controls): ToS (website + data + content removal), EULA (desktop/Android binaries only; also webapp/src-tauri/installer/EULA.rtf), Privacy Policy, Disclaimer, SECURITY.md, CODE_OF_CONDUCT.md. When a policy changes, bump LEGAL_VERSION (webapp/src/stores/prefs.ts) and the dates in about.legal.desc (EN/VI) so the legal gate asks again. Contact addresses legal@ / privacy@ / security@ / takedown@freeitchgames.win are Cloudflare Email Routing forwards.
Known gotchas (cost a release each — don't relearn)
universal-apple-darwin+bundle_dmg.shis fragile in headless CI. Use per-arch (aarch64-apple-darwin,x86_64-apple-darwin).macos-13is retired from free GitHub-hosted runners. Cross-compile x86_64 frommacos-latest.- Tauri icons must exist before build.
npm run tauri icon -- path/to/source.png; the 1024×1024 source is gitignored, generated icons are committed. - Tauri doesn't ship
.pkg. Built withpkgbuildafter tauri-action and uploaded viagh release upload; the step usesif: ${{ !cancelled() && … }}so a DMG flake doesn't drop the PKG while a cancel still stops it. - Vite v8
baseonly accepts'./', an absolute URL, or an empty string. Web uses'/'(BrowserRouter deep links), Tauri'./'. - TS 6 deprecates
baseUrl. Usepathsalone. Stay on TS 6.0.x until typescript-eslint supports newer. - lucide-react v1+ removed brand icons (
Github). UseLibrary,Database, etc. bundle.windows.nsis.licensedoes NOT exist in@tauri-apps/cli2.11 — use top-levelbundle.licenseFile, and it must be RTF (WiX.msirequires it; NSIS auto-detects). File:webapp/src-tauri/installer/EULA.rtf.- The bundle
identifiermust be alphanumeric (com.poli0981.freegamesitchio): Android rejects-, Tauri's validator rejects_; one identifier serves every platform. - Android signing:
zipalignBEFOREapksigner; never edit the generated Gradle; never lose the keystore.versionCode = major*1e6 + minor*1e3 + patchfromtauri.conf.json.minSdkVersion30 (Android 11) is the install gate — change it only intauri.conf.json(bundle.android.minSdkVersion); no runtime OS check by design (reach ≈ 86.9 % mid-2026; seedocs/pc_spec.md). The Android JDK setup must NOT usecache: gradle(gen/ doesn't exist at checkout). webapp/dist/is gitignored andassets.directorymust be./dist. In Sept 2026 a config-less deploy served the rawwebapp/tree (source exposed).- A Worker with static assets gets no
ctx.access— verify the Access JWT yourself (access.ts).workers_devandpreview_urlsmust stayfalse: those hostnames bypass Access. public/_headersapplies only to responses served straight from assets, not to responses the Worker builds (set headers inworker/http.ts). Responses fetched throughenv.ASSETSdo carry them.- With
run_worker_firstas an array, the asset layer's SPA fallback answers every miss before the Worker — a stale hashed chunk gotindex.html+ the year-long immutable cache. Hencenot_found_handling: "none"+spa.ts. - Never share an in-flight promise across requests in a Worker (cache only resolved values): a pending subrequest dies with the request that started it (client disconnect) and every later awaiter hangs.
- itch.io cover URLs are signed per size, so you can't rewrite
/original/to a smaller variant — resizing goes through Image Transformations (cf.image) and R2. - GitHub Actions concurrency keeps at most one pending run per group by default and cancels the older pending one — use
queue: maxwhere a queued run carries unique input. A called workflow must not request its caller's group (deadlock → cancelled). - Pushes made with
GITHUB_TOKENtrigger no workflows (no ingest loop), and GitHub never startscheck_runworkflows for them (hence the scheduleddeploy-status.yml). Workers Builds posts its check run only when a build ends. wrangler typesreads.dev.vars—npm run cf:typespoints it at the emptytypes.envso the committedworker-configuration.d.tsnever depends on local secrets. Don't usesecrets.requireduntil every environment has the secrets (deploys fail without them).- D1 bindings have no
database_id:wrangler deploybinds (or creates) the database by name. Migrations are applied by hand (--remote), never in the build.
Safe-edit rules
- Don't write
data_game/from new code paths directly — go through a patch +apply_patch.py(or the Worker'srepo.ts), so concurrent writers never clobber each other andvalidate.pygates every commit. - Don't commit
webapp/dist/,webapp/src-tauri/target/,webapp/src-tauri/gen/,webapp/src-tauri/icon-source.png,.dev.vars,username.txt,assets/qr/.Cargo.lockis committed (since v4). - Never commit secrets (GitHub App key, admin emails, Turnstile secret, tokens). Data workflows push with the job's
GITHUB_TOKEN; the Worker writes as the GitHub App. - When adding an npm dep, also add it to
webapp/src/lib/about.tsso the About page lists it.
Current state (Sept 2026)
main = v3.9 line plus the pending v4 PRs: #81 (Phase 0 hotfix: real build on Cloudflare, security headers, Android CI) and #82 (Phase 1: rotating refresh / patch pipeline, CI hardening; commits 304b4d3, bbad450, 7b3882b). Branch v4 (stacked on #82, not pushed yet) holds Phase 2 (Worker platform, read-only app), the web review fixes, Phase 4 (D1 queue, admin, ingest, suggest, RSS discovery) and Phase 5 (licenses, policies, docs, version 4.0.0). Already in v4 from Phase 3: 18+ games hidden by default with a Settings opt-in, the age line in the legal gate, /removed, <html lang>. Still to do (Phase 3 visual redesign, awaiting mockup approval): Tailwind v4, top nav instead of the sidebar, a Welcome page at /, the legal gate as an overlay, URL-synced filters. Go-live = one merge of v4 into main after staging checks, then tag v4.0.0.