main
152
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a956c8e182 |
ci(qodana): one workflow, interpreter, Rust build deps, Svelte via Docker
What the first runs showed, and the changes:
- Python: SDKs.json was empty - native mode finds no interpreter by
itself, and without one every `dict[int, str]` and `X | None` was an
"invalid type hint" (11 of the 50 problems). The job now builds .venv
from the hashed requirements and passes QODANA_PYTHON_PATH.
- Rust: "Analysis scope: 0 packages" and a Critical "Cargo project
loading" sanity problem: the image has no C linker and none of
Tauri's libraries, and runs as the runner's user, who can neither
apt-get install nor write the image's CARGO_HOME. Now `-u root`,
CARGO_HOME in the cached Qodana cache dir, and a bootstrap that
installs the build packages.
- JS/TS: installing the Svelte plugin fails in native mode with every
released CLI up to 2026.2.3 ("qodana.build.date system property must
be set"; fixed upstream as QD-16701 on 2026-11-05, unreleased), so
this project runs in the Qodana image, on the node_modules and
generated files the job prepared.
One workflow (qodana.yml) with a job per project, as in
free-games-itchio-list: the token check covers all three, triggers skip
data and Markdown-only commits, no PR comment (annotations, Qodana
Cloud and the report artifact carry the results), --repository-root .
for the subdirectory projects, and each job keeps its own cache keys.
The qodana.yaml files use `exclude` and name their Qodana Cloud project.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
629cb1df75 |
ci: Qodana scans for Python, JS/TS and Rust
Three workflows, one per Qodana Cloud project and token (QODANA_TOKEN_PYTHON, QODANA_TOKEN_TS_JS, QODANA_TOKEN_RUST), each with the qodana.yaml of its project directory: - qodana-python.yml: Qodana for Python over scripts/, native mode on the runner's Python 3.15 with the hashed requirements installed. - qodana-js.yml: Qodana for JS over web/, native mode on Node 26 after `npm ci --ignore-scripts` and `npm run build:admin`, with the Svelte plugin so .svelte files are read. - qodana-rust.yml: Qodana for Rust (EAP, Docker only) over web/src-tauri/, with a stand-in web/dist so generate_context!() expands. Each runs on pushes to main and pull requests that touch its files, and by hand. The scan is skipped when its token is not available (forks, Dependabot), as sonarcloud.yml does; the action is pinned by SHA, the job's token can read contents and write only checks and PR comments, and each workflow has its own cache keys so the three do not collide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
005f923c4f |
build(deps): Bump SonarSource/sonarqube-scan-action
Bumps the actions-all group with 1 update: [SonarSource/sonarqube-scan-action](https://github.com/sonarsource/sonarqube-scan-action). Updates `SonarSource/sonarqube-scan-action` from 8.4.1 to 8.5.0 - [Release notes](https://github.com/sonarsource/sonarqube-scan-action/releases) - [Commits](https://github.com/sonarsource/sonarqube-scan-action/compare/e805d875275f728fac1885b413934be6d7c63f65...59202af334b4ba7b5db01b701b8e79f8b3bef0f4) --- updated-dependencies: - dependency-name: SonarSource/sonarqube-scan-action dependency-version: 8.5.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-all ... Signed-off-by: dependabot[bot] <support@github.com> |
||
|
|
fa8dbaf92b |
docs: put the project in maintenance mode
The project is stable: no new features, code changes only for security advisories and runtime/dependency updates, and issues and pull requests are for the data (add, correct, remove a game). - README: status, build, quality, version, licence and runtime badges, a "Project status" section, and a data-first Contribute section - CONTRIBUTING, SECURITY, Contact, the PR template and CLAUDE.md state the same policy - Issue forms: add_game and edit_game are new; feature_request, feedback and off_topic are removed; blank issues are off (config.yml) - About page: "Report something" links the five remaining forms Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
e7c9d05e7c |
build(deps): Bump the actions-all group with 2 updates
Bumps the actions-all group with 2 updates: [softprops/action-gh-release](https://github.com/softprops/action-gh-release) and [SonarSource/sonarqube-scan-action](https://github.com/sonarsource/sonarqube-scan-action). Updates `softprops/action-gh-release` from 3.0.3 to 3.0.4 - [Release notes](https://github.com/softprops/action-gh-release/releases) - [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md) - [Commits](https://github.com/softprops/action-gh-release/compare/efb35369e0ad2afab669f228072c1b0d510eae64...8a0cd651e5b2365bd640854a3fa5dfc70e4875f3) Updates `SonarSource/sonarqube-scan-action` from 8.4.0 to 8.4.1 - [Release notes](https://github.com/sonarsource/sonarqube-scan-action/releases) - [Commits](https://github.com/sonarsource/sonarqube-scan-action/compare/4d1dc0c91ee400dcf8a6dddf2f705ac36542e304...e805d875275f728fac1885b413934be6d7c63f65) --- updated-dependencies: - dependency-name: softprops/action-gh-release dependency-version: 3.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-all - dependency-name: SonarSource/sonarqube-scan-action dependency-version: 8.4.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-all ... Signed-off-by: dependabot[bot] <support@github.com> |
||
|
|
2b4b988cc7 |
docs: sync the community docs with the repository
A read-through against the code found about forty claims that had drifted. README: - Dynamic game-count and version badges. The count is index.json's total. The version is the newest v* release by semver, since an app release published last would otherwise count as "latest". - SvelteKit 3, every desktop package, and the shard layout without a last shard number. - The media host, which commits deploy, and the nine workflows the table left out. - The accessibility statement and form in the link lists. SECURITY.md: - Fix the broken bullet list. - Describe the real stack, and pushes with the built-in GITHUB_TOKEN. - Add to scope: the media host, /api/ingest/*, /api/updates/desktop and the avatar origin. AUTHORS.md and username.txt: - Donations are in FUNDING.yml. - The Add-games template is gone. - Test devices come from pc_spec. The issue forms lose the hash URLs, the GitHub Pages permalink, and the editing and GPG options. web/README.md, dev_env (EN + VI), the acknowledgements and the contact page are updated to match. The last two are not consent documents, so nobody is asked to accept the terms again. CHANGELOG [Unreleased] lists all three commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b5b8b4eb12 |
docs: an accessibility statement, its issue form and a PR template
ACCESSIBILITY.md sits at the root beside SECURITY.md. GitHub reads it from .github/, the root or docs/ and shows it as the repository's Accessibility tab. It covers: - WCAG 2.2 AA as a target, not a claim. - What works today and what does not, with a way around each limitation where there is one. - The browsers, apps and assistive technology actually used (keyboard, Narrator, NVDA). - How to report a barrier without saying anything about yourself. - Replies on SECURITY.md's terms: first within a week, blockers first, no fixed deadlines. - The rules for contributors and coding agents. .github/ISSUE_TEMPLATE/accessibility.yml files a report under the new `accessibility` label, with no required final checkbox. The pull request template is the short checklist, and completes the community profile. CLAUDE.md says the statement is a claim about the code, and names the two limitations that look solved but are not. .gitattributes keeps the new file out of source archives like the other community files. It also stops leaving out Contact, PRIVACY_POLICY and ToS: the web build reads every legal document and stops when one is missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
984bc6ef12 |
build(deps): Bump SonarSource/sonarqube-scan-action
Bumps the actions-all group with 1 update: [SonarSource/sonarqube-scan-action](https://github.com/sonarsource/sonarqube-scan-action). Updates `SonarSource/sonarqube-scan-action` from 8.3.0 to 8.4.0 - [Release notes](https://github.com/sonarsource/sonarqube-scan-action/releases) - [Commits](https://github.com/sonarsource/sonarqube-scan-action/compare/d209202bc7d53ff1cc128f7f907dac145c9d6ae9...4d1dc0c91ee400dcf8a6dddf2f705ac36542e304) --- updated-dependencies: - dependency-name: SonarSource/sonarqube-scan-action dependency-version: 8.4.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-all ... Signed-off-by: dependabot[bot] <support@github.com> |
||
|
|
4672f871b3 |
ci(discovery): queue waiting runs and sweep a whole year
Every window swept before needs one more pass for the games without English, and backfill windows are dispatched several at a time. The discovery group kept ONE waiting run, so a third cancelled the one already waiting - the lesson data-write learned in October. - queue: max in both discovery workflows; pipeline.test.ts now holds every shared concurrency group to one queue setting. - Backfill Discovery takes max_pages (through env, default 40): a whole recent year is about 30 pages of 100. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0f600a99aa |
build(web): migrate to SvelteKit 3 and adapter-static 4
@sveltejs/kit 2.70.3 -> 3.1.0, @sveltejs/adapter-static 3.0.10 -> 4.0.0. Closes GHSA-27mw-82g8-phjj and GHSA-pfhg-q3qv-9q6r (dismissed in v4.4.1 as not applicable), and drops the cookie override kit 2 needed: npm audit reports nothing. - Config: svelte.config.js is refused by kit 3. Its options are now sveltekitConfig(tauri) in build/sveltekit-config.ts, passed by vite.config.ts as sveltekit(sveltekitConfig(IS_TAURI)), with tests for both CSP flavours, paths.relative, the sitemap entry and the prerender 404 handler. The unused, now-deprecated "@" alias and typescript.config hook are gone; tsconfig.json extends $app/tsconfig and owns `include`. - $lib -> #lib (package.json imports), with .js on module imports. tsconfig repeats the mapping in `paths`: svelte-check --tsgo only maps .svelte imports through `paths`, and without it every #lib component fell back to the *.svelte shim (props unchecked; proven with a planted error). - Param matchers: src/params/legaldoc.ts -> src/params.ts (defineParams). vite build imports that file with Node's own import(), so it imports #lib/legal.ts and legal.ts imports ./schema.ts by their real names. - PWA: @vite-pwa/sveltekit never supported kit 3 and wrote the worker when the SERVER build closed, which kit 3's environment build does first. build/pwa.ts wires vite-plugin-pwa in a buildApp hook after kit's prerender and before its adapter, with the same globs and URL mapping. - goto options renamed (replace, reset: false, refreshAll); the /games filter write is a shallow goto; a legacy /#/ path that matches no route falls back to location.replace (kit 3's goto rejects it). Hook types from @sveltejs/kit/hooks; the client handleError returns nothing to keep kit's message. - runed 0.35 (bits-ui) peers kit ^2: overridden to our kit, as for echarts-wordcloud. Verified against a kit 2 build of the same commit: same 11,037 files bar hashed chunks, same non-immutable precache URLs, navigateFallback "/", identical CSP per page (hashes masked), manifest and sitemap byte-equal, same visible text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d057e3f620 |
fix(deps): patch the open advisories in npm, cargo and pip; Node 24 actions
npm (web/): - undici 7.30.1 through a "miniflare" override (GHSA-xjj6-f33q-929c, GHSA-4qv8-2p9j-2685, GHSA-f72v-6w45-q472). miniflare pins undici exactly and wrangler 4.152.0, the newest, still ships 7.29.1. Local tooling only. - zod 4.6.7 (GHSA-mrhx-c7xc-m3w5, via knip). - Minor/patch: @lucide/svelte 1.61.0, @tanstack/svelte-virtual 3.13.48, bits-ui 2.20.2, vite 8.3.5, wrangler 4.152.0, knip 6.43.0, @tauri-apps/cli 2.12.4. cargo (web/src-tauri/Cargo.lock, targeted updates only): - rustls 0.23.46 (RUSTSEC-2026-0346..0349), rustls-webpki 0.103.16 (RUSTSEC-2026-0382), webpki-root-certs 1.0.11 (RUSTSEC-2026-0383), tokio 1.54.1 (RUSTSEC-2026-0386), anyhow 1.0.104 (RUSTSEC-2026-0190), event-listener 5.4.2 (RUSTSEC-2026-0221), plist 1.10.3 -> quick-xml 0.42.0 (RUSTSEC-2026-0194/0195). Dependabot had none of these. - Left upstream: zip 4 (tauri-plugin-updater 2.14.0 asks for ^4), glib 0.18 (GTK 3), unmaintained proc-macro-error/zbus_names/zvariant. pip: certifi 2026.10.23, coverage 7.17.0 (tests), with every file's hash. CI: actions/checkout v4/v5 -> v6 and actions/setup-python v5 -> v6 everywhere (the Node 20 majors warned on every data job), rust-cache 2.9.3 by SHA. dependabot.yml says why those majors are taken by hand. CLAUDE.md: the undici override and when to drop it; why SvelteKit stays on 2 (GHSA-27mw-82g8-phjj and GHSA-pfhg-q3qv-9q6r need setHeaders or use:enhance, which the app does not have). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
529b8603c3 |
ci(pipeline): queue data-write runs and start each on the latest main
- queue: max on every data-write workflow (GitHub, 2026-05-07): up to 100 runs wait, first in, first out. The default kept one, and a run queueing behind a running job cancelled it. - ref: main on every checkout. A run's commit is fixed when it is created, so a run that waited behind another data job wrote shards from before that job's commit; Auto Update JSON failed four days in ten on the rebase conflict in data/index.json. - timeout-minutes everywhere, and JOB_BUDGET_MINUTES at least 20 below it for the long loops (never for the ingest, which empties its queue). - pipeline.test.ts fails on a data-write workflow missing any of these. - The admin's Pipeline and Automation pages, docs/ADMIN.md and the auto-approve comment no longer say a push evicts a waiting job; the idle gate stays, so a batch's ingest starts at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6e7d1bcf15 |
chore(pipeline): retire Check Dead Links
It sent a HEAD request to every store page every five days, 5.5 hours at 9,847 games, and flagged 404/410. Steam sends a delisted app's page to the store front, which answers 200: scripts/dead_links.log never existed and no game ever became "delisted". The purge's GetItems screen sees delisted games (unlisted) instead. Removes the workflow, bash/dead_links.sh, scripts/check_dead_links.py and DEAD_LINKS_LOG, and the workflow from web/shared/pipeline.ts and the failure notifier's list. check_store_page() stays: it is the health check's last resort. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1da4d5940e |
feat(pipeline): keep the store, peaks and re-fetch runs bounded
- Refresh Store Data: the page and appdetails cycles grow past 2,000 pages / 500 appdetails a day (7 -> 14, 30 -> 60) instead of the run, the slice comes in a daily order, and the store step has a budget. A game appdetails no longer calls free is listed in a ::warning::, the case the purge's two-signal rule cannot see; nothing is written. - Refresh All-Time Peaks: 30 -> 60 days past 400 pages a day, never shorter, so SteamCharts is still asked at most once a month a game; the budget stops it with a warning like a refusal does. - Force Re-fetch: --part K/N, oldest last_updated first, each game cleared on a copy that replaces it only once appdetails has answered; no final loop stamping every game. Parts queue one after another. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
74e1845fc4 |
feat(purge): remove a game only when two of Steam's answers agree
The purge asked appdetails about every game, 5.3 hours at 9,847, and one answer was enough to remove one - appdetails has answered success:false for live games in bursts (2026-09-27). GetItems now screens the whole catalogue in a few minutes (unknown, hidden, unlisted, coming soon, not is_free; never type), and check_game_health() must confirm each flag; "unavailable" is asked again five minutes later. A failed batch or a network error never reads as gone, blocklisted games are not asked about, and more than 500 flags or 100 removals stops the run before anything is written. Removals are logged as before. A notice names earlier removals Steam has since undone. The dry run on the real catalogue: 8 flagged, 3 removed (the three unlisted games), 5 kept because appdetails calls them free. Daily now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
49f0c56ed7 |
feat(pipeline): read reviews for every game from Steam's store API
Update Reviews asked appreviews about one game at a time, two seconds a game. At 9,847 games it ran into GitHub's six-hour limit on 2026-10-23 and was stopped with nothing saved; five of its ten runs before that were cancelled while waiting. - SteamClient.fetch_items(): GetItems for a whole catalogue, a failed batch costing only its own ids (the anti-cheat sweep now uses it). - update_reviews.py reads review_summary() for every game, ~2 minutes, daily. Steam floors its percentage; reviews_from_summary() floors appreviews in integers too, so an ingest and the refresh agree. A count of 0 is never written on GetItems' word (it said 0 for a game with thousands): a stored percentage or "N/A" is asked on appreviews, at most 300 a run. last_updated moves on a change, and on a 14-20 day heartbeat that keeps the site's 30-day "stale" badge quiet without rewriting every record every run. - core/budget.py: JOB_BUDGET_MINUTES. Past it process_batch() starts no further game, so a job saves what it did instead of being killed; a ::warning:: when a loop used three quarters of its budget. - rotation.cycle_for() / daily_order(): cycles that grow with the catalogue as multiples of their base, and an order that changes daily. - The player-count loops skip the batch pauses meant for store pages (Top Online ~27 min instead of 48) and take a deadline, like update_all_full(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
df91b42c4f |
ci(sonar): import test coverage instead of excluding it
The scan job now runs both suites with coverage before scanning: pytest --cov (pyproject.toml: scripts/ without its tests, relative paths, coverage.xml) and `npm run test:coverage` (vite.config.ts: v8, lcov with projectRoot "..", so paths start web/). sonar-project.properties names both reports and drops sonar.coverage.exclusions. requirements-test.txt gains pytest-cov 7.1.0 and coverage 7.16.2 with every file's hash (uv pip compile --universal --generate-hashes --python-version 3.15; the existing pins are unchanged). web gains @vitest/coverage-v8 5.0.5, matching vitest; none of its packages has an install script. Coverage stays off for a plain `npm test`. The token check is its own job, so the scan steps no longer each repeat the condition; the workflow grants nothing at the top and the scan job contents: read. `npm run build:admin` replaces the bare `svelte-kit sync`: it syncs too, and writes the admin bundle the Worker's tests load. Locally: 576 vitest tests at 58.35% of lines, 299 pytest tests at 57.7%. Against main's new-code lines (SonarCloud's isNew), the reports cover 92 of 114 coverable lines, 80.7%: the 80% gate is borderline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
98430b7bb6 |
ci(sonar): give the TypeScript analyser the project's tsconfigs
The first CI scan logged "Failed to parse tsconfig web/tsconfig.json": it extends .svelte-kit/tsconfig.json, which only `svelte-kit sync` writes, so web/src was analysed without its compiler options or types. The job now installs web/ (npm ci --ignore-scripts) and syncs before scanning; both outputs are gitignored, so neither is analysed itself. sonar.typescript.tsconfigPaths names all six tsconfigs: auto-detection found only the two named tsconfig.json, leaving the Worker, shared/ and the tests on fallback options. The toolchain plan no longer claims sonar-python knows 3.15: the scan warns and analyses as 3.14. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b465540faf |
ci: analyse with SonarCloud in CI instead of automatic analysis
A sonarcloud.yml workflow runs SonarSource/sonarqube-scan-action (pinned to the v8.3.0 commit) on pull requests and main, with full history for blame. Commits that touch only what the data jobs write are skipped, and a run without SONAR_TOKEN (forks, Dependabot) passes without scanning. .sonarcloud.properties becomes sonar-project.properties, now naming the organization and project key; its scope settings are unchanged. The toolchain test, Web CI's paths, CLAUDE.md and docs follow the rename. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ca7f8e3e0c |
test(web): hold the Node, Python and TypeScript pins together
web/shared/toolchain.test.ts fails when a pin drifts: - web/.node-version is one major, which every setup-node step, engines.node and the @types/node major repeat; - every setup-python step installs sonar.python.version, which the --python-version in requirements.txt's header repeats; codeql.yml is the one named exception (3.14); - each workflow has as many version keys as setup steps, and every version is a quoted literal (unquoted, YAML reads 3.10 as 3.1); - `typescript` is 6, @typescript/native is npm:typescript@~7, and no script calls a bare tsc. Web CI also runs on .github/workflows/** and .sonarcloud.properties, so a pull request that moves only a pin meets the test - and pipeline.test.ts, which reads the workflows for the data-write group, no longer misses a workflow-only change either. Proven by breaking one pin at a time (one workflow's Python, .node-version alone, a setup-node switched to node-version-file): each failed the test with the file named. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f3585fe8e6 |
build(web): Node 26, and TypeScript 7 beside TypeScript 6
Node: - web/.node-version (26) is the pin. The five setup-node steps say '26' as a literal, not node-version-file: release-android.yml checks out the tag it rebuilds, and older tags have no such file. - engines.node >=26 (advisory) and @types/node ^26.6.5; TypeScript 6 still type-checks clean with the new types. TypeScript: - TypeScript 7.0 has no JavaScript API, and svelte-check (which refuses `typescript` 7) and `svelte-kit sync` (route $types) need one, so `typescript` stays ^6.0.3 and 7 is the alias "@typescript/native": "npm:typescript@~7.0.2", the layout svelte-check documents. ~7.0 because 7.1 breaks svelte-check's experimental mode. - `check` runs svelte-check --tsgo, on TypeScript 7. `check:admin` stays on 6: --tsgo maps a tsconfig outside the workspace root onto the app's components (seen: false errors in src/routes instead of the admin's). - The four `tsc -p` steps and typecheck:worker call node node_modules/@typescript/native/bin/tsc: both packages declare a `tsc` bin, so node_modules/.bin/tsc could be either. - "DOM.Iterable" leaves the admin tsconfigs (an empty lib since 6.0). - Dependabot ignores the alias (it mishandles npm: aliases) and keeps ignoring `typescript` majors, now with the reason. Checked: type errors planted in a route's script, as a wrong prop on src/lib/ui/Badge.svelte, in a +page.ts and in one file per tsc config were each reported by TypeScript 7 (the admin's by 6), every step exiting 1; --listFilesOnly gives the same programs on 6 and 7; the four tsc checks take 0.5 s instead of 6.0 s. From a clean `npm ci --ignore-scripts` on Node 26.12.1: typecheck, the tests, build, verify-dist (web and tauri), cf:check and knip (report unchanged) pass, with still exactly three packages that have install scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
09991874d7 |
ci(python): run the pipeline, its tests and every data job on Python 3.15
- Sixteen setup-python steps move from '3.12' to '3.15'. codeql.yml alone stays on '3.14': CodeQL's supported-languages page lists Python up to 3.14 and its extractor runs on that interpreter; the pipeline uses no 3.15-only syntax. - The updater-json job of release-desktop.yml sets up 3.15 instead of running the runner image's python3. - Four text-mode writes name encoding="utf-8": the dead-link log, both leaderboards and clear_temp(). 3.15 makes UTF-8 the default (PEP 686); on Windows an older Python writes cp1252, which cannot encode the leaderboards' emoji, so those scripts crashed there. CI already wrote UTF-8, so no committed output changes. - requirements.txt's compile line and sonar.python.version say 3.15. `uv pip compile --universal` gives the same lock for 3.12 and 3.15, and every pin installs on 3.15 with --require-hashes --only-binary :all: (charset-normalizer's cp315 wheels were already hashed). Checked on Python 3.15.1: the 299 tests pass with EncodingWarning and DeprecationWarning as errors; edit_game.py --check and delete_game.py --check pass; a full save_main(load_main()) on a copy of the data leaves all 36 files byte-identical to HEAD, as on 3.14, so no shard and no index.json changes when the data jobs move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3145ed2f1a |
fix: SonarCloud's findings on the updater-json job
- scripts/updater_json.py takes no file path from the command line any more, only the tag; it reads latest.json, assets.json and the .sig files from the working directory and rewrites latest.json there (pythonsecurity:S8707, path traversal from CLI arguments). The job runs it inside its work directory. New test for main(). - release-desktop.yml asks for nothing at workflow level and gives each of its two jobs the `contents: write` it needs (githubactions:S8233: with a second job, a workflow-level write reached both by default). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a04f24abb6 |
ci: point every desktop release's latest.json at its download URLs
tauri-action v1 (Dependabot moved release-desktop.yml from v0 after 2.2.0) writes each artefact's API URL (api.github.com/repos/<repo>/releases/ assets/<id>) into latest.json, and has no setting to change that. The desktop updater feed accepts only this release's download URLs and answers 204 to anything else, so desktop-v2.2.1 was offered to no installed app until its latest.json was fixed by hand. A new `updater-json` job runs after every platform's build: it downloads latest.json and the .sig files, rewrites each URL to github.com/<repo>/releases/download/<tag>/<file> (scripts/updater_json.py, which refuses a version that is not the tag's, an asset of another release, any other URL shape, or a signature that is not its file's .sig), and uploads it back. Run on 2.2.1's original file it produces exactly the hand-fixed one, and a second run changes nothing. Also renames the v0 input includeUpdaterJson to v1's uploadUpdaterJson (same default). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
988e4f4ce6 |
ci: pin third-party actions, install without lifecycle scripts, hash Python deps
SonarCloud's GitHub Actions and shell findings, all supply-chain hardening for jobs that hold a write token (and, in the release workflows, the signing keys): - The six third-party actions in release-desktop.yml and release-android.yml are pinned to the commit their major tag points at today, with the version in a comment for Dependabot. dtolnay/rust-toolchain is pinned to a master commit and given `toolchain: stable`: its `stable` branch is regenerated on top of master, so a commit of it can vanish. - Every `npm ci` is `npm ci --ignore-scripts`. Only fsevents, workerd and wrangler's esbuild have install scripts; the last two find their binary in their optional platform package without them, and fsevents is only file watching. `npx` is gone from the workflows (`npm run cf:check`, `npm test -- <file>`). - requirements.txt pins every runtime package, transitive ones included (certifi, charset-normalizer, idna), with the sha256 of every file on PyPI; every install is `--require-hashes --only-binary :all:`. The new requirements-test.txt holds pytest the same way. python-tests.yml runs the data jobs' exact install line first, so a Dependabot bump is proven before a data job meets it. - `[[ ]]` instead of `[ ]` in the bash wrappers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3d0e22cd76 |
fix(pipeline): read anti-cheat from Steam's own disclosure; retire the extension's paths
None of the 4,704 games added since September had an anti-cheat. The only
code that read the anti-cheat a developer declares on Steam was the browser
extension (archived; last push 2026-06-25); the pipeline matched appdetails'
categories, which only ever say "Valve Anti-Cheat enabled".
- core/anticheat.py (new, pure): the store page's div.anticheat_section
(its HEADING decides kernel, never the class; a Boot Protection row is a
flag) and GetItems' extra_details.anticheat; labels from
ANTI_CHEAT_PATTERNS, else Steam's name cut at 40 characters; same_family()
("EAC" = "Easy Anti-Cheat", "ACE" = "Anti-Cheat Expert (ACE)"); and
apply_anti_cheat(), the one rule.
- fetch_full() fills, never replaces, from the page it already fetches (or
GetItems for a gated page), else the VAC category, now found in any
position (with a stored "" only the first category was checked).
- refresh_anticheat.py (new): GetItems for every game (~160 requests), a
store page only for a disclosure without a name. Steam wins; an override
wins over Steam and is not fought (OverrideGuard); no disclosure never
clears; a failed batch is unknown. Runs at the end of the daily Refresh
Store Data job with its own commit; only=anticheat runs it alone. That job
now checks out main.
- The queue takes links only: merge_extension_data(), EXTENSION_FIELDS,
ARRAY_FIELDS and the schema.ts mirrors go. translate_notes() becomes
clean_notes(): the extension's leading "Has paid DLC" and "[hover-add]"
are dropped (517 notes, once, never news).
- refetch_all.py keeps "-" instead of resetting it to ""; a retired override
whose was equals its value no longer bumps last_updated on every save.
- "denuvo" joins Denuvo AC on both sides; anti-cheat.test.ts holds the TS
patterns and enum to constants.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
2ff7b0c260 |
fix(ci): mark-dead-games.yml did not parse
The commit step's plain `run:` scalar contained "chore(data): mark dead games", and a colon followed by a space inside a plain scalar is a mapping indicator: "mapping values are not allowed here" (line 61). GitHub answered every push of this branch with a failed run named after the file, and on main the scheduled job would have stopped running. A block scalar carries the command as text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
997cbd9208 |
docs: removals, auto-approve and the new admin pages
CLAUDE.md: - data/blocklist.jsonl as a data invariant: applied first in save_main(), logged once, the 500 guard, never named data_*, whole-line writers, the shared golden fixture, Restore as the only way back. - The pipeline rules for ingest (blocked links skipped, non-games refused, ingest-new.yml as the removal job on ref main), auto-approve's gates, and Remove/Restore's D1 semantics and WRITABLE paths. - "Publication stays a human decision" now says who decides: a reviewer, or the capped auto-approve rule a reviewer turned on. - Admin routes: the Overview at /admin, the queue at /admin/queue, and no /admin/api route answering 401/403 for anything but Access. - save_main() has 14 callers, not 11 (here and in core/overrides.py). docs/ADMIN.md: the App's Contents permission covers temp_info, overrides and the blocklist (never a shard); why Actions read is not needed; the discovery token's reach with auto-approve on; sections 11-13 for auto-approve, removing/restoring and the Overview/Pipeline/Quality pages; the mock's mock_pipeline switch. README.md: the admin's description, the new scripts and the ingest workflow's triggers. Also: the delete_game.py help example kept its line break, and the mock's removed rows carry the reason's label as the Worker writes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
986077d3f1 |
feat(pipeline): data/blocklist.jsonl - save_main drops and logs removed games
A game the maintainer removes is now a standing instruction in Git, like an
override, instead of a one-time deletion that the browser extension or the
next ingest could undo.
- core/blocklist.py reads data/blocklist.jsonl (one line per game, written by
/admin and by delete_game.py). save_main() applies it FIRST on every write,
for all 14 callers, without touching the caller's list, and whatever
apply_human_overrides says.
- A dropped game is logged once in scripts/removed_games.jsonl as
status_code "admin_removed", built from the blocklist line alone with
removed_at = blocked_at, so writing it again changes no byte. A later purge
line for the same game keeps winning, as dedup_removed() decides.
- More than 500 drops in one save raises BlocklistError before anything is
written: a damaged file fails the job instead of emptying the catalogue.
- remove_games.py applies the blocklist now and heals a missing log line.
bash/ingest.sh runs it before ingest_new.py, and ingest-new.yml also
triggers on blocklist pushes and checks out the tip of main. One workflow
on purpose: data-write keeps one pending run, so a second workflow and the
ingest would keep evicting each other.
- ingest_new.py skips blocked links before any Steam request, and a link
that is not text ({"link": 730}) no longer crashes the run (and every
three-hourly retry after it).
- Discovery counts blocked appids as known.
- delete_game.py is a non-interactive CLI writing the same bytes as /admin
(golden fixture), with --list and --check; check-overrides.yml runs --check.
- mark-dead-games.yml commits through bash/commit_push.sh, like every other
data job: it rebases and retries, and commits the log with the shards.
- dependabot.yml ignores Svelte-era framework majors and @types/node majors
instead of the React-era names.
With an empty blocklist every path is a no-op.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b01341c3a2 | Merge branch 'main' into dependabot/github_actions/actions-all-ec267f2cc7 | ||
|
|
f997d79054 |
feat(data): all_time_peak, from SteamCharts and our own samples
The dataset only knew how many people play a game now. The only peak it had, peak_today, is not today's: nothing resets it, so it is the highest of our sparse samples (online games 3x a week, the rest twice a month) since the last refetch - The First Descendant reads 8,052 against a real all-time peak of 264,860. all_time_peak is the most players a game has had at once, and it only ever goes up (core/peaks.py): - refresh_peaks.py reads SteamCharts' recorded all-time peak, the only public source older than our sampling (SteamDB forbids scraping; Valve's APIs only know today). It takes a crc32 thirtieth of the catalogue a day, so each game is asked once a month, ~186 pages 2-3 s apart. SteamCharts has no API, no terms and no robots.txt, so it is read politely (core/steamcharts.py): a User-Agent that names the project, 500/404 read as "not tracked", and a 403, 429 or Cloudflare challenge stops the run with a warning, keeping everything raised so far. Measured: 9 of 10 games with a few dozen players or more are tracked; most of the 0-3 player long tail is not. - Every run also raises the field from our own samples (peak_today, current_players) without a request, and apply_players() raises it on every sample - never from a 0. That seeds 2,329 games on the first run. - Nothing else is written, last_updated included (player counts never moved it), and refetch_all.py does not clear the field: it cannot be fetched again. "Refresh All-Time Peaks" runs daily at 19:30 UTC, between Refresh Store Data and the snapshot, in the data-write group, committing through commit_push.sh; a manual run with "backfill" asks about every game (~4 h). The crc32 rotation moves to core/rotation.py, shared with refresh_store_data.py. test_refresh_peaks.py holds the rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2b0e9927ba |
chore(deps): Bump the actions-all group across 1 directory with 7 updates
Bumps the actions-all group with 7 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/github-script](https://github.com/actions/github-script) | `8` | `9` | | [actions/setup-node](https://github.com/actions/setup-node) | `4` | `7` | | [actions/upload-pages-artifact](https://github.com/actions/upload-pages-artifact) | `4` | `5` | | [actions/setup-java](https://github.com/actions/setup-java) | `4` | `6` | | [android-actions/setup-android](https://github.com/android-actions/setup-android) | `3` | `4` | | [softprops/action-gh-release](https://github.com/softprops/action-gh-release) | `2` | `3` | | [tauri-apps/tauri-action](https://github.com/tauri-apps/tauri-action) | `0` | `1` | Updates `actions/github-script` from 8 to 9 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v8...v9) Updates `actions/setup-node` from 4 to 7 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v4...v7) Updates `actions/upload-pages-artifact` from 4 to 5 - [Release notes](https://github.com/actions/upload-pages-artifact/releases) - [Commits](https://github.com/actions/upload-pages-artifact/compare/v4...v5) Updates `actions/setup-java` from 4 to 6 - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](https://github.com/actions/setup-java/compare/v4...v6) Updates `android-actions/setup-android` from 3 to 4 - [Release notes](https://github.com/android-actions/setup-android/releases) - [Commits](https://github.com/android-actions/setup-android/compare/v3...v4) Updates `softprops/action-gh-release` from 2 to 3 - [Release notes](https://github.com/softprops/action-gh-release/releases) - [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md) - [Commits](https://github.com/softprops/action-gh-release/compare/v2...v3) Updates `tauri-apps/tauri-action` from 0 to 1 - [Release notes](https://github.com/tauri-apps/tauri-action/releases) - [Changelog](https://github.com/tauri-apps/tauri-action/blob/dev/CHANGELOG.md) - [Commits](https://github.com/tauri-apps/tauri-action/compare/v0...v1) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions-all - dependency-name: actions/setup-java dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions-all - dependency-name: actions/setup-node dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions-all - dependency-name: actions/upload-pages-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions-all - dependency-name: android-actions/setup-android dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions-all - dependency-name: softprops/action-gh-release dependency-version: '3' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions-all - dependency-name: tauri-apps/tauri-action dependency-version: '1' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions-all ... Signed-off-by: dependabot[bot] <support@github.com> |
||
|
|
de9d380994 |
feat(pipeline): refresh tags, languages and store details on a rotation
update_data.py only enriches records that are incomplete, and fetcher.py's apply_details()/apply_scraped() only fill EMPTY fields. Once a game had its tags, languages, platforms or header image, nothing looked at them again short of the manual refetch_all.py. A dry run on three games found all three with tags that had drifted since they were first fetched. scripts/refresh_store_data.py re-reads them a slice at a time, daily: - store page (tags, languages, language_details, has_paid_dlc): every game once every 7 days; - appdetails (name, header_image, description, developer, publisher, release_date, platforms, metacritic, drm_notes): every game once every 30 days. Measured on the live catalogue: 600-675 store pages and 121-166 appdetails calls a day, about 40 minutes at the client's throttle, instead of ~8,800 requests in one sitting. A game is due when crc32(appid) % cycle == day % cycle. Not appid % cycle: Steam appids are nearly all multiples of 10, which would leave 27 of the 30 monthly slices empty. These fields were fill-if-empty until now, so the refresh keeps the same care: it replaces a value only with a non-empty one (a failed request or an age-gated page changes nothing), never writes MANUAL_FIELDS or notes, reads has_paid_dlc only from a real store page, compares description as stored (truncated to DESCRIPTION_MAX), bumps last_updated only on a real change, and writes through save_main() so overrides and index.json behave as for every other job. scripts/tests/test_refresh_store_data.py pins each rule and the even spread of the slices. "Refresh Store Data" runs at 16:00 UTC: scheduled runs here start 2-5 hours late, the morning data jobs run 2-4 hours, and the data-write group keeps only one pending run, so the slot is the one nothing long overlaps. It is in data-write, commits through bash/commit_push.sh, and is on the CI-failure notification list. README, CLAUDE.md and the changelog describe it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8702060b51 |
ci(codeql): scan when .mjs or .cjs files change
CodeQL analyses .mjs files (alerts #16 and #17 are in web/scripts/verify-dist.mjs), but the push and pull_request path filters listed only .js/.jsx, so the commit that fixed those alerts never started a scan and they stayed open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
0ad53b5136 |
ci(android): build the released tag and install platform-tools only
android-v2.0.0 stopped at "Set up Android SDK": setup-android's default package list still names the legacy `tools` package, which the SDK repository no longer carries. Install platform-tools only; Gradle fetches the build-tools and platform it needs. A manual run now checks out the tag it releases instead of main, and the release notes name that commit (on a manual run github.sha is main's HEAD). The tag input has no default any more, because a run replaces the named release's APK. The notes also stop claiming device testing for a build that has not been tested yet; docs/android-support.md keeps the device list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
75c5fcb8ab |
feat(desktop): a working auto-updater, and updates that no longer reset the app
The updater plugin, its signing key and its capability shipped with 2.0.0, but nothing called check(), and its endpoint could not have worked anyway: releases/latest/download/latest.json follows GitHub's single "latest" release, and this repository publishes dataset, Android and desktop releases on one page. - The feed is now /api/updates/desktop. The Worker reads releases.atom (no anonymous-API rate limit, never lists drafts), takes the highest desktop-vX.Y.Z, and serves its latest.json only if the version matches the tag and every artefact URL is this repository's own release download. Anything unusable is a 204, the updater's "no update". Public, edge-cached, and - like /api/activity - never touches the GitHub App token. - The app checks five seconds after launch, once per session and only after consent, and offers the update in a dialog: plain-text release notes, download progress, then relaunch through tauri-plugin-process. The npm plugins match their Rust crates' minor versions, which the Tauri CLI requires. That exposed a second bug. purge_stale_webview_data cleared the webview's storage whenever the stamped version differed from the running one - so every update would have wiped consent, theme, language and the cached catalogue. Only builds before 2.0.0 ever registered a service worker, so needs_purge() now clears only for a missing or pre-2.0 stamp, with unit tests. Release plumbing: both release workflows use a plain `npm ci` (web-ci.yml already forbade the `|| npm install` fallback that once hid a broken lockfile), their notes stop describing a fixed 1400x900 window, GitHub Pages and raw.githubusercontent, and web-ci builds and verifies the Tauri configuration on every change. The privacy policy states what each app's update check contacts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
146f8ad3e8 |
feat(data): per-shard sha256 in index.json, and last_updated only when data changed
index.json was the client's only freshness signal and it was stamped on every save. Two failures followed from that: - A run that changed nothing still produced a new last_updated, so every visitor re-downloaded all ~6 MB of shards (the 2026-09-16 "Dead link check" commit touched only index.json). - Nothing tied an index to the shard bytes it describes. index.json is edge- cached for 30 s and the shards for 300 s, on top of raw.githubusercontent's own CDN, so a client could receive the NEW index with an OLD shard and store that pair in IndexedDB under the new stamp, where it stayed until the next change. _save_index() now records the sha256 of each shard's exact bytes and keeps the previous last_updated when files, counts and hashes are unchanged - leaving index.json byte-identical, so such a run makes no commit. save_jsonl writes LF explicitly: on Windows, text mode wrote CRLF, which is not the blob Git stores or raw.githubusercontent serves, and the hash would have described bytes no client receives. An index without hashes bumps exactly once. The pipeline had no tests. scripts/tests/test_index_hash.py pins all of this, python-tests.yml runs it, and notify-ci-failure now also watches Web CI, Python Tests, Daily Snapshot and Top Offline, which it silently did not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
c8daa927ea |
fix(web): palette crash on open, and prerendered pages that lied
Ctrl-K threw the moment it opened. /charts/anti-cheat/list was listed in both
CHART_PAGES and EXTRA, the palette renders routes in a keyed each, and a
duplicate key throws in a Svelte 5 production build. paletteRoutes() now
deduplicates and nav.test.ts holds the lists unique. The palette also gained
the arrow-key and Enter navigation its footer has always advertised, and the
shortcut hint says ⌘K on Apple keyboards.
Every prerendered page was wrong in ways only the built HTML shows:
- The consent dialog was in all of them. The gate rendered whenever
`accepted` was false, which it always is before storage is read, so
returning visitors saw it flash on every full load. It now waits for
consent.hydrated.
- It also covered /legal/*, the documents it asks people to accept, which
therefore could not be read before accepting them.
- `loading && !data` is false before a load starts, so /games shipped "No game
matches these filters.", /health "Nothing flagged" and the chart pages
"None". Resource.pending (no data, no error) replaces all 15 checks.
- Meta descriptions interpolated counts that are zero at build time ("Overview
of 0 F2P Steam games"); they are static now, and the headers show counts
only once data exists.
scripts/verify-dist.mjs reads dist/ in CI and fails on each of these. The
sidebar's "Manage" group is "Project" - nothing in a read-only app is managed -
and /publishers finally has a sidebar entry.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
43623e79d2 |
fix(worker): harden the admin surface, and add the tests ADMIN.md claimed
docs/ADMIN.md:473 says the CLI/Worker override parity is "asserted by a parity
test". There were NO tests in this repository at all - no *.test.ts, no
test_*.py, no vitest config, no test script. That claim is now true.
## The security defects, worst first
**Negative limit disabled the SQL bound.** admin.ts x2 and edit.ts all used
`Math.min(Number(raw) || fallback, max)`. Number("-1") is -1, which is truthy,
so `|| fallback` never fired and Math.min(-1, 200) is -1 - and SQLite treats a
negative LIMIT as NO limit. `/api/admin/audit?limit=-1` returned the entire,
never-pruned audit log; `?status=pending&limit=-1` the whole queue. Replaced
with clampLimit/clampOffset, which also truncate fractions (`?limit=1.9` bound
a non-integer). Table-tested, including a mutation run confirming the old
expression fails the new tests.
**exp was optional.** `typeof claims.exp === "number" && claims.exp < now`
accepts a token with no exp, or exp as a string - and such a token then never
expired. exp is now required; nbf, if present, must be numeric.
**CSRF was one heuristic deep.** access.ts accepts the JWT from the
CF_Authorization cookie, which a browser attaches to cross-site requests too,
so the fail-open Sec-Fetch-Site sniff was the only thing between a forced
cross-site POST and a repository write. State-changing requests now accept the
JWT ONLY from the Cf-Access-Jwt-Assertion header, which browsers never attach
on their own. curl and the discovery service token send it explicitly and are
unaffected; the Sec-Fetch-Site check stays as defence in depth.
**A key rotation locked everything out for an hour.** An unknown `kid` returned
null with no refetch against a 1-hour JWKS cache, so a Cloudflare rotation
denied admin AND ingest as an opaque 404. Now refetches once on a miss, floored
at 60s so junk kids cannot make this Worker hammer Cloudflare.
**An outage looked like a bad token.** A failed JWKS fetch threw inside the
catch-all and became 404. It is now AccessUnavailableError -> 503.
**Every server crash was reported as "Session expired".** fetch() had no
try/catch, so any throw became a Cloudflare 1101 with an HTML body - and both
admin pages sniff Content-Type and report exactly that string. The single most
misleading diagnostic in the admin surface, and it hid real D1 and GitHub
failures. Dispatch moved into route(); fetch() wraps it and logs the detail.
**A failed audit row reported a successful write as failed.** audit() was
awaited AFTER the commit landed and outside any try, so a D1 hiccup told the
reviewer the approval failed when it had already committed - and they would
retry it. Now best-effort with a loud log. edit.ts had its own copy of the
INSERT with the same defect; both use lib/audit.ts.
**A reject could downgrade an approval.** reconcile.ts writes
ingest_decisions('approved') only after observing the appid in data/, using DO
NOTHING; reject used a bare DO UPDATE, so the two disagreed about who may
overwrite whom. Added `WHERE ingest_decisions.decision <> 'approved'`, verified
against a real local D1: the approved row keeps its decision, reason and
attribution, the rejected row still updates.
**header_image was unvalidated.** Length-capped only, then rendered in an <img>
on the review screen; the page CSP happened to block it, which is the wrong
thing to rely on. Now must be a Steam CDN URL. Caught while testing: 100% of
the 3,665 real header_image values carry `?t=<mtime>`, so a pattern without it
would have blanked every queued game's image. All 3,665 pass; evil.com,
javascript:, data:, and the steamstatic.com.evil.com suffix trick do not.
**Concurrent reconciles.** The 15-min cron and the manual button could both run
a ~6MB fetch, and the button has no busy state. Added a per-isolate in-flight
guard, with a comment honest about its scope: cross-isolate overlap is still
possible and is left alone because every write there is idempotent.
**/api/admin/health echoed raw D1 and GitHub error text**, the latter including
GitHub's response body via github-app.ts. Now a status enum; detail to logs.
## Tests
19 tests. override-doc.test.ts round-trips the four REAL committed
data/overrides/*.json files - all Python-written - through the TypeScript
serialiser and asserts byte equality, which tests the actual property against
actual production data rather than a fixture written to match the code. edit.ts
now calls that shared serialiser instead of an inlined copy, so the test covers
the path that runs.
Verified the tests can fail: three mutations (old clamp expression, dropped
trailing newline, indent 4) each turned them red.
Wired into web-ci.yml, and into check-overrides.yml - which did not trigger on
web/worker/routes/edit.ts, so the "change one, change the other" mirror had no
CI enforcement in either direction.
## knip was blind to the entire Worker
knip.json's project globs were src/** only. Adding worker/** immediately found
five dead exports in the module holding the repo-write credential: commitFile
(deleted - an unused wrapper), readHeadAndFile, installationToken,
MANUAL_FIELDS and adminCsp (all now module-private). worker/env.d.ts is
ignored: it is an ambient declaration nothing imports by design.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
122075c9bf |
feat(admin): genre dropdown, bulk retag; ingest reduced to the extension
## /admin/edit
Genre is a dropdown instead of free text, built from the genres ACTUALLY in
use with their counts — the counts are the point, since they are how a vague
bucket announces itself ("Action" holds 287 games, "Adventure" 232). A
"Custom…" option still allows a value the catalogue has never used, and the
game's current genre is always offered even when absent from that list, so
loading a game can never silently change it.
New "Bulk by genre" mode: pick a genre, see everything carrying it, retag up
to ten in ONE commit. The cap is enforced in the UI (remaining checkboxes
disable, with "limit reached") and again on the server.
Supporting work:
- commitFiles() replaces commitFile(): several files in one commit, reading
the head and every file in ONE aliased GraphQL query so expectedHeadOid
still means what it says across ten files. Ten commits would be ten pushes,
ten workflow triggers and ten chances to land half a change.
- listByGenre() and genreCounts() in records.ts. Genres are counted by regex
over shard TEXT rather than parsing 3,426 records — verified to agree with a
full parse exactly (92 distinct, 3,426 games). listByGenre prefilters by
substring and only then parses, so cost scales with matches, not catalogue
size; an exact field check rejects the substring appearing in a description.
Found by driving the page: renderBulk() rebuilt all 60 cards on every checkbox
click, replacing the element just clicked and losing keyboard focus. Selection
now updates in place.
## Ingest: extension only
Removed the GitHub-issue and Telegram paths. Kept the browser extension, which
pushes to scripts/temp_info.jsonl directly, plus the /admin approve flow that
appends to the same file.
Deleted: bot-ingest.yml, ingest-from-issue.yml, ISSUE_TEMPLATE/add_games.yml,
docs/telegram-bot.md, and both copies of the bot QR.
Four things would have broken, and a fifth was already broken:
- BUILD BREAK. web/tsconfig.json sets noUnusedLocals, so removing the Telegram
and add-games UI orphans the PlusCircle and Bot icon imports — four TS6133
errors, web-ci red, site unbuildable.
- A live button. About.tsx linked issues/new?template=add_games.yml; GitHub
silently ignores an unknown template and drops the user on a blank form.
- CommandPalette linked "/add" — a route that does not exist. Every
`<Route path=…>` in web/src confirms it: left behind when sign-in and
in-app editing were removed, so selecting it navigated to the 404 page.
- notify-ci-failure.yml matches by exact workflow NAME and its own comment
warns a stale entry breaks silently.
- ALREADY BROKEN: README and CONTRIBUTING both linked
poli0981/free-steam-games-list-extension, which does not exist
(`gh repo view` → "Could not resolve"). The real repo is
poli0981/steam-f2p-extension. That was the link to the one path being kept.
README's Contribute section and CONTRIBUTING §2 still documented signing into
the web app with a Classic PAT — removed months ago. Both rewritten: the
extension is the submission route, issues are for reports, and discovery plus
/admin is how most games actually arrive now.
docs/ToS.md §5.1 governed sending a Telegram user_id to a bot that no longer
exists; removed, and TERMS_VERSION bumped 2 → 3 because ConsentGate exists to
re-ask when the terms change. Workflow counts corrected 24 → 23 in README and
SECURITY.md.
Kept deliberately: the maintainer's own @SkullMute0011 DM row and its QR, with
the "whitelist requests" framing dropped — that is a contact channel, not part
of the ingest path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
dbbd035a6d |
feat(data): human overrides — corrections that survive the pipeline
Stage 1 of editing existing games. Python only: no Worker, no D1, no new credential, no UI. The client is the command line. The problem it solves. MANUAL_FIELDS (anti_cheat, anti_cheat_note, is_kernel_ac, notes, type_game, safe, genre) are protected only by fill-if-empty — a scrape will not overwrite a non-empty value. That cannot tell a human correction from earlier scraper output, so `normalize_genres.py --apply`, which rewrites genre for EVERY game, silently reverts a fix, and refetch_all.py re-fetches everything. Nothing in the repo recorded that a human had ever decided anything. So an override is not a one-time edit. It is a standing instruction at data/overrides/<appid>.json, and apply_overrides() is called from inside save_main() — verified to be the single choke point: 11 scripts call it and grep finds nothing else writing data/data_*.jsonl. The correction is therefore re-imposed as the last act of every write, and no script needs to know this layer exists. Any future dataset writer inherits it for free. Four designs were explored independently and all four converged on this same hook, which is the main reason to trust it. Direct shard-writing from the Worker was rejected by every judge: shard identity is unstable (save_main re-chunks from scratch, so a game moves between shards), the commit must carry a whole file, and index.json has to be bumped in the same commit. Three rules that are not obvious, each of which is a real trap: - An override may NOT set a field empty. The next scrape would refill it and this layer would blank it again, rewriting shards on every run forever. Clearing is what retiring is for; empty values are refused with that message. - Retiring is not deleting. Deleting the file leaves the field pinned to the human value, because fill-if-empty never restores the old one. A retired entry keeps `was`, writes it back once, then permanently no-ops because the values no longer match — self-cleaning and idempotent by construction. - `notes` is shared with the pipeline, which appends its own segments (Dead game / Delisted / No longer free!). Those appends are ONE-WAY: mark_dead_games.py only calls append_note_idempotent() while is_dead is False, so once a game is flagged the marker is never re-added. A plain whole-value override of notes would erase it permanently, with nothing able to restore it. preserve_machine_notes() re-attaches any marker the human value would drop; the marker strings are asserted against their real producers. Deliberately tolerant at apply time: an unknown field, a bad value, a game no longer in the catalogue, or a malformed file is counted and reported, never raised — a typo must not abort a run that has already spent two hours scraping. That tolerance is also why silence would be dangerous, so .github/workflows/check-overrides.yml validates every override on push and PR. Verified against the real dataset: 20 assertions covering apply, idempotency (a second pass rewrites nothing and does not re-bump last_updated), survival of a simulated refetch and of normalize_genres, marker preservation on appid 3632680 (a genuinely dead game carrying the marker), retire-then-inert, rejection of all four bad-input classes, orphans, and one malformed file not stopping the others. An end-to-end `--apply` on the live catalogue produced exactly one changed line in one shard plus index.json.last_updated — no shard churn. With no overrides present the hook costs one isdir() and returns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
192f17622f |
chore: drop the bulk generated markdown, keep the two leaderboards
games/ held 1.8 MB of markdown regenerated from data/*.jsonl on every data
change: 18 all-games_part*.md files, anti-cheat-list.md, a README index of
them, and a .gen-hash checksum whose only purpose was to make regenerating
them cheaper. The web app has been the way to browse the catalogue since v3.0,
so these were a second rendering of the same records that had to be rebuilt,
committed and reviewed forever.
Kept: games/top-online.md and games/top-offline.md.
Removed with them, because nothing else used them:
scripts/generate_tables.py, scripts/generate_anti_cheat_list.py
.github/workflows/update-daily.yml ("Generate Markdown Tables")
.github/workflows/anti-cheat-list.yml
bash/daily.sh, bash/anti_cheat.sh
and the generate_tables call inside ingest.sh, purge.sh, refetch_all.sh and
ingest-from-issue.yml.
Nothing depended on the anti-cheat markdown: /charts/anti-cheat reads
data/*.jsonl, not that file.
Four references would have broken, two of them user-visible:
- web/public/sitemap.xml advertised all-games_part1.md to search engines, and
About.tsx had an "All games (markdown)" button. Both would now 404. The
sitemap entry is gone; the button points at data/ instead.
- .github/workflows/mark-dead-games.yml ended with
`gh workflow run "Generate Markdown Tables"`. That command exits non-zero
against a workflow that does not exist, so every successful dead-games run
would have finished red.
- notify-ci-failure.yml's allowlist is matched by exact workflow NAME, and the
file's own comment warns that a stale entry breaks silently.
Also fixed the stale poli0981.github.io URL in the bug-report template while
removing its games/*.md option.
Verified: typecheck, build, all 24 workflow YAML files parse, sitemap.xml is
well-formed, and no reference to the removed files survives outside CHANGELOG
and docs/plan (both historical).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
bb7c5f861d |
fix(ci): backfill workflow was an invalid file — expression inside a comment
Actions evaluates ${{ }} anywhere in the workflow file, shell comments in a
`run:` block included. A comment explaining why an expression fallback was the
wrong tool here quoted the rejected form verbatim, and the quoted contexts do
not exist — so the whole file failed to parse and every push produced a failed
run with no jobs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
a39ef5bf65 |
fix(ci): stop ingest-from-issue dropping requests and misreporting outcomes
Three defects on the same path, all of which fail silently. 1. A queued request could vanish. The only commit came AFTER `python scripts/ingest_new.py`, so a transient Steam failure aborted the job with the appended line living nowhere but the runner's disk. Nothing was committed, nothing retried, and the reporting step never ran to say so. The queue entry is now committed BEFORE the ingest is attempted, so a failure leaves a durable request instead of nothing. 2. The issue was answered "Game added!" for games that were not added. The check was `temp_info.jsonl === ''`, but clear_temp() empties that file at the end of every ingest_new.py run - including one where every entry was rejected as duplicate, dead, or unreachable. It now asks the dataset, and distinguishes "added" from "already in the catalogue" by snapshotting which appids were present before the run. 3. The reporting step had no `if: always()`, so it only ever ran on the happy path - the one case that needed no explanation. Also gives `Ingest New Game Links` a 3-hourly schedule. This is orphan recovery, not polling: `data-write` keeps at most ONE run pending and cancels the older one, so an ingest queued behind a ~115-minute data job is evicted the moment any other data-write workflow queues, and nothing retries it. Re-running on a clear file is free - ingest_new.py returns immediately and `git diff --staged --quiet` skips the commit - so it is idempotent by construction, and it is what makes fix 1's "will be retried" true. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
bb99685dce |
feat(ingest): automated discovery — Worker endpoint, daily sweep, backfill
The repo had no code that could find a new game. update_data.py only enriches
records that already exist and ingest_new.py only reads a file a human filled
in, so "surface new releases within a day" needed a producer, not rewiring.
Three pieces, split so that discovery is automated and publication is not:
/api/ingest/* service-token-authenticated producer surface. Writes rows
with status='pending' and can do nothing else - it cannot
approve, commit, or reach the repository.
discover_new.py daily. Walks store search newest-first, stops after two
fully-known pages.
backfill_discover TEMPORARY. Sweeps one release window at a time to close
the historical gap, then gets deleted.
Measured today: Steam lists 16,668 free games under the catalogue's own
filters; data/ holds 3,424.
The backfill locates a window by BINARY SEARCH over search offsets rather than
walking from page 0. The store sort is Released_DESC, so offset is the date
axis and page_date() is monotonic - about 8 fetches to reach a 2024 window
instead of 90. Verified against live Steam: 2025Q4 located at page 21 in 7
probes.
Both scripts run under `permissions: contents: read` plus a grep step that
fails the run if either references save_main/save_jsonl/clear_temp/fetch_full.
The permission alone would block a push; the grep names the reason, which a
403 from git does not.
Fixes found while building this:
- _RE_RELEASED required a `col ` prefix the live markup does not have
(`<div class="search_released responsive_secondrow">`), so every search row
came back with an empty date. The cheap "Coming soon" skip therefore never
fired and each such title cost a full appdetails call to reject.
- Game names are publisher-controlled and routinely non-Latin. print() of one
raised UnicodeEncodeError on a cp1252 console, killing the sweep after every
Steam request had already been paid for.
The two Access credentials are pinned apart on both sides: verifyAccessJwt
accepts any configured AUD, so without an explicit principal check an admin
session would satisfy /api/ingest/* and - far worse - the ingest token would
satisfy /api/admin/*.
Neither workflow can run until a Cloudflare Access service token named
`f2p-discovery` exists and its credentials are repository secrets; they fail
loudly rather than skipping, and docs/ADMIN.md §7 has the setup.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
c4805065db |
fix: stop two live paths from silently destroying the ingest queue
Both found while designing the admin write path, both already in production, and both lose data with no error anywhere. ingest-from-issue.yml overwrote scripts/temp_info.jsonl with fs.writeFileSync. That file is a shared queue with more than one producer — the Telegram bot appends to it, and the admin Worker is about to — so a single [add-game] issue destroyed every queued entry not yet consumed. The diff just showed a cleared file. It now appends, and validates that each line of the issue's json block is one complete JSON object before writing, because a pretty-printed block would trip the second bug below. load_jsonl wrapped its entire read loop in one try, so the first malformed line ended the read and returned only the lines before it. ingest_new.py then processed that partial list and called clear_temp(), truncating the file and deleting every unread line after the bad one. A bad line is now skipped and reported with its line number; only an OSError aborts the read. The logging in that handler is deliberately ASCII. The first version used the same warning glyph as the surrounding code and raised UnicodeEncodeError on a cp1252 console — from inside the except block, which escaped the handler and aborted the whole read, reintroducing exactly the loss being fixed. Logging in an error path must not be able to throw. Verified rather than assumed: the generated workflow JS passes node --check, appending preserves a pre-existing queue entry, and load_jsonl now returns both good lines from a file whose middle line is garbage. Also saves the four design and adversarial-review documents behind this work to docs/plan/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
5ab4f27347 |
docs: split the licence, correct the legal docs, add deployment and admin guides
The docs had drifted badly from the code. This makes them true. LICENCE SPLIT. Code stays MIT; the dataset moves to CC BY 4.0 in a new LICENSE-DATA, which is explicit about what the project cannot license: the `description` field is the publisher's own store blurb stored verbatim (median 213 characters), `header_image` is a URL to publisher artwork, and game names are trademarks used nominatively. Blanket-MIT over that material was a claim the project was never entitled to make. Facts — player counts, review percentages, platform support — are noted as unownable; the CC BY grant covers the compilation and the project's own derived fields. PRIVACY POLICY, rewritten rather than patched. The previous one said "no backend, no server-side log, no cookies, no server other than GitHub's CDN". Every clause of that is now false: the site runs on a Cloudflare Worker that processes IP and User-Agent and keeps operational logs. It also documents the one honest exception — the Activity page still calls api.github.com and loads GitHub avatars from the browser, so GitHub sees a visitor's IP there while no other page does — and lists every localStorage, IndexedDB and Cache Storage key with why it exists. SECURITY.md, rewritten. The old one claimed "no executable code, no dependencies, no backend, no user data" and said there was no contact address, while four other files publish one. It also told people to report vulnerabilities in public issues. There is now a real scope (the Worker, the proxies, D1, Actions secrets, the Tauri signing key), a private channel via GitHub's vulnerability reporting, and specific things worth probing. EULA corrected on two counts: it asserted a single MIT licence over everything, and claimed there is "no installable software" despite the signed desktop and Android builds. ToS and DISCLAIMER updated for the licence split, the new domain, and the removal of in-browser editing and GPG signing — the disclaimer described a signing flow that no longer exists. README contradictions fixed: 1.2k games -> 3,424 across 5 shards not 2, 11 workflows -> 24, two scripts listed that were never in the tree, a TanStack Table dependency the project does not use, and nine links to the old Pages URL. New: docs/DEPLOYMENT.md (how a push becomes a deploy, the two wrangler.jsonc fields that silently break it, and the stale-service-worker trap when verifying) and docs/ADMIN.md (Cloudflare Access plus the GitHub App, with the exact permission set — Contents write and Metadata only, and why Actions and Workflows are deliberately withheld). Two live fixes came out of writing this. GitHub Pages was serving a blank page: it had the new index.html, which references assets at `/`, while Pages serves under /free-steam-games-list/, so every asset 404'd — and deploy-pages.yml kept redeploying it. That workflow now publishes a redirect that preserves the #/ fragment, since Pages cannot issue a 301. And CI pinned Python at 3.11, 3.12 and 3.14 across different workflows while the docs claimed 3.12 was pinned; all fourteen references are now 3.12, which makes the claim true. TERMS_VERSION 1 -> 2, so everyone re-consents. A privacy policy that now discloses edge logging where it previously promised none is not a change to apply silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
692fcb4731 |
feat(cloudflare): Worker, wrangler.jsonc, and same-origin data + image proxies
Adds the deployment config and Worker that Cloudflare Workers Builds runs. There is deliberately no deploy workflow — pushing main is what deploys. Context for the base-path change, which the plan had scheduled later: the domain is already live and already broken. It currently serves the repo's web/ directory unbuilt, so `/` returns an index.html pointing at /src/main.tsx (TypeScript the browser cannot execute) and the whole source tree is publicly readable. `assets.directory: "./dist"` plus `base: "/"` is what fixes that, and neither is useful without the other, so they land here. wrangler.jsonc: - Lives in web/ because Workers Builds detects the package manager from a lockfile in its root directory, and there is no root package.json. - workers_dev: false — a zone-scoped Access policy does not cover the workers.dev hostname, which would later expose /admin and the repo-write credential behind it on an unprotected origin. - run_worker_first covers /api/*, /img/*, /admin and /admin/*. /img/* must be listed or images fall through to the SPA handler and return index.html as text/html; /admin must be listed or the SPA fallback serves the public shell where Access is expected. /assets/* is deliberately absent — routing hashed bundles through the Worker would bill every JS and CSS request. /api/data/* proxies the dataset that stays canonical in Git, against an anchored allowlist of exactly the three paths the app fetches. Proxying rather than shipping data/ inside dist/ keeps the daily pipeline commits decoupled from deploys: as build output, every data commit would force a full rebuild, and if watch paths ever excluded data/ the site would serve a frozen dataset with no error anywhere. index.json is sent no-cache because it carries last_updated, the client's only cache-invalidation signal. /img/* replaces the images.weserv.nl hop, so no visitor request reaches a third party. It allowlists both Steam CDN hosts — akamai and fastly, the latter being ~44% of records, which the old service-worker rule missed entirely. Thumbnails prefer Steam's capsule asset and fall back to the header when it does not exist (verified: 3,634 b vs 37,878 b on a cold appid, and the hashed-path records fall back correctly). Transformations are OFF: they bill separately from Workers Paid and Cloudflare has no spend cap, so real /img/* volume gets measured first. Also: tsconfig.worker.json, because tsconfig.json only includes ["src", "vite.config.ts"] and the Worker would otherwise ship untyped; web-ci.yml now runs both typechecks plus `wrangler deploy --dry-run`; 404.html is deleted as not_found_handling replaces it; and index.html, robots.txt and sitemap.xml stop declaring the GitHub Pages URL canonical now that this origin serves the site. Verified against a local `wrangler dev`: static assets, SPA fallback, all three data paths with correct cache headers, all three image variants, and 404s for unknown variants, non-numeric appids, non-allowlisted data paths and encoded traversal attempts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
eae8f6354d |
fix: detect MIME-type chunk failures, install pinned Python deps everywhere
isChunkLoadError only matched the three "failed to fetch dynamically imported module" style messages. Those cover a 404 on a stale chunk, but not what a host with an SPA fallback returns: a request for a deleted hashed chunk is answered with index.html at HTTP 200 and Content-Type: text/html, so the browser rejects it on MIME type instead. The message looks nothing like a fetch failure, so the reload-once recovery would silently stop working the moment the site moves behind Cloudflare's not_found_handling. Added the "failed to load module script" / "expected a javascript module script" patterns and lifted the regex to a named constant. Python installs now go through requirements.txt at all 11 sites — nine bash/*.sh scripts and the two inline workflow steps — instead of a bare `pip install requests`. The bare install silently ignored the urllib3 pin, so CI was resolving a dependency the requirements file was trying to constrain. mark-dead-games.yml already did this correctly. requirements.txt and requirements-dev.txt move from `>=x,<y` ranges to exact `==` pins: the pipeline should install identical versions on every run, and Dependabot does not raise PRs for range-specified dependencies, so the new pip ecosystem entry in dependabot.yml would otherwise have had nothing to do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
72d00c1014 |
fix(ci): drop the expired PAT, close an unauthenticated write path, land snapshots
The scheduled pipeline has been failing daily since 2026-08-13 (last successful
`Auto update` was
|