17 Commits
Author SHA1 Message Date
poli0981andClaude Opus 5.5 a618aeae2f docs: games in every language
- docs/plan/19-all-languages.md: what kept them out, what every workflow
  does with them, the re-sweep, and what is left.
- CLAUDE.md: discovery's language rules, ASCII appids, search folds but
  storage never does, re-cut descriptions are not news.
- docs/ADMIN.md: the filters as they are now, re-sweeping a whole year,
  the Languages line in the candidate dialog.
- README.md, docs/plan/11 (the decision changed), CHANGELOG [Unreleased].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 17:28:47 +07:00
poli0981andClaude Opus 5.5 529b8603c3 ci(pipeline): queue data-write runs and start each on the latest main
- queue: max on every data-write workflow (GitHub, 2026-05-07): up to
  100 runs wait, first in, first out. The default kept one, and a run
  queueing behind a running job cancelled it.
- ref: main on every checkout. A run's commit is fixed when it is
  created, so a run that waited behind another data job wrote shards
  from before that job's commit; Auto Update JSON failed four days in
  ten on the rebase conflict in data/index.json.
- timeout-minutes everywhere, and JOB_BUDGET_MINUTES at least 20 below
  it for the long loops (never for the ingest, which empties its queue).
- pipeline.test.ts fails on a data-write workflow missing any of these.
- The admin's Pipeline and Automation pages, docs/ADMIN.md and the
  auto-approve comment no longer say a push evicts a waiting job; the
  idle gate stays, so a batch's ingest starts at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 22:22:08 +07:00
poli0981andClaude Opus 5.5 7449489519 docs: anti-cheat is the developer's declaration on Steam; the extension is archived
- DISCLAIMER section 5 and ToS item 5: anti-cheat is read from the
  anti-cheat disclosure on the Steam store page, which the developer fills
  in; an undeclared one shows "-", and a declaration can be wrong. Both are
  consent documents: the web gate shows these two diffs once.
- README, CONTRIBUTING, Contact (shown at /legal/contact, not
  consent-gated): no more extension; a missing game or a correction goes in
  Discussions > Ideas; the manual fork path queues store links only.
- CLAUDE.md: the anti-cheat sweep and its rules, the trio as the one manual
  set a writer replaces (an override still wins), OverrideGuard, links-only
  queue, clean_notes, the pattern mirror; 15 data writers.
- ADMIN, dev_env (+vi), web/README: the extension is no longer a producer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:38:50 +07:00
poli0981andClaude Opus 5.5 997cbd9208 docs: removals, auto-approve and the new admin pages
CLAUDE.md:
- data/blocklist.jsonl as a data invariant: applied first in save_main(),
  logged once, the 500 guard, never named data_*, whole-line writers, the
  shared golden fixture, Restore as the only way back.
- The pipeline rules for ingest (blocked links skipped, non-games refused,
  ingest-new.yml as the removal job on ref main), auto-approve's gates, and
  Remove/Restore's D1 semantics and WRITABLE paths.
- "Publication stays a human decision" now says who decides: a reviewer, or
  the capped auto-approve rule a reviewer turned on.
- Admin routes: the Overview at /admin, the queue at /admin/queue, and no
  /admin/api route answering 401/403 for anything but Access.
- save_main() has 14 callers, not 11 (here and in core/overrides.py).

docs/ADMIN.md: the App's Contents permission covers temp_info, overrides
and the blocklist (never a shard); why Actions read is not needed; the
discovery token's reach with auto-approve on; sections 11-13 for
auto-approve, removing/restoring and the Overview/Pipeline/Quality pages;
the mock's mock_pipeline switch.

README.md: the admin's description, the new scripts and the ingest
workflow's triggers.

Also: the delete_game.py help example kept its line break, and the mock's
removed rows carry the reason's label as the Worker writes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 13:08:13 +07:00
poli0981andClaude Opus 5 4e206cb64c fix(admin): /admin wrote the maintainer's address into public Git
Every approval committed `Approved in /admin by <email>.`, every override
edit `Edited in /admin by <email>.`, and `data/overrides/<appid>.json`
recorded the same address as `set_by`. All three are public forever, and
the Git AUTHOR was already the App's own bot identity - only the body text
named a person.

They now say `admin` (ADMIN_ATTRIBUTION, shared/admin-api.ts). The four
override files that already carried the address are rewritten; `set_by` is
informational, so nothing in the pipeline reads it.

appendLinks() loses its `actor` PARAMETER rather than merely stopping at
the one call site that passed `who.email`: a parameter nobody can supply
cannot be supplied by mistake. A new case in git-commit.test.ts asserts the
generated body contains no `@` at all.

D1 is untouched and still records the acting Access identity in
audit_log.actor, ingest_queue.decided_by, ingest_decisions.decided_by and
commit_jobs.requested_by - 0001_init.sql calls that deliberate, and the
audit view filters on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 20:01:28 +07:00
poli0981andClaude Opus 5 0a4c5013c5 fix(admin): serve the admin API under /admin, in the page's Access app
/admin reported "Your Cloudflare Access session has expired" straight after
signing in, in a private window too. The page and its API were two Cloudflare
Access applications: /admin (AUD ca93...) and /api/admin (AUD a15d..., read
from the kid= of each login redirect). Access issues its session cookie per
application and fetch() cannot follow the sign-in redirect, so the session
for /admin never reached the API. Every call came back as an opaque redirect,
which api.ts correctly reads as an expired session, and a reload only renewed
the page's session. The Worker never answers 401/403 on these routes, so the
redirect could only have come from the edge.

The API now lives at /admin/api/*, under the page's own application:
ADMIN_API_PREFIX in shared/admin-routes.ts, used by the router, both
handlers, the SPA's api() (views pass routes like "queue?...") and the mock
backend. /api/admin/* is no longer served, and its AUD leaves
ACCESS_AUD_ADMIN.

Every existing test called the handlers directly, so none could see this.
worker/index.test.ts drives the real router and Access check with a signed
RS256 token: a reviewer reaches /admin/api/*, a write still needs the header
rather than the cookie alone, another application's token and the old prefix
both get 404. security.test.ts keeps the prefix under /admin and API paths
out of the views. Verified in preview:admin under the production CSP: queue,
stats, me, health and the ping write all answer from /admin/api/*.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 11:38:30 +07:00
poli0981andClaude Opus 5 727aadabf8 feat(admin): a Svelte admin app embedded in the Worker, with read-only decided rows
The two Worker-rendered admin pages (admin-ui.ts, edit-ui.ts: 1,400 lines of
HTML and script inside template literals) are replaced by a Svelte 5 app in
web/admin/. It is built on its own and embedded in the Worker as the gitignored
worker/generated/admin-bundle.ts, so no admin code ever reaches dist/, the
service worker's precache or the packaged apps. worker/routes/admin-spa.ts
serves the shell with a per-response nonce CSP after the existing Access checks,
serves hashed assets immutably, and 404s any other path under /admin.

Queue
- Rows that are decided (approved, committed, rejected) or whose game is already
  published have no checkbox, show a lock and are skipped by select-all. The
  rules come from shared/queue-rules.ts, the same module the Worker enforces.
- Decisions confirm every row first and report skipped rows with their reasons.
- Tabs with counts, cross-status search, sort, page size, all kept in the URL.
- Keyboard: j/k, x, a, r, d, e, /, Esc, ?. Ignored while typing or composing.
- Details dialog: payload, durable decision, other rows for the game, Reopen for
  a rejected row with the reason when it cannot be reopened.
- Phone layout as cards.

Edit
- One game: every change reviewed in a diff dialog before the commit; retire;
  delete a settled override file; unsaved-change guard.
- Several games: any field, set or retire, selection kept across pages and
  genres (up to 10), each game shown in the review.

Jobs, Audit and Health get their own paged, filterable pages. Health draws its
table on a 503 too, and Test write proves the write path.

An expired Access session reloads the page once; a refusal that survives the
reload is reported instead of looping. A non-JSON response is a server error,
never "session expired".

Local development: npm run dev:admin runs the real handlers against an in-memory
D1 with demo data, and npm run preview:admin serves the built bundle through the
real serveAdmin with the production CSP. Nothing in worker/ or admin/src imports
the mock, and the Worker has no authentication switch.

Also: design tokens move to src/styles/theme.css, shared by both builds;
verify-dist fails if admin code reaches dist/; vite-plugin-pwa is declared, as
src/app.d.ts references its types; docs/ADMIN.md, DEPLOYMENT.md and CLAUDE.md
describe the new admin.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 22:46:33 +07:00
poli0981andClaude Opus 5 290b1ad715 feat(admin): finish the review and edit screens, and test what a browser can't
/admin had four API endpoints with no UI at all and a set of controls that
lied about what they would do. This closes both.

## Views that did not exist

/api/admin/audit, /api/admin/health and /api/admin/me had no consumer, and
/api/admin/stats returned commit_jobs counts the page fetched and discarded -
so the one table that records a commit attempt BEFORE it is made, the evidence
that a commit died mid-flight, was only readable through `wrangler d1
execute`. Added a view switcher with Queue / Commit jobs / Audit log / Health,
plus a /api/admin/jobs endpoint to back it.

## Controls that lied

- Approve and Reject stayed enabled on the approved/committed/rejected tabs,
  where DECIDABLE guards both the server-side read and every UPDATE, so the
  click always returned 409 with no hint why. Now disabled, with the bar
  saying those rows are already decided.
- Reconcile had no busy state, so a double-click fired two ~6MB runs.
- Retire (edit page) had no guard at all, and the page had no busy flag: tab
  switching, Discard, Load and the genre picker all stayed live mid-commit.
- Retiring a field with no override is a server-side no-op, and the page still
  said "Retired genre". It now says nothing was committed.
- setMode() never cleared the other mode's state, so One -> Bulk -> One left
  `dirty` populated while the counter reported bulk.sel.size - the count and
  the payload disagreed.
- loadGenres() swallowed every error into an empty list with no message, so a
  failure left both pickers empty and bulk mode silently inert.
- No beforeunload guard: an unsaved edit vanished on navigate.

## Search was a lie too

The filter box filtered the 60 loaded rows, so an appid on page 3 reported
"Nothing on this page matches that filter" - indistinguishable from "not in
the queue". Search is now server-side, with LIKE ... ESCAPE so a name
containing % or _ is matched literally. Verified against a local D1: the
escaped form matches 1 row where the unescaped form matches 2.

Also added sort (newest/oldest/players/reviews/name) as a closed set mapped to
fixed SQL, and rendered reviews_pct and current_players_num - both selected by
the API since the schema was written, with a comment saying they exist "so the
review screen can sort", and never once displayed. NULLS LAST verified in D1.
payload_json, stored verbatim on every insert and never surfaced, now has a
details dialog. MAX_DECIDE comes from the API instead of a client literal.

## The "Session expired" lie, and the third console error

Both pages treated ANY non-JSON response as an expired session. Combined with
the Worker having had no top-level try/catch, that meant every server crash
was reported to the reviewer as a login problem. They now distinguish the
cases, and use `redirect: "manual"` so an expired Access session is OBSERVED
as a redirect rather than followed to cloudflareaccess.com - which is exactly
what produced the reported

  Connecting to 'https://polished-hall-48fd.cloudflareaccess.com/cdn-cgi/...'

console error: the CSP is connect-src 'self', so the cross-origin hop from
fetch() was blocked. Re-auth is now a full-page reload, which is the only way
Access can run its login flow.

## Tests

/admin cannot be loaded locally - it is behind Access and there is no way to
mint a JWT offline - so admin-ui.test.ts asserts everything checkable without
a browser: the nonce appears in both the header and the script tag and differs
per response, the actor is escaped (a hostile actor string does not survive as
markup), and crucially the emitted inline script PARSES. That last one is not
theoretical: the page lives inside a TypeScript template literal, so a
backtick in a comment silently ends it and a literal newline inside a JS
string is a parse error tsc cannot see. Both mistakes were made while writing
this change and caught by these tests; mutation-checked afterwards by
re-introducing each.

39 tests, up from 19.

## docs/ADMIN.md

Four corrections, all places the doc described something that does not exist:
the edit-drafts table (no such table - also fixed in wrangler.jsonc), the
`f2p-discovery` service-token NAME check (impossible - Cloudflare reports the
opaque Client ID - and it contradicted the accurate account earlier in the
same file), the parity test (now real, so it says what it actually is), and
"what is not built yet", which listed only audit pruning and now lists the
Worker's inability to delete an override file, single-field bulk mode,
selection loss across pages, un-rejecting being raw SQL, and the absence of a
cross-isolate reconcile lock.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 12:37:37 +07:00
poli0981andClaude Opus 5 56559a13f6 docs(contact): move to contact@poli0981.dev and one links page
Nine maintainer channels collapse to two: the email, and
https://poli0981.dev/links/ as the single source of truth for everything
else. Handles now live off-repo so a retired account cannot strand a dead
link in a shipped build — which had already happened three times.

Replaced lopop05905@proton.me at all 6 sites (AUTHORS.md, SECURITY.md,
docs/Contact.md, username.txt, and both the handle and the mailto: href in
About.tsx). Edited by hand rather than by regex: a naive email pattern
misses the 10 `github-actions[bot]@` occurrences and matches things that
must not change (the you@example.com placeholder, 128x128@2x.png, the
data/overrides set_by identities, ~200 `uses: …@v6` action pins).

Stale content removed or corrected while in here:

- CONTRIBUTING.md carried a whole WARNING block about whitelisting a
  Telegram user_id for a bot that no longer exists. Replaced with a
  general "public and permanent" privacy note.
- docs/Contact.md had an orphaned bullet body (lines 17-19) left behind
  when the Telegram ingest bot was deleted — no title line, dangling
  arrows. Section rewritten; it now says outright that the bot and the
  issue template are gone and the extension is the only submission path.
- docs/ADMIN.md credited temp_info.jsonl's appends to "the issue workflow
  and the Telegram bot". Both are gone; the producers are the extension
  and this Worker.
- README.md described CONTRIBUTING as covering "web-app sign-in, Telegram
  bot flow" (both removed) and ToS.md as holding Telegram user_id rules
  (it no longer mentions them).
- CODE_OF_CONDUCT.md had no reporting address at all. Added one.
- Two legacy /#/about URLs now point at the real /about path.
- Deleted the dead Telegram QR assets and the About page card that
  displayed one.

Licensing claims corrected in AUTHORS.md and twice in About.tsx: all
three still said the whole repo including data was MIT. The dataset moved
to CC BY 4.0 in LICENSE-DATA and these were the last places asserting
otherwise.

SECURITY.md's parenthetical claimed the address was published in
CONTRIBUTING.md — it never was, only a pointer. Replaced with the real
list plus a forward reference to the security.txt added next.

typecheck and build pass; knip reports only the pre-existing backlog.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 11:53:40 +07:00
poli0981andClaude Opus 5 fa2afd68d5 feat(admin): /admin/edit — correct a published game from the browser
Stage 2. The Worker writes the same artifact scripts/edit_game.py writes,
data/overrides/<appid>.json, so the two producers are interchangeable and this
screen is presentation over a mechanism that already worked without it. The
Worker still never touches data/.

  GET  /api/admin/game?appid=   current values + any existing override
  POST /api/admin/edit          set and/or retire fields
  /admin/edit                   the page, linked from the review queue

Reading one record out of 6 MB. worker/lib/records.ts fetches shards as TEXT,
finds the literal `/app/<appid>/`, slices that line to its newline boundaries
and JSON.parses only it — the needle is anchored on both sides by the canonical
link format, so it cannot match a longer appid or hit text inside a
description, and the record is re-checked against its own `link` because the
needle can legitimately appear in a note. Shard names come from index.json
rather than being hardcoded, and which shard a record was found in is never
retained: save_main() re-chunks the catalogue every run, so that fact is true
only for one request.

git-commit.ts gains a generic commitFile() sharing appendLinks' retry, and an
anchored path ALLOWLIST. The App credential can write anywhere in the
repository; WRITABLE is what stops a bug upstream from turning that into a
general-purpose commit engine. data/ shards are not on it and must never be.
commitFile()'s builder is re-run on every retry against the file's current
text, so losing a race merges rather than clobbers.

Two properties are asserted rather than assumed, because both fail silently:

- The Worker's JSON output is BYTE-IDENTICAL to save_override()'s. Verified at
  651 bytes each on a document containing non-Latin text, quotes, a backslash
  and an embedded newline. Had the formatting differed, every edit alternating
  between the CLI and the UI would have produced a whole-file diff.
- The Worker's validateValue() agrees with Python's validate_value() on 23
  cases, including the ones that matter: `is_kernel_ac` as the STRING "true"
  is rejected by both, a numeric genre is rejected by both, and both refuse an
  empty value with the same reason.

The Worker's validator is deliberately the weaker copy. Python is
authoritative, runs on every pipeline write, and check-overrides.yml runs it on
every push — and because the App token is not GITHUB_TOKEN, an override commit
triggers that workflow. So a file the UI accepts but Python would reject fails
CI within a minute instead of sitting inert.

`was` is read server-side from the catalogue and never taken from the request.
It is the value Retire restores, so a client-supplied one would let a crafted
request rewrite history.

Verified by driving the real page in a browser against a stubbed API, which
caught a bug reading alone did not: save() set the result banner and the reload
that followed wiped it — the same defect fixed earlier in admin-ui.ts, and it
was present on both the save and retire paths. Also confirmed: dirty tracking
clears when a field is typed back to its original, the boolean select sends a
real `true` rather than the string, an unknown appid surfaces the API error
instead of a blank form, and the `<script>` payload in a game name renders as
text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 23:56:45 +07:00
poli0981andClaude Opus 5 dbbd035a6d feat(data): human overrides — corrections that survive the pipeline
Stage 1 of editing existing games. Python only: no Worker, no D1, no new
credential, no UI. The client is the command line.

The problem it solves. MANUAL_FIELDS (anti_cheat, anti_cheat_note,
is_kernel_ac, notes, type_game, safe, genre) are protected only by
fill-if-empty — a scrape will not overwrite a non-empty value. That cannot
tell a human correction from earlier scraper output, so
`normalize_genres.py --apply`, which rewrites genre for EVERY game, silently
reverts a fix, and refetch_all.py re-fetches everything. Nothing in the repo
recorded that a human had ever decided anything.

So an override is not a one-time edit. It is a standing instruction at
data/overrides/<appid>.json, and apply_overrides() is called from inside
save_main() — verified to be the single choke point: 11 scripts call it and
grep finds nothing else writing data/data_*.jsonl. The correction is therefore
re-imposed as the last act of every write, and no script needs to know this
layer exists. Any future dataset writer inherits it for free.

Four designs were explored independently and all four converged on this same
hook, which is the main reason to trust it. Direct shard-writing from the
Worker was rejected by every judge: shard identity is unstable (save_main
re-chunks from scratch, so a game moves between shards), the commit must carry
a whole file, and index.json has to be bumped in the same commit.

Three rules that are not obvious, each of which is a real trap:

- An override may NOT set a field empty. The next scrape would refill it and
  this layer would blank it again, rewriting shards on every run forever.
  Clearing is what retiring is for; empty values are refused with that message.
- Retiring is not deleting. Deleting the file leaves the field pinned to the
  human value, because fill-if-empty never restores the old one. A retired
  entry keeps `was`, writes it back once, then permanently no-ops because the
  values no longer match — self-cleaning and idempotent by construction.
- `notes` is shared with the pipeline, which appends its own segments
  (Dead game / Delisted / No longer free!). Those appends are ONE-WAY:
  mark_dead_games.py only calls append_note_idempotent() while is_dead is
  False, so once a game is flagged the marker is never re-added. A plain
  whole-value override of notes would erase it permanently, with nothing able
  to restore it. preserve_machine_notes() re-attaches any marker the human
  value would drop; the marker strings are asserted against their real
  producers.

Deliberately tolerant at apply time: an unknown field, a bad value, a game no
longer in the catalogue, or a malformed file is counted and reported, never
raised — a typo must not abort a run that has already spent two hours scraping.
That tolerance is also why silence would be dangerous, so
.github/workflows/check-overrides.yml validates every override on push and PR.

Verified against the real dataset: 20 assertions covering apply, idempotency
(a second pass rewrites nothing and does not re-bump last_updated), survival of
a simulated refetch and of normalize_genres, marker preservation on appid
3632680 (a genuinely dead game carrying the marker), retire-then-inert,
rejection of all four bad-input classes, orphans, and one malformed file not
stopping the others. An end-to-end `--apply` on the live catalogue produced
exactly one changed line in one shard plus index.json.last_updated — no shard
churn. With no overrides present the hook costs one isdir() and returns.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 23:47:24 +07:00
poli0981andClaude Opus 5 38a30be9ad fix(admin): second review round — retry classification, status guards, docs
Re-ran the review that had 27 of 49 verifier agents die on a usage limit, this
time adjudicating each unverified claim against current code. 22 agents, none
failed. Most of the outstanding claims did not survive; four did.

**The retry was dead code.** appendLinks classified a stale-head rejection by
regex over GitHub's prose, requiring the literal substring "oid". GitHub
actually answers with type "STALE_DATA" and the message `Expected branch to
point to "<sha>" but it did not. Pull and try again.` — where the only variable
span is a hex sha, whose alphabet contains neither 'o' nor 'i'. So the pattern
could never match, the first lost race rethrew immediately, and MAX_ATTEMPTS
was unreachable for the one purpose the module documents it for. Worse, the
alternation DID match `Ref cannot be updated` — a permanent branch-protection
rejection — burning four round trips before reporting it as a race.

Now classified by fact rather than wording: the machine-readable "STALE_DATA"
type as a fast path, falling back to re-reading the head and comparing. If the
branch moved we lost a race and retrying is right; if it did not, the failure
will repeat identically and belongs in front of the operator now.

**reject and defer/requeue could overwrite a decided row.** approve carried
`AND status IN ('pending','deferred','failed')` on its UPDATE; the other two
used a bare `WHERE id=?`. Rows are pre-filtered by DECIDABLE when read, but
that is TOCTOU — an admin page held open across a reconcile, or two reviewers
acting at once, and a stale page could reject a game whose approval had already
been committed to Git, or one already published. All branches now carry the
guard, with a note at DECIDABLE saying why it is enforced twice.

**A reconcile that failed outright looked like success.** The "ok" argument sat
outside the ternary, so `index unreachable` and `shard unreachable` painted the
green banner. Split on the row count rather than the message — bail() always
reports checked >= 1 and the benign idle skip reports 0 — so it will not drift
when the wording changes. The banner now also reports aged-out rows and says
what to do about them, which the API returned and the UI discarded.

**docs/ADMIN.md never caught up with the per-route AUD refactor.** It still told
operators to add new AUDs to `ACCESS_AUD`, a var that no longer exists, and
still documented `INGEST_SERVICE_PRINCIPAL` and a `ingest: principal not
allowed` log line that were both removed in ed30c5da. Following it would
produce exactly the `aud mismatch` the section exists to prevent. Corrected,
with the two per-route vars tabulated, plus a table of the three
`reject_reason` values a `failed` row can carry and what each requires.

One correction the adjudicator made to itself, kept: renaming the service token
is NOT harmless. The Worker-side name check is gone, but the name still has to
match the Service Token rule in the Access policy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 23:01:29 +07:00
poli0981andClaude Opus 5 16fa13dd3c feat(admin): review queue at /admin, approve-to-commit, reconcile cron
Completes the human side of the ingest pipeline. 646 discovered games were
sitting in D1 with no way to act on them except wrangler d1 execute.

  /admin                     review screen, served BY THE WORKER
  POST /api/admin/decide     approve / reject / defer / requeue, in bulk
  POST /api/admin/reconcile  run the reconciler now
  GET  /api/admin/stats      per-status counts for the tabs
  worker/lib/git-commit.ts   GraphQL createCommitOnBranch -> temp_info.jsonl
  worker/lib/reconcile.ts    promotes 'approved' only after seeing data/
  scheduled()                the same pass, every 15 minutes

The page is served by the Worker rather than added to the SPA. The public app
has no sign-in and no editing — that was removed deliberately — so admin markup
and endpoint names have no business in a bundle every visitor downloads, and
/admin cannot fall through to the public shell if a route changes. It ships its
own CSP with a per-response nonce; every value from Steam or D1 is written with
textContent.

The three actions are deliberately asymmetric:

  reject   final and local. Writes ingest_decisions('rejected'), which is what
           actually stops tomorrow's sweep re-offering the game — the partial
           unique index only covers OPEN rows.
  defer    parks the row while keeping the appid blocked.
  approve  does NOT publish. It appends to scripts/temp_info.jsonl and stops.

Approve stopping there is the whole design. ingest_new.py still decides whether
a game is real, free and reachable, so a commit is a REQUEST, not an outcome.
Only reconcile.ts, after observing the appid in data/, may write
ingest_decisions('approved') — because /api/ingest/known feeds decided appids
back to the discovery sweep, and marking something decided that never got
published makes it invisible to every future sweep.

Commits use createCommitOnBranch so GitHub signs them (Verified), carry
expectedHeadOid so a concurrent pipeline commit is retried against the newer
file rather than overwritten, and APPEND — the issue workflow and Telegram bot
write to that file too. Because the App installation token is not GITHUB_TOKEN,
the push triggers Ingest New Game Links on its own.

Verified before committing: the page was rendered and driven in a real browser
against a stubbed API (Access makes /admin unreachable to anything but the
owner's login). That caught three bugs a read-through did not — the commit
confirmation being wiped by the reload that followed it, the master checkbox
staying ticked with nothing selected, and unsequenced loads letting a stale
response paint under the wrong tab.

An adversarial review of this code found more, of which the serious ones:

- A game could be lost permanently and silently. ingest_new.py SKIPS an entry
  that hits a network error, writes nothing to removed_games.jsonl, then clears
  temp_info.jsonl regardless — so the request is gone and the game is in
  neither file. The row would sit in 'approved' forever, and since
  /api/ingest/known counts 'approved' as known, no future sweep would ever
  offer that appid again. No UI action could reach it either: 'approved' is not
  in DECIDABLE. Approvals unobserved after 12h now age out to 'failed', which
  is decidable, and re-approving re-commits the link. 12h, not less: ingest-new
  shares the data-write concurrency group with ~115-minute jobs.
- removed_games.jsonl is unpruned history going back to March. Matching on
  appid alone marked a fresh approval 'failed' on the strength of a months-old
  entry, so a game that stopped being free and later became free again could
  never be re-added. Now requires removed_at > decided_at.
- 'failed' is outside uq_queue_open_appid, so a failed row can coexist with a
  new pending row for the same appid. Approving both violated the index inside
  a D1 batch AFTER the commit had landed. Rival open rows are now stood down in
  the same batch, before the promotion, and a desync is reported rather than
  swallowed.
- appendLinks deduplicates its input; approving two rows sharing an appid wrote
  the link twice.
- A no-op approval returned the branch head as its commit sha, so commit_jobs
  and audit_log attributed an unrelated commit to the approval.
- "Select all on this page" ignored the active filter: filtering to one game
  and ticking it selected all 60 loaded rows, 59 of them invisible, and the
  next click would have approved them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 22:37:57 +07:00
poli0981andClaude Opus 5 86061611a9 docs(admin): distinguish the three Access setup failures
All three present as "the endpoint is broken" and each has a different fix.
Cloudflare documents the Service Auth requirement explicitly; the AUD
allowlist and the principal pin are this repo's own additions and were
undocumented.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 19:21:14 +07:00
poli0981andClaude Opus 5 3f9447711d fix(repo): close .gitignore gaps that would leak credentials, untrack 3 files
You spotted .wrangler cache in the repo. It was not a one-off: the root
.gitignore had no .wrangler rule at all (only web/.gitignore did), so anything
run with wrangler from the REPO ROOT rather than web/ lands unignored. That is
how .wrangler/cache/wrangler-account.json — Cloudflare account id and account
email — reached a public repo.

Auditing outward from that found three more credential shapes with no rule,
all of which this project's own docs tell you to create:

- .dev.vars / .dev.vars.*  `wrangler dev` reads GH_APP_PRIVATE_KEY from here,
  and the PEM is INLINE, so the existing *.pem and *.key rules never see it.
  That App can commit to this repo. Highest-consequence gap of the three.
- *.jks / *.keystore / *.p12 / *.pfx  web/src-tauri/TAURI.md walks you through
  `keytool -genkeypair -keystore f2p-tracker-release.jks` using a RELATIVE path
  inside gen/android/, and reassures you keystore.properties is ignored — while
  the keystore itself was covered by nothing. Leak it and Android signing is
  compromised; lose it and the app can never be updated.
- .env.production / .env.development matched none of .env, .env.local or
  .env.*.local.

Two rules were silently hiding real source, verified with check-ignore:

- `env/` and `ENV/` are unanchored Python-virtualenv rules, so they matched
  web/worker/env/ at ANY depth — Worker source would vanish from `git add .`.
  (On this Windows checkout core.ignorecase makes ENV/ match env/ as well.)
- `[Nn]otes.md` matched docs/notes.md and docs/i18n/*/notes.md, the likeliest
  filename for real documentation in a repo whose output is markdown.
Both anchored to the repo root. `build/`, `dist/` and the one-off script rules
anchored for the same reason; web/dist stays covered by web/.gitignore.

Two rules did not do what their comments claimed:

- `*.log` matches scripts/dead_games.log, which mark-dead-games.yml explicitly
  stages and mark_dead_games.py appends to. It survives only because ignore
  rules do not apply to already-tracked files — rename or recreate it once and
  it disappears with no error. Now an explicit negation with the reason.
- `.vscode/` excludes the DIRECTORY, and git will not descend into an excluded
  directory, so the four `!.vscode/*.json` negations under it were dead code.
  Changed to `.vscode/*` so the exceptions actually apply.

Untracked (files left on disk, none referenced by any build):
- .wrangler/cache/wrangler-account.json  account id + email, the reported file
- web/src-tauri/.tauri-signing.key.pub   minisign F6E36504EAE8E3A2, which is
  NOT the updater key in tauri.conf.json (9F39774AE5453C53). A stale pre-
  rotation public key that web/src-tauri/.gitignore already forbids.
- .idea/.gitignore                       tracked despite the .idea/ rule

Verified: 21 check-ignore assertions both ways; the index delta is exactly
those three deletions (386 -> 383 files); no remaining tracked file is matched
by any ignore rule; nothing untracked is left unignored.

Also documents in docs/ADMIN.md that every new Access application needs its AUD
appended to ACCESS_AUD — omitted before, and it fails only after the
application starts working, which makes it look like an unrelated bug.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 19:03:36 +07:00
poli0981andClaude Opus 5 bb99685dce feat(ingest): automated discovery — Worker endpoint, daily sweep, backfill
The repo had no code that could find a new game. update_data.py only enriches
records that already exist and ingest_new.py only reads a file a human filled
in, so "surface new releases within a day" needed a producer, not rewiring.

Three pieces, split so that discovery is automated and publication is not:

  /api/ingest/*      service-token-authenticated producer surface. Writes rows
                     with status='pending' and can do nothing else - it cannot
                     approve, commit, or reach the repository.
  discover_new.py    daily. Walks store search newest-first, stops after two
                     fully-known pages.
  backfill_discover  TEMPORARY. Sweeps one release window at a time to close
                     the historical gap, then gets deleted.

Measured today: Steam lists 16,668 free games under the catalogue's own
filters; data/ holds 3,424.

The backfill locates a window by BINARY SEARCH over search offsets rather than
walking from page 0. The store sort is Released_DESC, so offset is the date
axis and page_date() is monotonic - about 8 fetches to reach a 2024 window
instead of 90. Verified against live Steam: 2025Q4 located at page 21 in 7
probes.

Both scripts run under `permissions: contents: read` plus a grep step that
fails the run if either references save_main/save_jsonl/clear_temp/fetch_full.
The permission alone would block a push; the grep names the reason, which a
403 from git does not.

Fixes found while building this:

- _RE_RELEASED required a `col ` prefix the live markup does not have
  (`<div class="search_released responsive_secondrow">`), so every search row
  came back with an empty date. The cheap "Coming soon" skip therefore never
  fired and each such title cost a full appdetails call to reject.
- Game names are publisher-controlled and routinely non-Latin. print() of one
  raised UnicodeEncodeError on a cp1252 console, killing the sweep after every
  Steam request had already been paid for.

The two Access credentials are pinned apart on both sides: verifyAccessJwt
accepts any configured AUD, so without an explicit principal check an admin
session would satisfy /api/ingest/* and - far worse - the ingest token would
satisfy /api/admin/*.

Neither workflow can run until a Cloudflare Access service token named
`f2p-discovery` exists and its credentials are repository secrets; they fail
loudly rather than skipping, and docs/ADMIN.md §7 has the setup.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 18:17:32 +07:00
poli0981andClaude Opus 5 5ab4f27347 docs: split the licence, correct the legal docs, add deployment and admin guides
The docs had drifted badly from the code. This makes them true.

LICENCE SPLIT. Code stays MIT; the dataset moves to CC BY 4.0 in a new
LICENSE-DATA, which is explicit about what the project cannot license: the
`description` field is the publisher's own store blurb stored verbatim (median
213 characters), `header_image` is a URL to publisher artwork, and game names
are trademarks used nominatively. Blanket-MIT over that material was a claim
the project was never entitled to make. Facts — player counts, review
percentages, platform support — are noted as unownable; the CC BY grant covers
the compilation and the project's own derived fields.

PRIVACY POLICY, rewritten rather than patched. The previous one said "no
backend, no server-side log, no cookies, no server other than GitHub's CDN".
Every clause of that is now false: the site runs on a Cloudflare Worker that
processes IP and User-Agent and keeps operational logs. It also documents the
one honest exception — the Activity page still calls api.github.com and loads
GitHub avatars from the browser, so GitHub sees a visitor's IP there while no
other page does — and lists every localStorage, IndexedDB and Cache Storage key
with why it exists.

SECURITY.md, rewritten. The old one claimed "no executable code, no
dependencies, no backend, no user data" and said there was no contact address,
while four other files publish one. It also told people to report
vulnerabilities in public issues. There is now a real scope (the Worker, the
proxies, D1, Actions secrets, the Tauri signing key), a private channel via
GitHub's vulnerability reporting, and specific things worth probing.

EULA corrected on two counts: it asserted a single MIT licence over everything,
and claimed there is "no installable software" despite the signed desktop and
Android builds. ToS and DISCLAIMER updated for the licence split, the new
domain, and the removal of in-browser editing and GPG signing — the disclaimer
described a signing flow that no longer exists.

README contradictions fixed: 1.2k games -> 3,424 across 5 shards not 2, 11
workflows -> 24, two scripts listed that were never in the tree, a TanStack
Table dependency the project does not use, and nine links to the old Pages URL.

New: docs/DEPLOYMENT.md (how a push becomes a deploy, the two wrangler.jsonc
fields that silently break it, and the stale-service-worker trap when verifying)
and docs/ADMIN.md (Cloudflare Access plus the GitHub App, with the exact
permission set — Contents write and Metadata only, and why Actions and
Workflows are deliberately withheld).

Two live fixes came out of writing this. GitHub Pages was serving a blank page:
it had the new index.html, which references assets at `/`, while Pages serves
under /free-steam-games-list/, so every asset 404'd — and deploy-pages.yml kept
redeploying it. That workflow now publishes a redirect that preserves the #/
fragment, since Pages cannot issue a 301. And CI pinned Python at 3.11, 3.12
and 3.14 across different workflows while the docs claimed 3.12 was pinned;
all fourteen references are now 3.12, which makes the claim true.

TERMS_VERSION 1 -> 2, so everyone re-consents. A privacy policy that now
discloses edge logging where it previously promised none is not a change to
apply silently.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-10 16:09:04 +07:00