- docs/plan/19-all-languages.md: what kept them out, what every workflow does with them, the re-sweep, and what is left. - CLAUDE.md: discovery's language rules, ASCII appids, search folds but storage never does, re-cut descriptions are not news. - docs/ADMIN.md: the filters as they are now, re-sweeping a whole year, the Languages line in the candidate dialog. - README.md, docs/plan/11 (the decision changed), CHANGELOG [Unreleased]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
836 lines
41 KiB
Markdown
836 lines
41 KiB
Markdown
# Admin setup
|
||
|
||
How the maintainer-only surface is wired: Cloudflare Access in front of
|
||
`/admin`, and a GitHub App behind it so the Worker can commit to the
|
||
repository.
|
||
|
||
Nothing here is needed to run or browse the site. The public app is read-only.
|
||
|
||
---
|
||
|
||
## Why a GitHub App rather than a token
|
||
|
||
The site previously kept a GitHub personal access token in the browser's
|
||
localStorage on a public origin. That is gone. The replacement needs a
|
||
credential that lives only on the server, and a GitHub App is the right shape:
|
||
|
||
| | Classic PAT | Fine-grained PAT | GitHub App |
|
||
|---|---|---|---|
|
||
| Expires silently | yes — this stalled the pipeline for a month in Aug 2026 | yes, max 366 days | **no** |
|
||
| Scope | whole account | per-repo | per-repo, per-permission |
|
||
| Credential at rest | the token itself | the token itself | a private key that mints 1-hour tokens |
|
||
| Attribution in history | your account | your account | its own bot identity |
|
||
|
||
Neither the App's identity nor yours appears as a name in what `/admin` writes: commit bodies, `data/overrides/*.json` and `data/blocklist.jsonl` say **`admin`** (`ADMIN_ATTRIBUTION` in `web/shared/admin-api.ts`), because all three are public forever. The acting Access identity is recorded in D1 — `audit_log.actor`, `ingest_queue.decided_by`, `ingest_decisions.decided_by`, `commit_jobs.requested_by` — which is where `/admin/audit` reads it from and filters on it.
|
||
|
||
The App's private key never expires; the tokens it mints die in an hour. That
|
||
removes the failure mode that already bit this project once.
|
||
|
||
---
|
||
|
||
## 1. Create the App
|
||
|
||
**GitHub → Settings → Developer settings → GitHub Apps → New GitHub App**
|
||
(<https://github.com/settings/apps/new>)
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| GitHub App name | `f2p-tracker-admin` (must be globally unique) |
|
||
| Homepage URL | `https://free-steam-games.win` |
|
||
| Webhook | **Uncheck "Active"** — the Worker calls GitHub, not the reverse |
|
||
| Where can this be installed | **Only on this account** |
|
||
|
||
### Permissions
|
||
|
||
Under **Permissions → Repository permissions**, set exactly these and nothing
|
||
else:
|
||
|
||
| Permission | Access | Why |
|
||
|---|---|---|
|
||
| **Contents** | **Read and write** | commit to `scripts/temp_info.jsonl`, `data/overrides/*.json` and `data/blocklist.jsonl` — and nothing else: the Worker refuses every other path (`WRITABLE` in `web/worker/lib/git-commit.ts`), so it never writes a shard or `data/index.json` |
|
||
| **Metadata** | Read-only | mandatory; GitHub adds it automatically |
|
||
|
||
Leave everything else at **No access**. In particular:
|
||
|
||
- **Actions — not needed.** A push made with an App installation token *does*
|
||
trigger workflows (unlike the built-in `GITHUB_TOKEN`), so committing to
|
||
`scripts/temp_info.jsonl` starts `ingest-new.yml` on its own. Do not grant
|
||
Actions write just to dispatch runs. The Pipeline page and auto-approve
|
||
*read* workflow runs, which a public repository allows without it; if GitHub
|
||
ever refuses the token, the Worker retries anonymously (a small, shared
|
||
budget), and auto-approve waits rather than guess. Granting **Actions:
|
||
Read-only** would make that read independent of the anonymous budget, and is
|
||
harmless.
|
||
- **Issues, Pull requests, Workflows — not needed.** The Worker only writes
|
||
data files. Granting `Workflows` would let a compromised Worker rewrite
|
||
`.github/workflows/**`, which is the single worst permission to hand out.
|
||
|
||
Click **Create GitHub App**.
|
||
|
||
## 2. Collect the credentials
|
||
|
||
On the App's settings page:
|
||
|
||
1. Note the **App ID** (a number near the top).
|
||
2. **Private keys → Generate a private key.** A `.pem` file downloads. This is
|
||
the credential — treat it like a password. GitHub keeps no copy; if you lose
|
||
it, generate a new one and delete the old.
|
||
|
||
## 3. Install it on the repository
|
||
|
||
**Install App** in the left sidebar → your account → **Only select
|
||
repositories** → `free-steam-games-list` → Install.
|
||
|
||
Then note the **Installation ID**, which is the trailing number in the URL of
|
||
*Settings → Applications → Installed GitHub Apps → Configure*:
|
||
`https://github.com/settings/installations/<INSTALLATION_ID>`.
|
||
|
||
## 4. Give the Worker the credentials
|
||
|
||
From `web/`:
|
||
|
||
```bash
|
||
npx wrangler secret put GH_APP_ID
|
||
npx wrangler secret put GH_APP_INSTALLATION_ID
|
||
npx wrangler secret put GH_APP_PRIVATE_KEY
|
||
```
|
||
|
||
For the private key, paste the **entire** `.pem` including the
|
||
`-----BEGIN RSA PRIVATE KEY-----` and `-----END RSA PRIVATE KEY-----` lines.
|
||
|
||
Secrets set this way are encrypted at rest and are **not** in `wrangler.jsonc`,
|
||
so they never reach the repository. Never put them in `vars`.
|
||
|
||
The Worker signs a short JWT with the private key, exchanges it for an
|
||
installation token, and uses that token for about an hour before minting
|
||
another.
|
||
|
||
## 5. Put Cloudflare Access in front of `/admin`
|
||
|
||
The App is only half of it — without Access, `/admin` is public.
|
||
|
||
**Cloudflare dashboard → Zero Trust → Access controls → Applications → Add an
|
||
application → Self-hosted.**
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Application name | `f2p-tracker-admin` |
|
||
| Session duration | 24 hours or less |
|
||
| Domain | `free-steam-games.win`, path `admin` |
|
||
|
||
That one application covers the admin's API as well: it lives at
|
||
`/admin/api/*`, under the page's path. **Do not give the API an application
|
||
of its own.** Access issues its session cookie per application, and the page
|
||
calls its API with `fetch()`, which cannot follow Access's sign-in redirect -
|
||
so every call fails and the page reports "Your Cloudflare Access session has
|
||
expired", even straight after a fresh sign-in, in a private window too. The
|
||
API used to be `/api/admin/*` behind a second application and failed exactly
|
||
that way. If that application (`free-steam-games.win/api/admin`) still
|
||
exists, delete it: nothing is served there any more.
|
||
|
||
**Policy:** Action *Allow*, rule *Emails* → your address. Access applications
|
||
are deny-by-default, so everyone else gets the login screen and never reaches
|
||
the Worker.
|
||
|
||
Zero Trust's free plan covers up to 50 users, which is 49 more than this needs.
|
||
|
||
### Two things that are easy to get wrong
|
||
|
||
- **`workers_dev` must stay `false`.** A zone-scoped Access policy does not
|
||
cover `*.workers.dev`. If that hostname is live, `/admin` is reachable
|
||
around Access, with the repo-write credential behind it.
|
||
`web/wrangler.jsonc` already sets `workers_dev: false` — keep it that way.
|
||
- **The Worker must verify the Access JWT itself.** Do not trust the path
|
||
alone. Access forwards a `Cf-Access-Jwt-Assertion` header; the Worker
|
||
validates it against the team JWKS and checks the `aud` claim. Belt and
|
||
braces, because the cost of getting this wrong is repository write access.
|
||
|
||
Add a second Access identity (a backup email, or a one-time-PIN policy) before
|
||
you rely on this. If you lock yourself out, the break-glass path is editing
|
||
`data/*.jsonl` directly on github.com — which works precisely because Git, not
|
||
D1, remains the source of truth.
|
||
|
||
## 6. The database
|
||
|
||
`f2p-admin` (D1, region APAC) already exists and is bound as `DB` in
|
||
`web/wrangler.jsonc`. It never holds game records. Its tables:
|
||
|
||
| Table | Holds | Migration |
|
||
|---|---|---|
|
||
| `ingest_queue` | candidate games and their review status | `0001_init.sql` |
|
||
| `ingest_decisions` | the durable approved/rejected decision per appid | `0001_init.sql` |
|
||
| `commit_jobs` | every commit the Worker attempted, written before the attempt | `0001_init.sql` |
|
||
| `audit_log` | every admin action | `0001_init.sql` |
|
||
| `admin_locks` | the reconcile lease, so the cron and a manual run never overlap | `0002_admin_state.sql` |
|
||
| `admin_state` | the dataset generation the last sweep saw | `0002_admin_state.sql` |
|
||
|
||
(An older version of this page mentioned "edit drafts"; there is no such table.
|
||
`/admin/api/edit` commits straight to Git with no draft stage.) Migrations live
|
||
in `web/worker/migrations/` and are applied out of band:
|
||
|
||
```bash
|
||
npx wrangler d1 migrations apply f2p-admin --remote
|
||
```
|
||
|
||
The Worker tolerates a missing `0002`: reconcile falls back to its per-isolate
|
||
guard and skips the sweep watermark, and `/admin/health` lists the migrations
|
||
that are not applied yet. Deploying the Worker before applying the migration is
|
||
therefore safe, just degraded.
|
||
|
||
`audit_log` and `commit_jobs` rows older than `ADMIN_RETENTION_DAYS` (180 by
|
||
default, set in `wrangler.jsonc`) are deleted once a day by the cron, and the
|
||
deletion itself is recorded as an `admin.prune` audit row. The privacy policy
|
||
states the same window; change both together.
|
||
|
||
Confirm the database is on a paid plan before depending on it: since
|
||
2026-09-01, queries that exceed the free daily row limits **fail** rather than
|
||
throttle, including the queries you would use to diagnose the problem.
|
||
|
||
## 7. A service token for the discovery pipeline
|
||
|
||
`Discover New Games` runs on a GitHub Actions runner. There is no browser and
|
||
no human, so it cannot pass an email policy — it needs a **service token**.
|
||
|
||
**Zero Trust → Access controls → Service auth → Create Service Token.**
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| Name | `f2p-discovery` |
|
||
|
||
The name still matters, but on the Cloudflare side only: it is what the Access
|
||
policy's *Service Token* rule refers to. Rename the token in Zero Trust without
|
||
updating that rule and discovery stops working.
|
||
|
||
The Worker no longer checks the token's name. It cannot: Access reports a
|
||
service token's identity as its opaque Client ID (`<32 hex>.access`), never its
|
||
friendly name — undocumented, and established here only by reading
|
||
`wrangler tail`. The caller is pinned by AUD instead, which is both correct and
|
||
stronger; see the AUD section below.
|
||
|
||
The client secret is shown **once**. Copy both halves now.
|
||
|
||
Then add a second Access application covering `free-steam-games.win/api/ingest`,
|
||
with a policy of Action *Service Auth*, rule *Service Token* → `f2p-discovery`.
|
||
|
||
> **The policy Action must be *Service Auth*.** Cloudflare's own docs are blunt
|
||
> about it: "Make sure to set the policy action to Service Auth; otherwise,
|
||
> Access will prompt for an identity provider login." An `Allow` policy — even
|
||
> one carrying a Service Token rule — serves the interactive login page to a
|
||
> machine caller, and additionally admits the interactive identities from your
|
||
> other policies.
|
||
|
||
### Telling the three setup failures apart
|
||
|
||
They all look like "the endpoint is broken". They are not the same problem:
|
||
|
||
| Symptom | Cause | Fix |
|
||
|---|---|---|
|
||
| `404`, no redirect | No Access application covers the path | Create the application |
|
||
| Redirect to the login page, `service_token_status=False` | Application exists, policy Action is not *Service Auth* | Change the policy Action |
|
||
| `access: aud mismatch` in `wrangler tail` | Application exists, but its AUD is not in the var for that route | Add its AUD, redeploy |
|
||
| `access: wrong credential class for route` in `wrangler tail` | A human session hit `/api/ingest/*`, or a service token hit the admin routes | Use the right credential; the two are pinned apart on purpose |
|
||
|
||
`scripts/discover_new.py` decodes the first two from Access's own `meta` JWT and
|
||
prints them; the last two only appear in `wrangler tail`.
|
||
|
||
**The token alone is not enough.** Creating the service token without the
|
||
application does nothing: Access only injects a JWT on paths an application
|
||
claims, so `/api/ingest/*` stays uncovered, the request arrives at the Worker
|
||
bare, and `verifyAccessJwt` returns 404. The symptom is a plain `404` with no
|
||
redirect, and it is indistinguishable from a routing bug until you look at the
|
||
Worker log.
|
||
|
||
### Every new Access application needs its AUD added to the var for its route
|
||
|
||
Easy to miss, and it fails *after* the application starts working - which
|
||
makes it look like a different problem entirely.
|
||
|
||
Each Access application gets its **own** AUD tag, and `web/wrangler.jsonc`
|
||
carries two **comma-separated allowlists** — one per route group:
|
||
|
||
| var | routes it admits |
|
||
|---|---|
|
||
| `ACCESS_AUD_ADMIN` | `/admin`, including its API at `/admin/api/*` |
|
||
| `ACCESS_AUD_INGEST` | `/api/ingest/*` |
|
||
|
||
Add the AUD to the one matching the route the application covers. They are
|
||
split rather than pooled deliberately: with a single list a credential for
|
||
any application satisfied any route, so the unattended discovery token would
|
||
have satisfied `/admin/api/*`, which holds a credential that can write to this
|
||
repository. A token minted for the wrong application now fails verification
|
||
outright rather than being caught afterwards.
|
||
|
||
Add an application without extending the right list and every request through
|
||
it is rejected with `access: aud mismatch`.
|
||
|
||
You do not need the dashboard to read an AUD. An uncovered path 404s; a
|
||
covered one redirects to the login URL with the AUD in its `kid` parameter:
|
||
|
||
```bash
|
||
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://free-steam-games.win/api/ingest/ping
|
||
```
|
||
|
||
A `404` with no redirect means no application covers the path. A `302` whose
|
||
`kid=` value is absent from the matching list means the application exists but
|
||
the Worker will refuse it - add the value and redeploy (both are `vars`, not
|
||
secrets, so a push touching `web/` is enough).
|
||
|
||
From the Worker's own side:
|
||
|
||
```bash
|
||
cd web && npx wrangler tail --format pretty
|
||
```
|
||
|
||
`hasHeader: false, hasCookie: false` means Access never ran - no application
|
||
covers the path. `access: aud mismatch` means it ran and the allowlist for that
|
||
route is stale.
|
||
|
||
Finally, add three repository secrets (**Settings → Secrets and variables →
|
||
Actions**):
|
||
|
||
| Secret | Value |
|
||
|---|---|
|
||
| `WORKER_BASE_URL` | `https://free-steam-games.win` |
|
||
| `CF_ACCESS_CLIENT_ID` | the token's Client ID |
|
||
| `CF_ACCESS_CLIENT_SECRET` | the token's Client Secret |
|
||
|
||
Verify it end to end without touching the queue:
|
||
|
||
```bash
|
||
gh workflow run "Discover New Games" -f dry_run=true
|
||
```
|
||
|
||
### What the token can and cannot do
|
||
|
||
`/api/ingest/*` writes rows with `status='pending'` and nothing else. It cannot
|
||
approve, cannot commit, and cannot reach the repository. A leaked discovery
|
||
token buys an attacker a cluttered review screen — and, while auto-approve is
|
||
on (§11), requests for some of those rows, within its daily limit. A payload's
|
||
claims (free, a game, healthy) are only the caller's word; `ingest_new.py`
|
||
asks Steam itself and keeps only reachable, free apps of type `game`. Set
|
||
**Minimum wait** if you want a day to look before anything is approved. The workflows reinforce this
|
||
with `permissions: contents: read` and a grep step that fails the run if a
|
||
discovery script so much as references a dataset-writing helper.
|
||
|
||
The two credentials are pinned apart in two independent ways, and neither is
|
||
a name check — Cloudflare reports a service token's identity as its opaque
|
||
Client ID, not its friendly name, so a name check is not possible:
|
||
|
||
1. **Per-route AUD.** `verifyAccessJwt` is called with only the AUD group a
|
||
route accepts (`ACCESS_AUD_INGEST` or `ACCESS_AUD_ADMIN`), never a pooled
|
||
list, so a token minted for one application cannot satisfy the other.
|
||
2. **Credential class.** `worker/index.ts` additionally refuses a service
|
||
token on the admin surface and a human session on ingest
|
||
(`isIngestApi !== who.isServiceToken`).
|
||
|
||
An earlier version of this paragraph claimed a `f2p-discovery` name check
|
||
and a pooled AUD list. Both were wrong, and it contradicted the accurate
|
||
account higher up this page.
|
||
|
||
## 8. Working through the backlog
|
||
|
||
Measured 2026-09-10: Steam listed **16,668** free games that support English
|
||
(`category1=998`, `maxprice=free`, `supportedlang=english`); `data/` held
|
||
**3,424**. Since v4.6.0 discovery takes games in every language, with no
|
||
`supportedlang`: **18,790** free games on 2026-10-30, 1,592 of them without
|
||
English ([docs/plan/19-all-languages.md](plan/19-all-languages.md)).
|
||
|
||
Two jobs close that gap, and they are not the same kind of thing:
|
||
|
||
| Workflow | Schedule | Lifetime |
|
||
|---|---|---|
|
||
| `Discover New Games` | daily 05:00 UTC | permanent |
|
||
| `Backfill Discovery (TEMPORARY)` | Sundays 17:30 UTC | **delete when the backlog is done** |
|
||
|
||
For the recent gap — the catalogue's newest record was added 2026-06-30 — run
|
||
the daily job once with a deeper sweep:
|
||
|
||
```bash
|
||
gh workflow run "Discover New Games" -f max_pages=10
|
||
```
|
||
|
||
For older windows, dispatch the backfill a quarter at a time:
|
||
|
||
```bash
|
||
gh workflow run "Backfill Discovery (TEMPORARY)" -f year=2026 -f quarter=1
|
||
```
|
||
|
||
It locates the window by binary search over search offsets rather than walking
|
||
from page 0 (the store sort is `Released_DESC`, so offset *is* the date axis) —
|
||
about 8 page fetches instead of 90 for an old window. A window is finished when
|
||
a re-run reports `candidates: 0`; the scheduled run's window is the
|
||
`SWEEP_YEAR`/`SWEEP_QUARTER` block at the top of the workflow file, which you
|
||
bump as you go.
|
||
|
||
Every window swept before v4.6.0 found only games that list English, so it
|
||
needs one more pass. A whole year at a time works once `max_pages` covers it (a
|
||
recent year is about 30 pages of 100):
|
||
|
||
```bash
|
||
gh workflow run "Backfill Discovery (TEMPORARY)" -f year=2024 -f limit=1000 -f max_pages=60
|
||
```
|
||
|
||
A window is done when its log says `walked past the window`; `stopping: hit
|
||
--max-pages` means it was cut short. Runs dispatched together wait in turn
|
||
(`queue: max`). Each log counts the candidates without English, and a dry run
|
||
lists their languages.
|
||
|
||
When the backlog is closed, delete `scripts/backfill_discover.py`, delete
|
||
`.github/workflows/backfill-discover.yml`, and drop
|
||
`"Backfill Discovery (TEMPORARY)"` from the allowlist in
|
||
`.github/workflows/notify-ci-failure.yml`.
|
||
|
||
---
|
||
|
||
## 9. Reviewing the queue
|
||
|
||
`/admin/queue` is the review screen. (`/admin` itself is the Overview, §13; an
|
||
old bookmark such as `/admin?status=failed` is forwarded to the queue with its
|
||
filters.) The admin is a small Svelte app (`web/admin/`) built
|
||
separately from the public site and **embedded in the Worker**
|
||
(`worker/generated/admin-bundle.ts`), never placed in `dist/`. Everything in
|
||
`dist/` is served to anyone, precached by the service worker and packaged into
|
||
the desktop and Android apps, so admin code there would be public whatever
|
||
Access says. The Worker serves the app's shell and assets only after the Access
|
||
checks, with a per-response nonce CSP, and answers 404 for any other path under
|
||
`/admin`.
|
||
|
||
Tabs across the top are the row's status, with live counts:
|
||
|
||
| Status | Meaning |
|
||
|---|---|
|
||
| `pending` | proposed by the discovery sweep, awaiting a decision |
|
||
| `deferred` | set aside; still blocks the appid from being re-proposed |
|
||
| `approved` | committed to the queue file, waiting for the pipeline |
|
||
| `committed` | observed in `data/`, so genuinely published |
|
||
| `failed` | the pipeline refused it, **or** nothing observed it within 12h and it aged out. Read `reject_reason` |
|
||
| `rejected` | you refused it; never proposed again unless you reopen it |
|
||
|
||
**Read-only rows.** A row you can no longer decide has no checkbox. It shows a
|
||
lock, and *select all* skips it:
|
||
|
||
- `approved`, `committed` and `rejected` rows are already decided;
|
||
- a `pending`, `deferred` or `failed` row whose **game** is already published
|
||
(it has a committed row or an approved decision) is locked too. Approving it
|
||
would queue a link that is already live, and rejecting it would record a
|
||
rejection for a game the catalogue carries.
|
||
|
||
The server enforces the same rules (`web/shared/queue-rules.ts` is the single
|
||
definition both sides import). Anything a decision cannot touch comes back in a
|
||
**skipped** list with a reason, never silently dropped: `published`,
|
||
`not-decidable`, `open-row-exists`, `duplicate-in-request`, `no-op`,
|
||
`changed-concurrently` or `not-found`.
|
||
|
||
Select rows, then choose an action from the bar that appears at the bottom. A
|
||
confirmation lists every row, says what the action will do, and takes an
|
||
optional reason. At most 100 rows go into one decision.
|
||
|
||
Search matches a name or an appid in the current tab; tick *Search every
|
||
status* to look everywhere at once. The tab, search, sort, page and page size
|
||
are all in the URL, so a reload or a shared link lands on the same view.
|
||
|
||
Keyboard: `j`/`k` move, `x` selects, `a` approves, `r` rejects, `d` defers (or
|
||
sends back to pending), `e` opens the row, `/` searches, `Esc` clears the
|
||
selection, `?` lists them. Keys are ignored while you type in a field or while
|
||
a dialog is open, and never interrupt IME composition.
|
||
|
||
Click a game (or press `e`) for its details: the payload as submitted, the
|
||
durable decision for the game, and every other row the queue holds for the same
|
||
appid. **Languages** lists what the game supports, as Steam names them, with a
|
||
**No English** badge when English is not among them; such a game is decided
|
||
like any other, and auto-approve treats it the same. Rows queued before v4.6.0
|
||
show "—": discovery did not record languages then. A `rejected` row has **Reopen** there. It moves that one row back to
|
||
`pending` and deletes the rejection, but only if no other row for the game is
|
||
open or committed and the game has no approved decision; otherwise the dialog
|
||
says which of those stopped it. There is no bulk reopen, on purpose. A game
|
||
that was *removed from the catalogue* (§12) cannot be reopened at all — its
|
||
rejection says `removed:` — and the dialog sends you to **Restore** instead,
|
||
which is the only way to undo a removal.
|
||
|
||
### The three actions are not symmetric
|
||
|
||
**Reject** is final and purely local. It writes `ingest_decisions('rejected')`,
|
||
which is what stops tomorrow's sweep offering the game again — the partial
|
||
unique index only covers *open* rows, so that table is what makes a rejection
|
||
stick. The reason box is kept with the decision.
|
||
|
||
**Defer** just parks the row. It stays open, so the appid remains blocked from
|
||
re-proposal and nothing durable is recorded.
|
||
|
||
**Approve does not publish.** It appends one `{"link": ...}` line per game to
|
||
`scripts/temp_info.jsonl` and stops there. What happens next is out of the
|
||
Worker's hands: `scripts/ingest_new.py` re-checks each game and can still
|
||
refuse it as a duplicate, delisted, not actually free, or unreachable.
|
||
|
||
That is why an approved row goes to `approved`, not `committed`, and why
|
||
nothing writes `ingest_decisions('approved')` at that moment. A commit is a
|
||
*request*, not an outcome. Recording it as decided would make a failed ingest
|
||
permanently invisible to every future sweep, because `/api/ingest/known`
|
||
reports decided appids to the discovery pipeline — the game would vanish with
|
||
no error anywhere.
|
||
|
||
The commit is made with the GraphQL `createCommitOnBranch` mutation, so GitHub
|
||
signs it and it shows as **Verified**. It carries `expectedHeadOid`, so if the
|
||
pipeline commits between the read and the write the mutation is rejected and
|
||
retried against the newer file rather than overwriting it. The file is
|
||
**appended** to, never replaced — Restore and pull requests add links to it
|
||
too, and the ingest clears it.
|
||
|
||
Because the commit comes from a GitHub App installation token rather than
|
||
`GITHUB_TOKEN`, it **does** trigger workflows, so `Ingest New Game Links` starts
|
||
on its own. (It also runs every three hours as orphan recovery.)
|
||
|
||
### How a row reaches `committed`
|
||
|
||
A Worker cron runs every 15 minutes and promotes `approved` rows it can observe
|
||
in the published dataset, writing `ingest_decisions('approved')` only then. The
|
||
**Reconcile** button runs the same pass immediately, for when you have just
|
||
watched the pipeline finish.
|
||
|
||
A row the pipeline refused becomes `failed` rather than rejected — deliberately,
|
||
so one bad day (a delisting, a Steam outage) does not permanently hide a game.
|
||
A `failed` row can be approved again, deferred, or sent **Back to pending**.
|
||
|
||
The same pass also **sweeps** undecided rows. A `pending`, `deferred` or
|
||
`failed` row whose game has appeared in `data/` by some other route (a link a
|
||
pull request queued, a manual edit) is marked `committed` by `reconcile`, with an
|
||
approved decision, so it stops asking for a review it no longer needs. The
|
||
sweep reads the dataset only when something is approved, when you press
|
||
Reconcile, or when `data/index.json` has changed since the last sweep.
|
||
|
||
`reject_reason` says which of three things happened:
|
||
|
||
| `reject_reason` | meaning |
|
||
|---|---|
|
||
| `rejected by ingest pipeline` | it reached `removed_games.jsonl` after you approved it — not free, delisted, or unreachable |
|
||
| `never appeared in data/ or removed_games.jsonl - approve again to retry` | 12h passed and nothing observed it. `ingest_new.py` silently skips an entry that hits a network error and then clears the queue file regardless, so the request is simply gone. **Approve it again** — that re-commits the link |
|
||
| `superseded by a duplicate approval` | another row for the same appid was approved instead |
|
||
|
||
That middle case is why the age-out exists at all. Without it the row would sit
|
||
in `approved` forever, and because `/api/ingest/known` counts `approved` as
|
||
known, the discovery sweep would never offer that appid again — the game would
|
||
be lost silently and permanently.
|
||
|
||
A tick with nothing approved costs one indexed D1 query and no fetches, so the
|
||
schedule is close to free.
|
||
|
||
### If something goes wrong
|
||
|
||
Every action writes to `audit_log`, and every commit writes a `commit_jobs`
|
||
row *before* it is attempted, so a Worker that dies mid-flight leaves a
|
||
`pending` job as evidence rather than silence. Both have their own pages,
|
||
filterable and paged: **Jobs** (`/admin/jobs`) and **Audit** (`/admin/audit`).
|
||
The queue header warns when any job is `failed` or stuck in `pending`.
|
||
|
||
**Health** (`/admin/health`) checks D1, the GitHub App installation, the applied
|
||
migrations and the reconcile lock live, and still draws its table when the
|
||
answer is a 503. **Test write** records a `ping` audit row, which proves the
|
||
whole write path from your browser through Access to D1.
|
||
|
||
If a commit fails the rows are left untouched, on purpose: a row marked
|
||
approved with no commit behind it is invisible to both the queue and the
|
||
reconciler. Re-approving is safe, because already-queued links are skipped.
|
||
|
||
If the commit lands but D1 then fails to update the rows, the error says
|
||
exactly that, names the commit, and records an `approve.desynced` audit row.
|
||
Approving again is safe, and once the pipeline publishes the games the sweep
|
||
marks the rows committed on its own.
|
||
|
||
When the Access session lapses, the page reloads once so Access can sign you in
|
||
again. If the API still refuses straight after that reload, the page says so
|
||
instead of reloading in a loop.
|
||
|
||
The same data from the command line:
|
||
|
||
```bash
|
||
npx wrangler d1 execute f2p-admin --remote --command "SELECT status, target_path, commit_sha, error, requested_by, created_at FROM commit_jobs ORDER BY created_at DESC LIMIT 10"
|
||
npx wrangler d1 execute f2p-admin --remote --command "SELECT appid, name, release_date, status FROM ingest_queue ORDER BY first_seen_at DESC LIMIT 40"
|
||
```
|
||
|
||
---
|
||
|
||
## 10. Correcting an existing game
|
||
|
||
The review queue covers *new* games. Corrections to games already published —
|
||
a wrong genre, a note to add, anti-cheat details — go through **overrides**.
|
||
|
||
An override is not a one-time edit. It is a standing instruction stored at
|
||
`data/overrides/<appid>.json`, and `save_main()` re-imposes it on every write
|
||
to `data/`. That matters because `MANUAL_FIELDS` are otherwise protected only
|
||
by fill-if-empty, which cannot tell a human correction from scraper output —
|
||
`normalize_genres.py --apply` rewrites `genre` for every game and would revert
|
||
your fix with no error anywhere.
|
||
|
||
```bash
|
||
# what does the catalogue say right now?
|
||
python scripts/edit_game.py 730 --show
|
||
|
||
# correct it, with a reason (kept in the file and visible in git log)
|
||
python scripts/edit_game.py 730 --set genre="Tactical Shooter" \
|
||
--reason "Steam's genre list is too coarse" --by you@example.com
|
||
|
||
# undo: restores the pre-edit value, then permanently stops acting
|
||
python scripts/edit_game.py 730 --retire genre
|
||
|
||
python scripts/edit_game.py --list # every override
|
||
python scripts/edit_game.py --check # validate them (this runs in CI)
|
||
```
|
||
|
||
The command writes only the override file. `data/` changes on the next
|
||
pipeline run, or immediately with `--apply`.
|
||
|
||
### Three things that will surprise you
|
||
|
||
**You cannot blank a field with an override.** Setting it to `""` is refused.
|
||
The next scrape would refill it and this layer would blank it again, rewriting
|
||
shards on every run forever. Use `--retire` to clear.
|
||
|
||
**Retiring is not the same as deleting the file.** Delete it and the field
|
||
stays pinned to your value, because fill-if-empty never brings the old one
|
||
back. `--retire` keeps the pre-edit value and writes it back once.
|
||
|
||
**Editing `notes` keeps the pipeline's own markers.** `notes` is shared: the
|
||
pipeline appends `Dead game`, `Delisted` and `No longer free!` segments to it,
|
||
and those appends are one-way — once a game is flagged dead the marker is never
|
||
re-added. Your text is combined with them rather than replacing them.
|
||
|
||
### The same thing from `/admin/edit`
|
||
|
||
`/admin/edit` does exactly what the command above does, and writes a
|
||
byte-identical file. A formatting difference between the two producers would
|
||
turn every alternating edit into a whole-file diff, so the equality is asserted
|
||
by `web/worker/lib/override-doc.test.ts`, which round-trips every committed
|
||
`data/overrides/*.json` — all Python-written — through the TypeScript
|
||
serialiser and compares bytes. It runs in `web-ci.yml` and in
|
||
`check-overrides.yml`.
|
||
|
||
**One game.** Load a game by appid, or paste its Steam URL. Each field shows
|
||
what the catalogue holds now and, where an override is active, its value, the
|
||
value from before the first edit, who set it and why. Change what you want, add
|
||
a reason, then **Review and save**: a dialog lists every change against the
|
||
catalogue and the current override before anything is committed. An overridden
|
||
field has **Retire**, which restores the pre-edit value once on the next
|
||
pipeline run. Leaving the page with unsaved changes asks first.
|
||
|
||
**Several games.** Pick games by genre, up to 10 at a time; the selection is
|
||
kept across pages and genres. Choose any one field and either set a value or
|
||
retire its override, then review: the dialog loads every selected game and
|
||
shows what the change does to each, including the ones it leaves unchanged.
|
||
One commit, one override file per game.
|
||
|
||
**Deleting an override file.** Once every entry in a file is retired *and* the
|
||
pipeline has restored each value, the file no longer changes anything, and the
|
||
page offers **Delete override file**. Until then it says what is still active
|
||
or not yet restored, and the Worker checks again against the file as it is at
|
||
commit time. (`scripts/edit_game.py` removes a file only when it holds no
|
||
entries at all, active or retired, so a file with retired entries stays until
|
||
you delete it here.)
|
||
|
||
The Worker writes only `data/overrides/<appid>.json`; it cannot touch `data/`,
|
||
and the page says so rather than letting you assume the catalogue changed.
|
||
|
||
Two safeguards worth knowing:
|
||
|
||
- The Worker's validation is a deliberately weaker MIRROR of
|
||
`validate_value()` in `scripts/core/overrides.py`, and the page's own checks
|
||
mirror the Worker's (`worker/routes/edit.test.ts` holds those two equal). They
|
||
exist so you are told immediately. The Python one is authoritative and runs on
|
||
every pipeline write, and `check-overrides.yml` runs it on every push — so
|
||
anything the UI accepts but Python would reject fails CI within a minute
|
||
instead of sitting inert. Change one, change the others.
|
||
- `was` is read server-side from the catalogue, never taken from the request.
|
||
It is what Retire restores, so a client-supplied value would let a crafted
|
||
request rewrite history.
|
||
|
||
---
|
||
|
||
## 11. Approving automatically
|
||
|
||
`/admin/automation` lets the Worker's cron approve the queue for you, one batch
|
||
at a time. A batch is exactly what **Approve** does by hand (§9): one commit
|
||
appending `{"link": ...}` lines to `scripts/temp_info.jsonl`, the rows go to
|
||
`approved`, and reconcile promotes them once the games appear in `data/`. It
|
||
never publishes anything itself: `ingest_new.py` still checks every game —
|
||
reachable, free, of Steam type `game`, not already listed — and refuses the
|
||
rest into `removed_games.jsonl`.
|
||
|
||
**It is off until you turn it on.**
|
||
|
||
| Setting | Default | Range |
|
||
|---|---|---|
|
||
| Run on the schedule | off | every 15 minutes when on |
|
||
| Games per batch | 100 | 1–100 |
|
||
| Games per day (UTC) | 300 | 1–**500**; the 500 is in code and re-applied to whatever is stored |
|
||
| Minimum wait | 0 h | 0–168 h a row must have waited — time to reject something by hand first |
|
||
| Order | oldest first | oldest, newest, most players |
|
||
| Sources | every source | tick some to restrict; `backfill:*` matches by prefix |
|
||
|
||
A row is eligible when it is `pending`, free, Steam type `game`, healthy, has
|
||
waited long enough, comes from a ticked source, its game has **no decision at
|
||
all** (never rejected, never removed) and is not already committed, and it is
|
||
not in `data/blocklist.jsonl`. **Next batch** on the page is the same query the
|
||
run uses, so what it lists is what would be approved.
|
||
|
||
A batch is committed only when **all** of these hold, and a check it cannot make
|
||
counts as *no* — the page's *Last run* line says which one stopped it:
|
||
|
||
- **The previous batch has been ingested**: `scripts/temp_info.jsonl` on `main`
|
||
is empty. Checked again inside the commit itself, so a batch never lands in
|
||
the middle of an ingest.
|
||
- **No data job is running or waiting.** The `data-write` group queues its
|
||
runs (`queue: max`), so the ingest a batch starts would only wait behind such
|
||
a job; the gate makes a batch land when its ingest can start at once. The
|
||
list of those workflows is `web/shared/pipeline.ts`, which a test holds to
|
||
the YAML.
|
||
- **Today's budget is not spent.** It is reserved before the commit and given
|
||
back if the commit fails.
|
||
|
||
So the throughput is one batch per ingest: the next batch waits until the
|
||
ingest the last one started has emptied the queue file.
|
||
|
||
**Run one batch now** ignores the on/off switch and obeys everything else; D1
|
||
records it under your identity, a scheduled batch under `auto-approve`. The
|
||
commit (`queue: N auto-approved games`) names no one. A failure is audited once
|
||
per distinct cause (`auto.approve.failed`), not every 15 minutes; a skipped tick
|
||
is not audited at all.
|
||
|
||
Start at around 100 a day for a cycle and look at what landed (Overview,
|
||
**Pipeline**, `removed_games.jsonl`) before raising it.
|
||
|
||
## 12. Removing and restoring games
|
||
|
||
`/admin/remove` takes games **out** of the catalogue, up to 100 at a time.
|
||
|
||
A removal is not a deletion the Worker performs — it cannot write a shard. It
|
||
commits a line per game to `data/blocklist.jsonl`, a standing instruction like
|
||
an override: `save_main()` leaves every listed game out of **every** write. The
|
||
commit starts **Ingest New Game Links**, whose first step
|
||
(`scripts/remove_games.py`) takes the games out of `data/` and logs each one
|
||
once in `scripts/removed_games.jsonl` (`status_code: admin_removed`, with the
|
||
reason's label). From then on no scrape, refetch or queued link can bring the
|
||
game back, and the discovery sweep never proposes it again.
|
||
|
||
**Remove.** Search by name or appid, or paste appids and store links (separated
|
||
by spaces, commas or new lines; up to 100). Each one is previewed; a game that is not in the catalogue, or
|
||
is already removed, is skipped with that reason. Pick a reason from the list
|
||
and add a note if it helps (up to 300 characters; required for *Other*). **Both
|
||
are public**: they go into the blocklist and the commit message, in a public
|
||
repository. The confirmation lists every game before anything is committed.
|
||
|
||
On the public site the game leaves the catalogue with the next data
|
||
generation. It is counted with the removed games on the dashboard, appears under
|
||
*Other* on `/charts/delisted`, and its page says it was removed and why. A
|
||
prerendered page for it stays until the next deploy, like a purged game's.
|
||
|
||
The review queue is told the same thing: the game's decision becomes `rejected`
|
||
with reason `removed:<code>` (even if it was approved), and every row for it
|
||
that was not rejected is. Removing a game that is already in the blocklist
|
||
commits nothing and only re-applies that, which is how to repair the queue if
|
||
the commit landed but D1 did not (`remove.desynced` in the audit log).
|
||
|
||
**Removed** (`/admin/remove?tab=removed`) lists every blocklisted game: when,
|
||
why, and whether the pipeline has applied it yet. **Restore** is one game at a
|
||
time and one commit: the line leaves the blocklist and the store link is queued
|
||
for `ingest_new.py` in the same commit, so the ingest cannot run in between and
|
||
find the game still blocked. Ingest fetches it fresh from Steam, so it can still
|
||
refuse it (no longer free, say), and the row then shows `failed`. Corrections
|
||
you made as overrides come back with it — the override files stay — but
|
||
anything changed by other means is gone. In D1, Restore deletes only the
|
||
`removed:` decision and enters an approved row (source `restore`) for reconcile
|
||
to promote.
|
||
|
||
Remove and Restore share a lease, so two of them never interleave.
|
||
|
||
From the command line (dry run unless `--apply`):
|
||
|
||
```bash
|
||
python scripts/delete_game.py 730 --reason duplicate --note "same game as 2357570"
|
||
python scripts/delete_game.py 730 --reason duplicate --note "same game as 2357570" --apply
|
||
python scripts/delete_game.py --list
|
||
python scripts/delete_game.py --check # validates the file; runs in CI
|
||
```
|
||
|
||
It writes the same bytes `/admin` does, and refuses a game that is not in
|
||
`data/`. It cannot restore: that has to clear the review queue's record too, so
|
||
use `/admin`. Deleting a line from the file by hand unblocks the game but leaves
|
||
D1's `removed:` decision, so discovery would go on skipping it.
|
||
|
||
Two safeguards: a save that would remove more than **500** games stops the job
|
||
before anything is written (`remove_games.py --max-drop N` is the deliberate
|
||
override), and a line the pipeline cannot honour blocks nothing — `--check`
|
||
fails CI on it.
|
||
|
||
## 13. Overview, Pipeline and Data quality
|
||
|
||
**Overview** (`/admin`) is the first page: the queue's counts and the backlog
|
||
by source, auto-approve's day and last run, how many removals are waiting for
|
||
the pipeline, data jobs running now, the data-quality totals, and any commit job
|
||
that failed or is stuck. Each card loads on its own, so one failing endpoint
|
||
blanks one card, not the page.
|
||
|
||
**Pipeline** (`/admin/pipeline`) shows the latest runs of every data workflow,
|
||
what is running or waiting for its turn, and how many lines are waiting in
|
||
`scripts/temp_info.jsonl` and `data/blocklist.jsonl`. It reads GitHub with a
|
||
one-minute cache; **Refresh** asks again.
|
||
|
||
**Data quality** (`/admin/quality`) counts published games that fail a check:
|
||
|
||
| Group | Checks | Fixed by |
|
||
|---|---|---|
|
||
| Needs a judgement | safety never set, safety `?` (the review backlog), kernel anti-cheat unknown | you, on the Edit page (an override) |
|
||
| Manual field left empty | online game with no anti-cheat, no genre, online/offline not set | you, on the Edit page |
|
||
| Missing store data | header, description, tags, languages, developer, publisher, release date | Refresh Store Data, which retries these on its own |
|
||
| For reference | dead, store page gone | the pipeline |
|
||
|
||
Click a count for the games, and a game for its Edit page.
|
||
|
||
---
|
||
|
||
## Working on the admin locally
|
||
|
||
A real Worker cannot show the admin locally: every `/admin` request, the
|
||
page's and its API's, needs an Access JWT, which cannot be minted offline, so
|
||
`wrangler dev` answers 404. The admin has its own dev server instead. It runs the **real**
|
||
handlers (`worker/routes/admin.ts` and every route it delegates to) against an
|
||
in-memory D1 with the real migrations, fed by the real `data/` shards, with
|
||
GitHub faked. Nothing
|
||
leaves the machine and nothing is written to disk.
|
||
|
||
```bash
|
||
cd web
|
||
npm run dev:admin # http://localhost:5174/admin, hot reload, demo data
|
||
npm run preview:admin # the built bundle through the real serveAdmin(), production CSP
|
||
```
|
||
|
||
The demo data (`web/admin/mock/seed.ts`) has every status, rows locked because
|
||
their game is published, a rejected row that cannot be reopened, commit jobs in
|
||
every state, audit history, an override on 730 to edit, auto-approve settings
|
||
with a batch already run today, two removed games (one the pipeline has applied,
|
||
one still waiting for it) and a set of workflow runs. Three switches, as cookies
|
||
set from the browser console:
|
||
|
||
| Cookie | Effect |
|
||
|---|---|
|
||
| `mock_github=down` | health reports GitHub unreachable, a 503 (or start the server with `MOCK_GITHUB=down`) |
|
||
| `mock_session=expired` | the API answers the way Access does for a lapsed session |
|
||
| `mock_pipeline=busy` | a data job is running, so auto-approve waits and the Pipeline page shows it |
|
||
|
||
None of this exists in the Worker. Nothing under `worker/` or `admin/src`
|
||
imports the mock (`admin/security.test.ts` checks), and there is no flag that
|
||
could switch authentication off.
|
||
|
||
`npm run build` builds the admin after the site, and `npm run typecheck` builds
|
||
it first, because the Worker imports the generated bundle.
|
||
|
||
## Known limits
|
||
|
||
- **A rejection stays until you reopen it.** There is no `suppress_until`, so a
|
||
game rejected for a temporary reason is not offered again on its own.
|
||
- **Restore is one game at a time**, and a removal reaches `data/` only when
|
||
the pipeline next runs (the commit starts it; the Removed tab says which games
|
||
are still waiting).
|
||
- **Auto-approve never rejects.** What it does not take stays `pending` for you.
|
||
- **One field per bulk change.** Several fields on several games means several
|
||
commits.
|
||
- **English only.** The admin has one audience, so it has no translations.
|