feat(admin): read-only decided and published rows, reopen, reconcile sweep, retention
The server half of the admin rework. The pages still run on it unchanged - every change here is additive - until the new UI replaces them. Read-only rows. web/shared/queue-rules.ts holds the queue's rules once, for the Worker and the UI: a row is locked when it is approved (its request is already committed to Git), committed (the game was observed in data/) or rejected, and also when its GAME is already published, since one appid can have several rows. /api/admin/decide refuses locked rows when it reads them and again in every UPDATE's WHERE, and reports each refused id with a reason - not-found, not-decidable, published, open-row-exists, duplicate-in-request, no-op, changed-concurrently. It used to drop them silently, so a mixed selection reported fewer decisions than it was sent. The queue listing carries `published` per row, and status=all searches every tab when there is a query. Bugs fixed on the way: - Requeueing or deferring a failed row while another row for the appid was open broke uq_queue_open_appid inside the batch: a 500 for the whole request. It is now a skip. - When an approval's commit landed and the queue update failed, the job's status was in the same rolled-back batch and stayed 'pending' forever, and the message promised reconcile would promote rows it never reads. The job is recorded first, and the message says what is actually true. - The health endpoint returned 503 without a body the page could render. New: POST /api/admin/reopen returns one rejected row to pending and deletes only its 'rejected' decision, in one transaction whose guards refuse a published game or an appid with another open row - replacing a raw-SQL step in the runbook. Jobs and audit page with totals and filters. Approval reasons are recorded in the audit row and the commit body. Reconcile sweeps pending, deferred and failed rows whose game is already in data/ to committed (a game published after its approval aged out stayed 'failed' forever), holds a cross-isolate D1 lease, and skips the ~6 MB dataset fetch while index.json's generation is unchanged. Migration 0002 adds the two tables; lib/locks.ts treats a missing table as unavailable, so the Worker and the migration can ship in either order. Overrides: the Worker can now delete a SETTLED override file - no active field, every retirement already restored - through createCommitOnBranch deletions, and every override commit writes a commit_jobs row. audit_log and commit_jobs are pruned daily after ADMIN_RETENTION_DAYS (180), which the privacy policy now states. Tests: the real handlers against the real migrations on node:sqlite (worker/testing/sqlite-d1.ts), with GitHub and the dataset faked - 57 new cases across decide, reopen, reconcile, leases, retention, override deletion and the createCommitOnBranch payload. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
26 files changed
+2171
-317
No files matched your search
@@ -100,6 +100,28 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
- New games ARE discovered automatically now, by `scripts/discover_new.py`
|
||||
into the D1 review queue — but discovery only PROPOSES. Publication stays a
|
||||
human decision made in `/admin`.
|
||||
- **Queue rules live in `web/shared/queue-rules.ts`, imported by the Worker AND
|
||||
the admin UI.** A row is read-only when its status is not decidable
|
||||
(`approved` = the request is committed to Git, `committed` = the game was
|
||||
observed in `data/`, `rejected`) or when its GAME is already published (a
|
||||
committed row or an approved decision for the appid — per game, because one
|
||||
appid can have several rows). `/api/admin/decide` refuses such rows in its
|
||||
read AND in every UPDATE's WHERE, and returns each refused id in `skipped`
|
||||
with a reason; it never drops one silently. `queue-rules.test.ts` parses
|
||||
`0001_init.sql` so the constants cannot drift from the CHECK constraint or
|
||||
`uq_queue_open_appid`.
|
||||
- **`reconcile.ts` is still the only writer of `ingest_decisions('approved')`,**
|
||||
and it now also sweeps: a pending/deferred/failed row whose appid is in
|
||||
`data/` becomes `committed`, replacing a stale `rejected` decision. It holds a
|
||||
D1 lease (`admin_locks`) and skips the ~6 MB fetch while `data/index.json`'s
|
||||
generation is unchanged (`admin_state`). Migration `0002_admin_state.sql` is
|
||||
applied by hand, so `lib/locks.ts` treats a missing table as "unavailable" and
|
||||
every caller falls back — deploy order must never matter. Keep it that way for
|
||||
any future migration a Worker change depends on.
|
||||
- `audit_log` and `commit_jobs` are pruned daily after `ADMIN_RETENTION_DAYS`
|
||||
(wrangler.jsonc, 180). `docs/PRIVACY_POLICY.md` states the number: change
|
||||
both together. `ingest_queue` and `ingest_decisions` are never pruned — the
|
||||
decisions are what keep a rejected game out of the discovery sweep.
|
||||
- Every workflow that writes to `data/` shares `concurrency: {group: data-write}`.
|
||||
A new data-writing workflow without it will race jobs that run ~2 hours.
|
||||
- Workflows push with the built-in `GITHUB_TOKEN`; each declares
|
||||
|
||||
@@ -108,6 +108,9 @@ Cloudflare D1 database — what was changed, when, and by which Access identity.
|
||||
That log exists so a bad change can be traced. It records no visitor data,
|
||||
because visitors never reach it.
|
||||
|
||||
Entries in that log, and the record of repository commits made from the admin
|
||||
area, are deleted automatically after 180 days.
|
||||
|
||||
## What is not collected
|
||||
|
||||
No accounts. No analytics or telemetry of any kind. No advertising or
|
||||
|
||||
+2
-1
@@ -15,7 +15,8 @@
|
||||
"project": [
|
||||
"src/**/*.{ts,svelte}",
|
||||
"src/**/*.css",
|
||||
"worker/**/*.ts"
|
||||
"worker/**/*.ts",
|
||||
"shared/**/*.ts"
|
||||
],
|
||||
// Ambient declaration for the Worker secrets, which wrangler types cannot
|
||||
// see because secrets are not in wrangler.jsonc. Nothing imports it by
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
allowedActions,
|
||||
DECIDABLE,
|
||||
isSelectable,
|
||||
lockReason,
|
||||
OPEN_STATUSES,
|
||||
QUEUE_STATUSES,
|
||||
sqlList,
|
||||
} from "./queue-rules";
|
||||
|
||||
describe("lockReason", () => {
|
||||
it("locks every decided status, whatever else is true", () => {
|
||||
for (const status of ["approved", "committed", "rejected"]) {
|
||||
expect(lockReason({ status })).toBe("decided");
|
||||
expect(lockReason({ status, published: true })).toBe("decided");
|
||||
expect(isSelectable({ status })).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("locks an undecided row whose game is already published", () => {
|
||||
for (const status of DECIDABLE) {
|
||||
expect(lockReason({ status, published: true })).toBe("published");
|
||||
expect(lockReason({ status, published: 1 })).toBe("published");
|
||||
expect(isSelectable({ status, published: true })).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("leaves an undecided, unpublished row selectable", () => {
|
||||
for (const status of DECIDABLE) {
|
||||
expect(lockReason({ status, published: false })).toBeNull();
|
||||
expect(lockReason({ status, published: 0 })).toBeNull();
|
||||
expect(isSelectable({ status })).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("allowedActions", () => {
|
||||
it("offers no action on a locked row", () => {
|
||||
expect(allowedActions({ status: "committed" })).toEqual([]);
|
||||
expect(allowedActions({ status: "pending", published: true })).toEqual([]);
|
||||
});
|
||||
|
||||
it("never offers a move to where the row already is", () => {
|
||||
expect(allowedActions({ status: "pending" })).toEqual(["approve", "reject", "defer"]);
|
||||
expect(allowedActions({ status: "deferred" })).toEqual(["approve", "reject", "requeue"]);
|
||||
expect(allowedActions({ status: "failed" })).toEqual(["approve", "reject", "defer", "requeue"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("sqlList", () => {
|
||||
it("quotes closed word lists", () => {
|
||||
expect(sqlList(DECIDABLE)).toBe("'pending','deferred','failed'");
|
||||
});
|
||||
|
||||
it("refuses anything that is not a lowercase word", () => {
|
||||
expect(() => sqlList(["pending'); DROP TABLE x; --"])).toThrow();
|
||||
expect(() => sqlList(["Pending"])).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe("the constants match the schema", () => {
|
||||
const sql = readFileSync(join(process.cwd(), "worker", "migrations", "0001_init.sql"), "utf-8");
|
||||
const words = (list: string) => [...list.matchAll(/'([a-z]+)'/g)].map((m) => m[1]);
|
||||
|
||||
it("QUEUE_STATUSES is the status CHECK constraint", () => {
|
||||
const check = /status\s+TEXT[\s\S]*?CHECK \(status IN \(([^)]+)\)\)/.exec(sql);
|
||||
expect(check).not.toBeNull();
|
||||
expect(words(check![1])).toEqual([...QUEUE_STATUSES]);
|
||||
});
|
||||
|
||||
it("OPEN_STATUSES is the uq_queue_open_appid predicate", () => {
|
||||
const index = /uq_queue_open_appid[\s\S]*?WHERE status IN \(([^)]+)\)/.exec(sql);
|
||||
expect(index).not.toBeNull();
|
||||
expect(words(index![1])).toEqual([...OPEN_STATUSES]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,110 @@
|
||||
/**
|
||||
* The review queue's rules, in ONE place, imported by both the Worker
|
||||
* (worker/routes/admin.ts) and the admin SPA (admin/src).
|
||||
*
|
||||
* The old admin page carried its own copy of DECIDABLE as an inline-script
|
||||
* literal, and a test existed only to keep the two strings equal. A shared
|
||||
* module removes the second copy instead of guarding it.
|
||||
*
|
||||
* Pure: no DOM, no Env, no I/O. queue-rules.test.ts also parses
|
||||
* worker/migrations/0001_init.sql and fails if the statuses or the open-row
|
||||
* index drift from the constants here.
|
||||
*/
|
||||
|
||||
/** Every value ingest_queue.status may hold (the CHECK constraint). */
|
||||
export const QUEUE_STATUSES = ["pending", "deferred", "approved", "committed", "rejected", "failed"] as const;
|
||||
export type QueueStatus = (typeof QUEUE_STATUSES)[number];
|
||||
|
||||
/**
|
||||
* Statuses a row may be decided FROM.
|
||||
*
|
||||
* approved the approval is already committed to Git as a request
|
||||
* committed the game was observed in data/ - it is published
|
||||
* rejected final; reopened explicitly, one row at a time, never in bulk
|
||||
*/
|
||||
export const DECIDABLE = ["pending", "deferred", "failed"] as const satisfies readonly QueueStatus[];
|
||||
|
||||
/** Statuses covered by uq_queue_open_appid: at most one such row per appid. */
|
||||
export const OPEN_STATUSES = ["pending", "deferred", "approved"] as const satisfies readonly QueueStatus[];
|
||||
|
||||
/** Rows one decision may touch: bounds the D1 batch, the commit, and a mis-click. */
|
||||
export const MAX_DECIDE = 100;
|
||||
|
||||
/** Games one override commit may change. */
|
||||
export const MAX_EDIT = 10;
|
||||
|
||||
export const DECIDE_ACTIONS = ["approve", "reject", "defer", "requeue"] as const;
|
||||
export type DecideAction = (typeof DECIDE_ACTIONS)[number];
|
||||
|
||||
/** Why a row named in a decision was not decided. Reported back, never silent. */
|
||||
export type SkipReason =
|
||||
/** No row has that id. */
|
||||
| "not-found"
|
||||
/** approved, committed or rejected. */
|
||||
| "not-decidable"
|
||||
/** The game is already in the catalogue (a committed row, or an approved decision). */
|
||||
| "published"
|
||||
/** Another open row exists for the appid, and this move would collide with it. */
|
||||
| "open-row-exists"
|
||||
/** The request named two rows for one appid; the newer one was used. */
|
||||
| "duplicate-in-request"
|
||||
/** Deferring a deferred row, or requeueing a pending one. */
|
||||
| "no-op"
|
||||
/** Decidable when read, changed by someone else before the write. */
|
||||
| "changed-concurrently";
|
||||
|
||||
export interface SkippedRow {
|
||||
id: string;
|
||||
appid: string | null;
|
||||
reason: SkipReason;
|
||||
}
|
||||
|
||||
export type LockReason = "decided" | "published";
|
||||
|
||||
export function isDecidable(status: string): boolean {
|
||||
return (DECIDABLE as readonly string[]).includes(status);
|
||||
}
|
||||
|
||||
/**
|
||||
* Why a row is read-only, or null when a reviewer may act on it.
|
||||
*
|
||||
* `published` is per GAME, not per row: one appid can have several rows, and a
|
||||
* pending or failed row for a game that is already live must not be approved
|
||||
* again (it would re-queue a published link) or rejected (it would record a
|
||||
* rejection for a game the catalogue carries).
|
||||
*/
|
||||
export function lockReason(row: { status: string; published?: boolean | number | null }): LockReason | null {
|
||||
if (!isDecidable(row.status)) return "decided";
|
||||
if (row.published) return "published";
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Whether the row gets a selection checkbox. */
|
||||
export function isSelectable(row: { status: string; published?: boolean | number | null }): boolean {
|
||||
return lockReason(row) === null;
|
||||
}
|
||||
|
||||
/** A move that would leave the row where it is. */
|
||||
export function isNoOp(action: DecideAction, status: string): boolean {
|
||||
return (action === "defer" && status === "deferred") || (action === "requeue" && status === "pending");
|
||||
}
|
||||
|
||||
/** The actions that make sense for one selectable row. */
|
||||
export function allowedActions(row: { status: string; published?: boolean | number | null }): DecideAction[] {
|
||||
if (lockReason(row)) return [];
|
||||
return DECIDE_ACTIONS.filter((a) => !isNoOp(a, row.status));
|
||||
}
|
||||
|
||||
/**
|
||||
* A closed tuple as SQL string literals: `'pending','deferred'`.
|
||||
*
|
||||
* Only ever for the constants above - values that are fixed at build time and
|
||||
* lowercase words - so a status list can appear in a WHERE clause without
|
||||
* binding one parameter per status. Anything else throws.
|
||||
*/
|
||||
export function sqlList(values: readonly string[]): string {
|
||||
for (const v of values) {
|
||||
if (!/^[a-z]+$/.test(v)) throw new Error(`sqlList: refusing ${JSON.stringify(v)}`);
|
||||
}
|
||||
return values.map((v) => `'${v}'`).join(",");
|
||||
}
|
||||
@@ -18,5 +18,5 @@
|
||||
"noFallthroughCasesInSwitch": true,
|
||||
"skipLibCheck": true
|
||||
},
|
||||
"include": ["worker/**/*.test.ts", "worker", "worker-configuration.d.ts"]
|
||||
"include": ["worker/**/*.test.ts", "worker", "shared", "worker-configuration.d.ts"]
|
||||
}
|
||||
@@ -21,10 +21,10 @@
|
||||
"noFallthroughCasesInSwitch": true,
|
||||
"skipLibCheck": true
|
||||
},
|
||||
"include": ["worker", "worker-configuration.d.ts"],
|
||||
"include": ["worker", "shared", "worker-configuration.d.ts"],
|
||||
// Tests run on Node under Vitest, not on workerd. Including them here
|
||||
// would either fail (no Node types) or, worse, pass by adding Node types
|
||||
// to the Worker's surface - which is exactly the confusion the split
|
||||
// above exists to prevent. They are typechecked by tsconfig.test.json.
|
||||
"exclude": ["worker/**/*.test.ts"]
|
||||
"exclude": ["worker/**/*.test.ts", "shared/**/*.test.ts", "worker/testing/**"]
|
||||
}
|
||||
Vendored
+3
-2
@@ -1,11 +1,12 @@
|
||||
/* eslint-disable */
|
||||
// Generated by Wrangler by running `wrangler types` (hash: ab5f02820b26f4634437c56991624e1b)
|
||||
// Generated by Wrangler by running `wrangler types` (hash: d7c533fddfc7f741c2e94fe3097a9b0d)
|
||||
// Runtime types generated with workerd@1.20260908.1 2026-09-10
|
||||
interface __BaseEnv_Env {
|
||||
DB: D1Database;
|
||||
ASSETS: Fetcher;
|
||||
SITE_ORIGIN: "https://free-steam-games.win";
|
||||
IMG_TRANSFORM: "false";
|
||||
ADMIN_RETENTION_DAYS: "180";
|
||||
ACCESS_TEAM_DOMAIN: "polished-hall-48fd.cloudflareaccess.com";
|
||||
ACCESS_AUD_ADMIN: "ca93cf2c7444c20f13b0667a3063d4ac37f702a802e712721da53abb510f9194,a15d6e938cf3109651473e37c0c0fdf84a851248d8f57fdfda26808b40d3c3cb";
|
||||
ACCESS_AUD_INGEST: "bd54655abb62f1bb72f8541177ef2406a066007a4c5ac60eb654f90f6ea88754";
|
||||
@@ -21,7 +22,7 @@ type StringifyValues<EnvType extends Record<string, unknown>> = {
|
||||
[Binding in keyof EnvType]: EnvType[Binding] extends string ? EnvType[Binding] : string;
|
||||
};
|
||||
declare namespace NodeJS {
|
||||
interface ProcessEnv extends StringifyValues<Pick<Cloudflare.Env, "SITE_ORIGIN" | "IMG_TRANSFORM" | "ACCESS_TEAM_DOMAIN" | "ACCESS_AUD_ADMIN" | "ACCESS_AUD_INGEST">> {}
|
||||
interface ProcessEnv extends StringifyValues<Pick<Cloudflare.Env, "SITE_ORIGIN" | "IMG_TRANSFORM" | "ADMIN_RETENTION_DAYS" | "ACCESS_TEAM_DOMAIN" | "ACCESS_AUD_ADMIN" | "ACCESS_AUD_INGEST">> {}
|
||||
}
|
||||
|
||||
// Begin runtime types
|
||||
|
||||
+17
-3
@@ -18,6 +18,8 @@ import { handleIngestApi } from "./routes/ingest";
|
||||
import { adminPage } from "./routes/admin-ui";
|
||||
import { editPage } from "./routes/edit-ui";
|
||||
import { reconcileApproved } from "./lib/reconcile";
|
||||
import { defaultAdminDeps } from "./lib/deps";
|
||||
import { isPruneTick, pruneAdminHistory } from "./lib/prune";
|
||||
|
||||
export default {
|
||||
async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
|
||||
@@ -55,16 +57,28 @@ export default {
|
||||
* D1 query and no fetches.
|
||||
*/
|
||||
async scheduled(_event: ScheduledController, env: Env, ctx: ExecutionContext) {
|
||||
// Retention for audit_log and commit_jobs, once a day (lib/prune.ts).
|
||||
const now = new Date();
|
||||
if (isPruneTick(now)) {
|
||||
ctx.waitUntil(
|
||||
pruneAdminHistory(env, now)
|
||||
.then((out) => {
|
||||
if (out.audit || out.jobs) console.log("prune", out);
|
||||
})
|
||||
.catch((err) => console.error("prune failed", err instanceof Error ? err.message : String(err))),
|
||||
);
|
||||
}
|
||||
|
||||
ctx.waitUntil(
|
||||
reconcileApproved(env)
|
||||
reconcileApproved(env, defaultAdminDeps)
|
||||
.then((out) => {
|
||||
// Log only what is worth reading. reconcileApproved RESOLVES (it does
|
||||
// not throw) when GitHub is unreachable, so without this an outage
|
||||
// that stops every reconcile would emit nothing at all. The idle skip
|
||||
// is excluded because it is the normal state ~96 times a day.
|
||||
if (out.skipped && out.skipped !== "nothing approved") {
|
||||
if (out.skipped && !["nothing approved", "dataset unchanged", "another run in progress"].includes(out.skipped)) {
|
||||
console.warn("reconcile skipped", out);
|
||||
} else if (out.published || out.removed || out.stale) {
|
||||
} else if (out.published || out.removed || out.stale || out.swept) {
|
||||
console.log("reconcile", out);
|
||||
}
|
||||
})
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
/**
|
||||
* Everything the admin handlers reach outside D1, as one injectable object.
|
||||
*
|
||||
* Production passes defaultAdminDeps. Tests and the local admin dev server
|
||||
* (admin/mock) pass fakes, so the REAL handlers - with their SQL, guards and
|
||||
* error mapping - run against an in-memory D1 without a GitHub App or the live
|
||||
* dataset. There is deliberately no flag in the Worker that swaps these.
|
||||
*/
|
||||
import { appendLinks, commitFiles } from "./git-commit";
|
||||
import { gh } from "./github-app";
|
||||
import { fetchRaw, findOverride, findRecord, genreCounts, listByGenre } from "./records";
|
||||
|
||||
export interface AdminDeps {
|
||||
appendLinks: typeof appendLinks;
|
||||
commitFiles: typeof commitFiles;
|
||||
gh: typeof gh;
|
||||
findRecord: typeof findRecord;
|
||||
findOverride: typeof findOverride;
|
||||
genreCounts: typeof genreCounts;
|
||||
listByGenre: typeof listByGenre;
|
||||
fetchRaw: typeof fetchRaw;
|
||||
now: () => Date;
|
||||
}
|
||||
|
||||
export const defaultAdminDeps: AdminDeps = {
|
||||
appendLinks,
|
||||
commitFiles,
|
||||
gh,
|
||||
findRecord,
|
||||
findOverride,
|
||||
genreCounts,
|
||||
listByGenre,
|
||||
fetchRaw,
|
||||
now: () => new Date(),
|
||||
};
|
||||
@@ -0,0 +1,88 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { commitFiles, DELETE_FILE, type GitHubTransport } from "./git-commit";
|
||||
|
||||
/**
|
||||
* The createCommitOnBranch payload, as GitHub would receive it. The mutation
|
||||
* shape is the contract - a deletion that went into `additions`, or a path
|
||||
* outside the allowlist, would be a repository write nobody reviewed.
|
||||
*/
|
||||
|
||||
interface Call {
|
||||
query: string;
|
||||
variables: Record<string, any>;
|
||||
}
|
||||
|
||||
function transport(files: Record<string, string | null>) {
|
||||
const calls: Call[] = [];
|
||||
const fn: GitHubTransport = async (_env, _path, init) => {
|
||||
const body = JSON.parse(String(init?.body)) as Call;
|
||||
calls.push(body);
|
||||
if (body.query.includes("createCommitOnBranch")) {
|
||||
return new Response(JSON.stringify({ data: { createCommitOnBranch: { commit: { oid: "newsha" } } } }));
|
||||
}
|
||||
const repository: Record<string, unknown> = { defaultBranchRef: { target: { oid: "head" } } };
|
||||
Object.entries(body.variables).forEach(([k, v]) => {
|
||||
if (!/^e\d+$/.test(k)) return;
|
||||
const path = String(v).replace(/^main:/, "");
|
||||
const text = files[path];
|
||||
repository[`f${k.slice(1)}`] = text === null || text === undefined ? null : { text, isBinary: false };
|
||||
});
|
||||
return new Response(JSON.stringify({ data: { repository } }));
|
||||
};
|
||||
return { fn, calls };
|
||||
}
|
||||
|
||||
const env = {} as Env;
|
||||
|
||||
describe("commitFiles", () => {
|
||||
it("sends additions and deletions in one commit, against the head it read", async () => {
|
||||
const { fn, calls } = transport({ "data/overrides/1.json": "{}", "data/overrides/2.json": "old" });
|
||||
const sha = await commitFiles(
|
||||
env,
|
||||
[
|
||||
{ path: "data/overrides/1.json", build: () => DELETE_FILE },
|
||||
{ path: "data/overrides/2.json", build: (current) => current + "new" },
|
||||
],
|
||||
"headline",
|
||||
"body",
|
||||
fn,
|
||||
);
|
||||
expect(sha).toBe("newsha");
|
||||
const input = calls.at(-1)!.variables.input;
|
||||
expect(input.expectedHeadOid).toBe("head");
|
||||
expect(input.fileChanges.deletions).toEqual([{ path: "data/overrides/1.json" }]);
|
||||
expect(input.fileChanges.additions.map((a: any) => a.path)).toEqual(["data/overrides/2.json"]);
|
||||
expect(atob(input.fileChanges.additions[0].contents)).toBe("oldnew");
|
||||
});
|
||||
|
||||
it("skips deleting a file that does not exist, and makes no empty commit", async () => {
|
||||
const { fn, calls } = transport({ "data/overrides/1.json": null });
|
||||
const sha = await commitFiles(env, [{ path: "data/overrides/1.json", build: () => DELETE_FILE }], "h", "b", fn);
|
||||
expect(sha).toBeNull();
|
||||
expect(calls.some((c) => c.query.includes("createCommitOnBranch"))).toBe(false);
|
||||
});
|
||||
|
||||
it("tells build() whether the file exists", async () => {
|
||||
const { fn } = transport({ "data/overrides/1.json": null, "data/overrides/2.json": "" });
|
||||
const seen: boolean[] = [];
|
||||
await commitFiles(
|
||||
env,
|
||||
[
|
||||
{ path: "data/overrides/1.json", build: (_c, exists) => (seen.push(exists), null) },
|
||||
{ path: "data/overrides/2.json", build: (_c, exists) => (seen.push(exists), null) },
|
||||
],
|
||||
"h",
|
||||
"b",
|
||||
fn,
|
||||
);
|
||||
expect(seen).toEqual([false, true]);
|
||||
});
|
||||
|
||||
it("refuses a path outside the allowlist, before any request", async () => {
|
||||
const { fn, calls } = transport({});
|
||||
await expect(
|
||||
commitFiles(env, [{ path: "data/data_001.jsonl", build: () => DELETE_FILE }], "h", "b", fn),
|
||||
).rejects.toThrow(/not an allowed path/);
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -6,8 +6,9 @@
|
||||
* in the repository shows as Verified. The REST path would require carrying a
|
||||
* signing key in the Worker, next to a credential that can already write.
|
||||
*
|
||||
* Scope: this module may append to ONE file, `scripts/temp_info.jsonl`. It
|
||||
* cannot touch `data/`. Publication stays the pipeline's job — the Worker only
|
||||
* Scope: this module may append to `scripts/temp_info.jsonl` and write or
|
||||
* delete `data/overrides/<appid>.json` (see WRITABLE). It cannot touch the
|
||||
* data/ shards. Publication stays the pipeline's job — the Worker only
|
||||
* enqueues a request, `ingest_new.py` decides whether the game is real, free
|
||||
* and reachable, and Git remains the source of truth either way.
|
||||
*/
|
||||
@@ -68,8 +69,16 @@ function b64encode(text: string): string {
|
||||
return btoa(bin);
|
||||
}
|
||||
|
||||
async function graphql<T>(env: Env, query: string, variables: unknown): Promise<T> {
|
||||
const res = await gh(env, "/graphql", {
|
||||
/** How requests reach GitHub. `gh` in production; a fake in tests. */
|
||||
export type GitHubTransport = typeof gh;
|
||||
|
||||
async function graphql<T>(
|
||||
env: Env,
|
||||
query: string,
|
||||
variables: unknown,
|
||||
transport: GitHubTransport = gh,
|
||||
): Promise<T> {
|
||||
const res = await transport(env, "/graphql", {
|
||||
method: "POST",
|
||||
body: JSON.stringify({ query, variables }),
|
||||
headers: { "Content-Type": "application/json" },
|
||||
@@ -102,7 +111,7 @@ interface HeadAndFile {
|
||||
* read and the mutation, the mutation is rejected rather than silently
|
||||
* overwriting the newer content.
|
||||
*/
|
||||
async function readHeadAndFile(env: Env, path: string): Promise<HeadAndFile> {
|
||||
async function readHeadAndFile(env: Env, path: string, transport: GitHubTransport = gh): Promise<HeadAndFile> {
|
||||
const data = await graphql<{
|
||||
repository: {
|
||||
defaultBranchRef: { target: { oid: string } } | null;
|
||||
@@ -117,6 +126,7 @@ async function readHeadAndFile(env: Env, path: string): Promise<HeadAndFile> {
|
||||
}
|
||||
}`,
|
||||
{ owner: REPO_OWNER, name: REPO_NAME, expr: `${BRANCH}:${path}` },
|
||||
transport,
|
||||
);
|
||||
|
||||
const oid = data.repository?.defaultBranchRef?.target?.oid;
|
||||
@@ -150,6 +160,8 @@ export async function appendLinks(
|
||||
env: Env,
|
||||
links: string[],
|
||||
actor: string,
|
||||
reason = "",
|
||||
transport: GitHubTransport = gh,
|
||||
): Promise<CommitResult> {
|
||||
let lastError = "";
|
||||
|
||||
@@ -161,7 +173,7 @@ export async function appendLinks(
|
||||
const wanted = [...new Set(links)];
|
||||
|
||||
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
|
||||
const { oid, text } = await readHeadAndFile(env, TEMP_INFO_PATH);
|
||||
const { oid, text } = await readHeadAndFile(env, TEMP_INFO_PATH, transport);
|
||||
|
||||
// Dedup against what is already queued. Without this, approving a game
|
||||
// twice before the pipeline runs queues it twice; ingest_new.py would
|
||||
@@ -217,7 +229,10 @@ export async function appendLinks(
|
||||
// The reviewer's identity belongs in the commit, not only in the
|
||||
// audit table: the audit table is not part of the backup that
|
||||
// matters, and Git is.
|
||||
body: `Approved in /admin by ${actor}.\n\nQueued for scripts/ingest_new.py, which decides whether each game is actually free, reachable and not already present. This commit is a request, not a publication.`,
|
||||
body:
|
||||
`Approved in /admin by ${actor}.` +
|
||||
(reason ? `\n\nReason: ${reason}` : "") +
|
||||
`\n\nQueued for scripts/ingest_new.py, which decides whether each game is actually free, reachable and not already present. This commit is a request, not a publication.`,
|
||||
},
|
||||
expectedHeadOid: oid,
|
||||
fileChanges: {
|
||||
@@ -225,6 +240,7 @@ export async function appendLinks(
|
||||
},
|
||||
},
|
||||
},
|
||||
transport,
|
||||
);
|
||||
return {
|
||||
sha: data.createCommitOnBranch.commit.oid,
|
||||
@@ -255,7 +271,7 @@ export async function appendLinks(
|
||||
let moved = ((err as GraphQlError).types ?? []).includes("STALE_DATA");
|
||||
if (!moved) {
|
||||
try {
|
||||
moved = (await readHeadAndFile(env, TEMP_INFO_PATH)).oid !== oid;
|
||||
moved = (await readHeadAndFile(env, TEMP_INFO_PATH, transport)).oid !== oid;
|
||||
} catch {
|
||||
// Cannot establish it either way; do not paper over the original error.
|
||||
}
|
||||
@@ -278,7 +294,8 @@ export async function appendLinks(
|
||||
async function readHeadAndFiles(
|
||||
env: Env,
|
||||
paths: string[],
|
||||
): Promise<{ oid: string; texts: string[] }> {
|
||||
transport: GitHubTransport,
|
||||
): Promise<{ oid: string; files: { text: string; exists: boolean }[] }> {
|
||||
const aliases = paths
|
||||
.map((_, i) => `f${i}: object(expression: $e${i}) { ... on Blob { text isBinary } }`)
|
||||
.join("\n ");
|
||||
@@ -298,30 +315,37 @@ async function readHeadAndFiles(
|
||||
}
|
||||
}`,
|
||||
vars,
|
||||
transport,
|
||||
);
|
||||
|
||||
const repo = data.repository;
|
||||
const oid = repo?.defaultBranchRef?.target?.oid;
|
||||
if (!oid) throw new Error(`cannot read ${BRANCH} head`);
|
||||
|
||||
const texts = paths.map((path, i) => {
|
||||
const files = paths.map((path, i) => {
|
||||
const obj = repo[`f${i}`];
|
||||
if (obj?.isBinary) throw new Error(`${path} is not text`);
|
||||
return (obj?.text ?? "") as string;
|
||||
// A null object is a file that does not exist on the branch - which a
|
||||
// deletion must know, because deleting a missing path fails the commit.
|
||||
return { text: (obj?.text ?? "") as string, exists: obj !== null && obj !== undefined };
|
||||
});
|
||||
return { oid, texts };
|
||||
return { oid, files };
|
||||
}
|
||||
|
||||
/** Returned by FileEdit.build() to delete the file in the same commit. */
|
||||
export const DELETE_FILE = Symbol("delete-file");
|
||||
|
||||
export interface FileEdit {
|
||||
path: string;
|
||||
/**
|
||||
* Receives the file's CURRENT text (empty when it does not exist) and returns
|
||||
* the new text, or null to leave the file alone. Called again on every retry,
|
||||
* so it must derive its result from the text it is given rather than from
|
||||
* anything captured earlier — that is what makes a retry merge with newer
|
||||
* content instead of clobbering it.
|
||||
* Receives the file's CURRENT text (empty when it does not exist) and
|
||||
* whether it exists, and returns the new text, null to leave the file alone,
|
||||
* or DELETE_FILE to remove it. Called again on every retry, so it must derive
|
||||
* its result from what it is given rather than from anything captured
|
||||
* earlier — that is what makes a retry merge with newer content instead of
|
||||
* clobbering it.
|
||||
*/
|
||||
build: (current: string) => string | null;
|
||||
build: (current: string, exists: boolean) => string | null | typeof DELETE_FILE;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -337,24 +361,29 @@ export async function commitFiles(
|
||||
edits: FileEdit[],
|
||||
headline: string,
|
||||
body: string,
|
||||
transport: GitHubTransport = gh,
|
||||
): Promise<string | null> {
|
||||
if (!edits.length) return null;
|
||||
// Deletions pass the same allowlist as writes: the App can delete anything.
|
||||
for (const e of edits) assertWritable(e.path);
|
||||
let lastError = "";
|
||||
|
||||
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
|
||||
const paths = edits.map((e) => e.path);
|
||||
const { oid, texts } = await readHeadAndFiles(env, paths);
|
||||
const { oid, files } = await readHeadAndFiles(env, paths, transport);
|
||||
|
||||
const additions: { path: string; contents: string }[] = [];
|
||||
const deletions: { path: string }[] = [];
|
||||
edits.forEach((e, i) => {
|
||||
const next = e.build(texts[i]);
|
||||
if (next !== null && next !== texts[i]) {
|
||||
const next = e.build(files[i].text, files[i].exists);
|
||||
if (next === DELETE_FILE) {
|
||||
if (files[i].exists) deletions.push({ path: e.path });
|
||||
} else if (next !== null && next !== files[i].text) {
|
||||
additions.push({ path: e.path, contents: b64encode(next) });
|
||||
}
|
||||
});
|
||||
// Every file already said what it needed to; an empty commit is noise.
|
||||
if (!additions.length) return null;
|
||||
if (!additions.length && !deletions.length) return null;
|
||||
|
||||
try {
|
||||
const data = await graphql<{ createCommitOnBranch: { commit: { oid: string } } }>(
|
||||
@@ -370,9 +399,10 @@ export async function commitFiles(
|
||||
},
|
||||
message: { headline, body },
|
||||
expectedHeadOid: oid,
|
||||
fileChanges: { additions },
|
||||
fileChanges: deletions.length ? { additions, deletions } : { additions },
|
||||
},
|
||||
},
|
||||
transport,
|
||||
);
|
||||
return data.createCommitOnBranch.commit.oid;
|
||||
} catch (err) {
|
||||
@@ -383,7 +413,7 @@ export async function commitFiles(
|
||||
let moved = ((err as GraphQlError).types ?? []).includes("STALE_DATA");
|
||||
if (!moved) {
|
||||
try {
|
||||
moved = (await readHeadAndFile(env, paths[0])).oid !== oid;
|
||||
moved = (await readHeadAndFile(env, paths[0], transport)).oid !== oid;
|
||||
} catch {
|
||||
/* cannot tell; surface the original error */
|
||||
}
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { acquireLease, readLease, readState, releaseLease, writeState } from "./locks";
|
||||
import { pruneAdminHistory, retentionDays, isPruneTick } from "./prune";
|
||||
import { makeEnv } from "../testing/fixtures";
|
||||
|
||||
const T0 = new Date("2026-09-17T00:00:00Z");
|
||||
const later = (ms: number) => new Date(T0.getTime() + ms);
|
||||
|
||||
describe("leases", () => {
|
||||
it("one holder at a time, until release or expiry", async () => {
|
||||
const { DB } = makeEnv();
|
||||
expect(await acquireLease(DB, "job", "a", 60_000, T0)).toBe("acquired");
|
||||
expect(await acquireLease(DB, "job", "b", 60_000, later(1_000))).toBe("held");
|
||||
expect((await readLease(DB, "job"))?.owner).toBe("a");
|
||||
|
||||
// Someone else's release does nothing.
|
||||
await releaseLease(DB, "job", "b");
|
||||
expect(await acquireLease(DB, "job", "b", 60_000, later(2_000))).toBe("held");
|
||||
|
||||
// An expired lease is taken over (a crashed isolate never releases).
|
||||
expect(await acquireLease(DB, "job", "b", 60_000, later(61_000))).toBe("acquired");
|
||||
await releaseLease(DB, "job", "b");
|
||||
expect(await readLease(DB, "job")).toBeNull();
|
||||
});
|
||||
|
||||
it("reports unavailable instead of throwing when the table is missing", async () => {
|
||||
const env = makeEnv();
|
||||
env.sqlite.db.exec("DROP TABLE admin_locks; DROP TABLE admin_state;");
|
||||
expect(await acquireLease(env.DB, "job", "a", 1_000, T0)).toBe("unavailable");
|
||||
await expect(releaseLease(env.DB, "job", "a")).resolves.toBeUndefined();
|
||||
expect(await readState(env.DB, "k")).toBeNull();
|
||||
await expect(writeState(env.DB, "k", "v", T0)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it("state round-trips", async () => {
|
||||
const { DB } = makeEnv();
|
||||
expect(await readState(DB, "k")).toBeUndefined();
|
||||
await writeState(DB, "k", "v1", T0);
|
||||
await writeState(DB, "k", "v2", later(1));
|
||||
expect(await readState(DB, "k")).toBe("v2");
|
||||
});
|
||||
});
|
||||
|
||||
describe("retention", () => {
|
||||
it("deletes audit and job rows older than the window, and records that it did", async () => {
|
||||
const env = makeEnv();
|
||||
const insertAudit = env.sqlite.db.prepare(
|
||||
"INSERT INTO audit_log (actor, action, target, detail_json, created_at) VALUES ('x', 'ping', NULL, '{}', ?)",
|
||||
);
|
||||
insertAudit.run("2026-01-01T00:00:00Z");
|
||||
insertAudit.run("2026-09-01T00:00:00Z");
|
||||
env.sqlite.db
|
||||
.prepare("INSERT INTO commit_jobs (id, kind, status, target_path, created_at) VALUES ('old', 'approve', 'committed', 'x', '2025-12-01T00:00:00Z')")
|
||||
.run();
|
||||
|
||||
const out = await pruneAdminHistory(env, T0);
|
||||
expect(out).toMatchObject({ audit: 1, jobs: 1 });
|
||||
const actions = env.sqlite.db.prepare("SELECT action FROM audit_log ORDER BY id").all() as any[];
|
||||
expect(actions.map((a) => a.action)).toEqual(["ping", "admin.prune"]);
|
||||
});
|
||||
|
||||
it("never lets a bad value shrink the window below a week", () => {
|
||||
expect(retentionDays({ ADMIN_RETENTION_DAYS: "180" } as Env)).toBe(180);
|
||||
expect(retentionDays({ ADMIN_RETENTION_DAYS: "0" } as unknown as Env)).toBe(180);
|
||||
expect(retentionDays({ ADMIN_RETENTION_DAYS: "abc" } as unknown as Env)).toBe(180);
|
||||
expect(retentionDays({ ADMIN_RETENTION_DAYS: "30" } as unknown as Env)).toBe(30);
|
||||
});
|
||||
|
||||
it("runs once a day", () => {
|
||||
expect(isPruneTick(new Date("2026-09-17T03:00:00Z"))).toBe(true);
|
||||
expect(isPruneTick(new Date("2026-09-17T03:15:00Z"))).toBe(false);
|
||||
expect(isPruneTick(new Date("2026-09-17T04:00:00Z"))).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* Cross-isolate leases and small persisted state, both in D1
|
||||
* (migrations/0002_admin_state.sql).
|
||||
*
|
||||
* TOLERANT OF THE MIGRATION NOT BEING APPLIED YET. D1 migrations are applied
|
||||
* by hand (docs/ADMIN.md), so for some window the deployed Worker may be newer
|
||||
* than the database. A missing table is reported as "unavailable" rather than
|
||||
* thrown, and every caller has a fallback - so deploy order never matters.
|
||||
*/
|
||||
|
||||
export type LeaseResult = "acquired" | "held" | "unavailable";
|
||||
|
||||
function missingTable(err: unknown): boolean {
|
||||
return /no such table/i.test(err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
|
||||
/**
|
||||
* Take the named lease for `ttlMs`, unless someone else holds an unexpired one.
|
||||
*
|
||||
* One statement, so there is no read-then-write gap: the upsert only
|
||||
* overwrites a lease whose expiry has passed, and `changes` says whether it
|
||||
* did.
|
||||
*/
|
||||
export async function acquireLease(
|
||||
db: D1Database,
|
||||
name: string,
|
||||
owner: string,
|
||||
ttlMs: number,
|
||||
now: Date,
|
||||
): Promise<LeaseResult> {
|
||||
const expires = new Date(now.getTime() + ttlMs).toISOString();
|
||||
try {
|
||||
const res = await db
|
||||
.prepare(
|
||||
`INSERT INTO admin_locks (name, owner, expires_at) VALUES (?1, ?2, ?3)
|
||||
ON CONFLICT(name) DO UPDATE SET owner = excluded.owner, expires_at = excluded.expires_at
|
||||
WHERE admin_locks.expires_at < ?4`,
|
||||
)
|
||||
.bind(name, owner, expires, now.toISOString())
|
||||
.run();
|
||||
return res.meta.changes === 1 ? "acquired" : "held";
|
||||
} catch (err) {
|
||||
if (missingTable(err)) {
|
||||
console.warn("locks: admin_locks is missing - apply migration 0002");
|
||||
return "unavailable";
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/** Release a lease this owner holds. Someone else's lease is left alone. */
|
||||
export async function releaseLease(db: D1Database, name: string, owner: string): Promise<void> {
|
||||
try {
|
||||
await db.prepare("DELETE FROM admin_locks WHERE name = ? AND owner = ?").bind(name, owner).run();
|
||||
} catch (err) {
|
||||
if (!missingTable(err)) throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/** The current lease, for /api/admin/health. null when free or unavailable. */
|
||||
export async function readLease(
|
||||
db: D1Database,
|
||||
name: string,
|
||||
): Promise<{ owner: string; expires_at: string } | null> {
|
||||
try {
|
||||
return await db
|
||||
.prepare("SELECT owner, expires_at FROM admin_locks WHERE name = ?")
|
||||
.bind(name)
|
||||
.first<{ owner: string; expires_at: string }>();
|
||||
} catch (err) {
|
||||
if (missingTable(err)) return null;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/** A persisted value, or undefined when unset. null when the table is missing. */
|
||||
export async function readState(db: D1Database, key: string): Promise<string | undefined | null> {
|
||||
try {
|
||||
const row = await db.prepare("SELECT value FROM admin_state WHERE key = ?").bind(key).first<{ value: string }>();
|
||||
return row?.value;
|
||||
} catch (err) {
|
||||
if (missingTable(err)) return null;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
export async function writeState(db: D1Database, key: string, value: string, now: Date): Promise<void> {
|
||||
try {
|
||||
await db
|
||||
.prepare(
|
||||
`INSERT INTO admin_state (key, value, updated_at) VALUES (?1, ?2, ?3)
|
||||
ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`,
|
||||
)
|
||||
.bind(key, value, now.toISOString())
|
||||
.run();
|
||||
} catch (err) {
|
||||
if (!missingTable(err)) throw err;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
/**
|
||||
* Retention for the admin tables, run from the cron (worker/index.ts).
|
||||
*
|
||||
* audit_log and commit_jobs grew forever: nothing ever deleted a row, and the
|
||||
* privacy policy described the audit log without saying how long it is kept.
|
||||
* ADMIN_RETENTION_DAYS (wrangler.jsonc) is that number, and
|
||||
* docs/PRIVACY_POLICY.md states it - change one, change the other.
|
||||
*
|
||||
* ingest_queue and ingest_decisions are deliberately NOT pruned: the decisions
|
||||
* are what stop the discovery sweep re-proposing a rejected game, and queue
|
||||
* history is what the admin screen shows for a game's past rows.
|
||||
*/
|
||||
import { audit } from "./audit";
|
||||
|
||||
export interface PruneResult {
|
||||
cutoff: string;
|
||||
audit: number;
|
||||
jobs: number;
|
||||
}
|
||||
|
||||
export function retentionDays(env: Pick<Env, "ADMIN_RETENTION_DAYS">): number {
|
||||
const days = Number(env.ADMIN_RETENTION_DAYS);
|
||||
// Never below a week: a typo in the var must not wipe the evidence trail.
|
||||
return Number.isFinite(days) && days >= 7 ? Math.floor(days) : 180;
|
||||
}
|
||||
|
||||
export async function pruneAdminHistory(
|
||||
env: Pick<Env, "DB" | "ADMIN_RETENTION_DAYS">,
|
||||
now: Date,
|
||||
): Promise<PruneResult> {
|
||||
const cutoff = new Date(now.getTime() - retentionDays(env) * 86_400_000).toISOString();
|
||||
const [auditRes, jobsRes] = await env.DB.batch([
|
||||
env.DB.prepare("DELETE FROM audit_log WHERE created_at < ?").bind(cutoff),
|
||||
env.DB.prepare("DELETE FROM commit_jobs WHERE created_at < ?").bind(cutoff),
|
||||
]);
|
||||
const result = { cutoff, audit: auditRes.meta.changes ?? 0, jobs: jobsRes.meta.changes ?? 0 };
|
||||
// Recorded only when something went, so an idle day leaves no row behind.
|
||||
if (result.audit || result.jobs) {
|
||||
await audit(env as Env, "retention", "admin.prune", null, result);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Once a day, in the first cron tick after 03:00 UTC. */
|
||||
export function isPruneTick(now: Date): boolean {
|
||||
return now.getUTCHours() === 3 && now.getUTCMinutes() < 15;
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { publishedAppids, reconcileApproved } from "./reconcile";
|
||||
import { acquireLease } from "./locks";
|
||||
import { decisionOf, fakeDeps, insertDecision, insertRow, makeEnv, rowStatus, shardLine } from "../testing/fixtures";
|
||||
|
||||
function dataset(appids: string[], stamp = "2026-09-17T00:00:00Z") {
|
||||
return {
|
||||
"data/index.json": JSON.stringify({ last_updated: stamp, files: [{ name: "data_001.jsonl", sha256: stamp }] }),
|
||||
"data/data_001.jsonl": appids.map((a) => shardLine(a)).join("\n") + "\n",
|
||||
"scripts/removed_games.jsonl": "",
|
||||
};
|
||||
}
|
||||
|
||||
describe("publishedAppids", () => {
|
||||
it("reads the link field only, not a store URL quoted in a note", () => {
|
||||
const text = [
|
||||
shardLine("730"),
|
||||
JSON.stringify({ link: "https://store.steampowered.com/app/440/", notes: "see https://store.steampowered.com/app/999/" }),
|
||||
].join("\n");
|
||||
expect([...publishedAppids([text])].sort()).toEqual(["440", "730"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("reconcileApproved", () => {
|
||||
it("promotes an approved row that has appeared in data/", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "a", appid: "100", status: "approved", decided_at: "2026-09-17T10:00:00Z" });
|
||||
const out = await reconcileApproved(env, fakeDeps({ raw: dataset(["100"]) }));
|
||||
expect(out).toMatchObject({ published: 1, checked: 1 });
|
||||
expect(rowStatus(env, "a")).toBe("committed");
|
||||
expect(decisionOf(env, "100")).toBe("approved");
|
||||
});
|
||||
|
||||
it("sweeps undecided and failed rows whose game is already live", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "p", appid: "200", status: "pending" });
|
||||
insertRow(env, { id: "f", appid: "201", status: "failed", reject_reason: "never appeared in data/" });
|
||||
insertRow(env, { id: "other", appid: "202", status: "pending" });
|
||||
insertDecision(env, "201", "rejected");
|
||||
|
||||
const out = await reconcileApproved(env, fakeDeps({ raw: dataset(["200", "201"]) }));
|
||||
expect(out.swept).toBe(2);
|
||||
expect(rowStatus(env, "p")).toBe("committed");
|
||||
expect(rowStatus(env, "f")).toBe("committed");
|
||||
expect(rowStatus(env, "other")).toBe("pending");
|
||||
// The game is in data/: a stale rejection is replaced by the fact.
|
||||
expect(decisionOf(env, "201")).toBe("approved");
|
||||
|
||||
const audit = env.sqlite.db.prepare("SELECT detail_json FROM audit_log WHERE action='reconcile.approved'").get() as any;
|
||||
expect(JSON.parse(audit.detail_json).appids.swept.sort()).toEqual(["200", "201"]);
|
||||
});
|
||||
|
||||
it("does not fetch the dataset again while it has not changed", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "p", appid: "300", status: "pending" });
|
||||
const raw = dataset(["999"]);
|
||||
const fetched: string[] = [];
|
||||
const deps = fakeDeps({ raw });
|
||||
const counting = { ...deps, fetchRaw: async (path: string, ttl: number) => (fetched.push(path), deps.fetchRaw(path, ttl)) };
|
||||
|
||||
await reconcileApproved(env, counting);
|
||||
expect(fetched).toContain("data/data_001.jsonl");
|
||||
|
||||
fetched.length = 0;
|
||||
const second = await reconcileApproved(env, counting);
|
||||
expect(second.skipped).toBe("dataset unchanged");
|
||||
expect(fetched).toEqual(["data/index.json"]);
|
||||
|
||||
// A manual run always looks.
|
||||
fetched.length = 0;
|
||||
await reconcileApproved(env, counting, { manual: true });
|
||||
expect(fetched).toContain("data/data_001.jsonl");
|
||||
});
|
||||
|
||||
it("costs no fetch at all when nothing is approved or sweepable", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "c", appid: "400", status: "committed" });
|
||||
const fetched: string[] = [];
|
||||
const deps = fakeDeps();
|
||||
const out = await reconcileApproved(env, { ...deps, fetchRaw: async (p) => (fetched.push(p), null) });
|
||||
expect(out.skipped).toBe("nothing approved");
|
||||
expect(fetched).toEqual([]);
|
||||
});
|
||||
|
||||
it("stays out while another isolate holds the lease", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "a", appid: "500", status: "approved", decided_at: "2026-09-17T10:00:00Z" });
|
||||
const deps = fakeDeps({ raw: dataset(["500"]) });
|
||||
expect(await acquireLease(env.DB, "reconcile", "someone-else", 60_000, deps.now())).toBe("acquired");
|
||||
const out = await reconcileApproved(env, deps);
|
||||
expect(out.skipped).toBe("another run in progress");
|
||||
expect(rowStatus(env, "a")).toBe("approved");
|
||||
});
|
||||
|
||||
it("still runs when migration 0002 has not been applied", async () => {
|
||||
const env = makeEnv();
|
||||
env.sqlite.db.exec("DROP TABLE admin_locks; DROP TABLE admin_state;");
|
||||
insertRow(env, { id: "a", appid: "600", status: "approved", decided_at: "2026-09-17T10:00:00Z" });
|
||||
const out = await reconcileApproved(env, fakeDeps({ raw: dataset(["600"]) }));
|
||||
expect(out.published).toBe(1);
|
||||
});
|
||||
|
||||
it("ages out an approval nothing ever observed", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "a", appid: "700", status: "approved", decided_at: "2026-09-16T00:00:00Z" });
|
||||
const out = await reconcileApproved(env, fakeDeps({ raw: dataset(["1"]) }));
|
||||
expect(out.stale).toBe(1);
|
||||
expect(rowStatus(env, "a")).toBe("failed");
|
||||
});
|
||||
});
|
||||
+183
-78
@@ -11,14 +11,22 @@
|
||||
*
|
||||
* This reconciler is therefore the ONLY writer of ingest_decisions('approved'),
|
||||
* and it writes one only after observing the appid in the published dataset.
|
||||
*
|
||||
* It also SWEEPS: a pending, deferred or failed row whose game is already in
|
||||
* data/ is marked committed. That happens when the browser extension queues a
|
||||
* game discovery also proposed, and when an approval aged out to 'failed' (see
|
||||
* STALE_AFTER_MS) and was published afterwards anyway - before the sweep such a
|
||||
* row stayed 'failed' forever for a game that was live.
|
||||
*/
|
||||
|
||||
const RAW_BASE =
|
||||
"https://raw.githubusercontent.com/poli0981/free-steam-games-list/main";
|
||||
import type { AdminDeps } from "./deps";
|
||||
import { acquireLease, releaseLease, readState, writeState } from "./locks";
|
||||
|
||||
/** Bounded so one cron tick cannot run away; the next tick picks up the rest. */
|
||||
const MAX_PER_RUN = 200;
|
||||
|
||||
/** Rows in one D1 batch. */
|
||||
const BATCH = 50;
|
||||
|
||||
/**
|
||||
* How long an approval may sit unobserved before it is called failed.
|
||||
*
|
||||
@@ -31,20 +39,14 @@ const MAX_PER_RUN = 200;
|
||||
*/
|
||||
const STALE_AFTER_MS = 12 * 60 * 60 * 1000;
|
||||
|
||||
interface IndexFile {
|
||||
files?: { name?: unknown }[];
|
||||
}
|
||||
/** A run fetches ~6 MB; a crashed isolate's lease frees itself after this. */
|
||||
const LEASE_MS = 10 * 60 * 1000;
|
||||
|
||||
async function fetchText(path: string): Promise<string | null> {
|
||||
// cacheTtl 0: this is the freshness check itself. A cached shard would keep
|
||||
// reporting a game as unpublished after it landed, and the row would sit in
|
||||
// 'approved' until the cache expired.
|
||||
const res = await fetch(`${RAW_BASE}/${path}`, {
|
||||
cf: { cacheTtl: 0 },
|
||||
headers: { Accept: "text/plain, application/json, */*" },
|
||||
});
|
||||
return res.ok ? await res.text() : null;
|
||||
}
|
||||
/** admin_state key: the dataset generation the last sweep ran against. */
|
||||
const WATERMARK_KEY = "reconcile.index";
|
||||
|
||||
/** Appids listed per outcome in the audit row. */
|
||||
const AUDIT_APPIDS = 200;
|
||||
|
||||
export interface ReconcileResult {
|
||||
checked: number;
|
||||
@@ -52,9 +54,20 @@ export interface ReconcileResult {
|
||||
removed: number;
|
||||
/** Approvals aged out because nothing ever observed them. */
|
||||
stale: number;
|
||||
/** Undecided or failed rows marked committed because the game is live. */
|
||||
swept: number;
|
||||
skipped?: string;
|
||||
}
|
||||
|
||||
const empty = (skipped: string, checked = 0): ReconcileResult => ({
|
||||
checked,
|
||||
published: 0,
|
||||
removed: 0,
|
||||
stale: 0,
|
||||
swept: 0,
|
||||
skipped,
|
||||
});
|
||||
|
||||
/**
|
||||
* appid -> when the pipeline removed it, from scripts/removed_games.jsonl.
|
||||
*
|
||||
@@ -89,104 +102,150 @@ function parseRemoved(text: string): Map<string, number> {
|
||||
}
|
||||
|
||||
/**
|
||||
* Promote 'approved' rows that have since appeared in the dataset.
|
||||
* Every appid in the published shards.
|
||||
*
|
||||
* Substring search rather than JSON parsing: the shards total ~6 MB, and
|
||||
* `"/app/<id>/"` is anchored on both sides by the canonical link format, so it
|
||||
* cannot collide with a longer appid or match inside a description.
|
||||
* A regex over the canonical link rather than JSON parsing: the shards total
|
||||
* ~6 MB, and `"link": ".../app/<id>/"` is anchored by the field name, so it
|
||||
* cannot match inside a description or a note quoting a store URL.
|
||||
*/
|
||||
export function publishedAppids(shards: string[]): Set<string> {
|
||||
const out = new Set<string>();
|
||||
const re = /"link"\s*:\s*"https?:\/\/store\.steampowered\.com\/app\/(\d+)\//g;
|
||||
for (const text of shards) {
|
||||
for (const m of text.matchAll(re)) out.add(m[1]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* In-flight guard, per isolate.
|
||||
*
|
||||
* A full run fetches index.json, every shard and removed_games.jsonl - roughly
|
||||
* 6 MB. The 15-minute cron and the manual /admin button can both start one, and
|
||||
* the button had no busy state at all, so a double-click fired two.
|
||||
*
|
||||
* Scope is honest about its limit: this stops repeats within one isolate, which
|
||||
* covers the double-click and a cron tick overlapping its predecessor. Two
|
||||
* different isolates can still overlap, and that is left alone deliberately -
|
||||
* every write in this function is idempotent (`WHERE status='approved'`,
|
||||
* `ON CONFLICT DO NOTHING`), so the only cost is duplicated fetching, and a
|
||||
* real cross-isolate lock would mean a D1 migration for a purely wasteful race.
|
||||
* The upsert that records a publication. Overwrites a 'rejected' decision:
|
||||
* the game IS in data/, which is a fact, and a stale rejection would leave the
|
||||
* admin screen offering to approve or reject a live game.
|
||||
*/
|
||||
function recordPublished(env: Env, appid: string, now: string): D1PreparedStatement {
|
||||
return env.DB.prepare(
|
||||
`INSERT INTO ingest_decisions (appid, decision, reason, decided_by, decided_at)
|
||||
VALUES (?1, 'approved', 'observed in data/', 'reconcile', ?2)
|
||||
ON CONFLICT(appid) DO UPDATE SET
|
||||
decision = 'approved', reason = excluded.reason,
|
||||
decided_by = excluded.decided_by, decided_at = excluded.decided_at
|
||||
WHERE ingest_decisions.decision <> 'approved'`,
|
||||
).bind(appid, now);
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-isolate guard, still kept under the D1 lease: it makes a double click
|
||||
* within one isolate share one run instead of the second one being refused.
|
||||
*/
|
||||
let inFlight: Promise<ReconcileResult> | null = null;
|
||||
|
||||
export async function reconcileApproved(env: Env): Promise<ReconcileResult> {
|
||||
export async function reconcileApproved(
|
||||
env: Env,
|
||||
deps: AdminDeps,
|
||||
options: { manual?: boolean } = {},
|
||||
): Promise<ReconcileResult> {
|
||||
if (inFlight) return inFlight;
|
||||
inFlight = runReconcile(env).finally(() => {
|
||||
inFlight = runLocked(env, deps, options).finally(() => {
|
||||
inFlight = null;
|
||||
});
|
||||
return inFlight;
|
||||
}
|
||||
|
||||
async function runReconcile(env: Env): Promise<ReconcileResult> {
|
||||
const open = await env.DB.prepare(
|
||||
`SELECT id, appid, decided_at FROM ingest_queue WHERE status = 'approved'
|
||||
ORDER BY decided_at ASC, id LIMIT ?`,
|
||||
)
|
||||
.bind(MAX_PER_RUN)
|
||||
.all<{ id: string; appid: string; decided_at: string | null }>();
|
||||
|
||||
const rows = open.results ?? [];
|
||||
// The common case by far. Returning before any fetch keeps the cron
|
||||
// essentially free on the many ticks where nothing is awaiting publication.
|
||||
if (!rows.length) {
|
||||
return { checked: 0, published: 0, removed: 0, stale: 0, skipped: "nothing approved" };
|
||||
async function runLocked(env: Env, deps: AdminDeps, options: { manual?: boolean }): Promise<ReconcileResult> {
|
||||
const owner = crypto.randomUUID();
|
||||
const lease = await acquireLease(env.DB, "reconcile", owner, LEASE_MS, deps.now());
|
||||
// "unavailable" = migration 0002 not applied yet: run anyway, guarded only
|
||||
// per isolate, exactly as before the lease existed.
|
||||
if (lease === "held") return empty("another run in progress");
|
||||
try {
|
||||
return await runReconcile(env, deps, options, lease !== "unavailable");
|
||||
} finally {
|
||||
if (lease === "acquired") await releaseLease(env.DB, "reconcile", owner);
|
||||
}
|
||||
}
|
||||
|
||||
const bail = (skipped: string): ReconcileResult => ({
|
||||
checked: rows.length, published: 0, removed: 0, stale: 0, skipped,
|
||||
});
|
||||
async function runReconcile(
|
||||
env: Env,
|
||||
deps: AdminDeps,
|
||||
options: { manual?: boolean },
|
||||
stateAvailable: boolean,
|
||||
): Promise<ReconcileResult> {
|
||||
const [approvedRes, sweepableRes] = await Promise.all([
|
||||
env.DB.prepare(
|
||||
`SELECT id, appid, decided_at FROM ingest_queue WHERE status = 'approved'
|
||||
ORDER BY decided_at ASC, id LIMIT ?`,
|
||||
)
|
||||
.bind(MAX_PER_RUN)
|
||||
.all<{ id: string; appid: string; decided_at: string | null }>(),
|
||||
env.DB.prepare(
|
||||
"SELECT COUNT(*) AS n FROM ingest_queue WHERE status IN ('pending','deferred','failed')",
|
||||
).first<{ n: number }>(),
|
||||
]);
|
||||
const approved = approvedRes.results ?? [];
|
||||
const sweepable = sweepableRes?.n ?? 0;
|
||||
|
||||
const index = await fetchText("data/index.json");
|
||||
if (!index) return bail("index unreachable");
|
||||
// The common case by far: nothing to promote and nothing to sweep costs two
|
||||
// indexed D1 queries and no fetch at all.
|
||||
if (!approved.length && !sweepable) return empty("nothing approved");
|
||||
|
||||
const index = await deps.fetchRaw("data/index.json", 0);
|
||||
if (!index) return empty("index unreachable", approved.length);
|
||||
|
||||
let shardNames: string[];
|
||||
let generation = "";
|
||||
try {
|
||||
const parsed = JSON.parse(index) as IndexFile;
|
||||
const parsed = JSON.parse(index) as {
|
||||
last_updated?: unknown;
|
||||
files?: { name?: unknown; sha256?: unknown }[];
|
||||
};
|
||||
shardNames = (parsed.files ?? [])
|
||||
.map((f) => f?.name)
|
||||
.filter((n): n is string => typeof n === "string" && /^data_\d{3}\.jsonl$/.test(n));
|
||||
generation = JSON.stringify([parsed.last_updated, (parsed.files ?? []).map((f) => f?.sha256 ?? f?.name)]);
|
||||
} catch {
|
||||
return bail("index unparseable");
|
||||
return empty("index unparseable", approved.length);
|
||||
}
|
||||
if (!shardNames.length) return bail("no shards listed");
|
||||
if (!shardNames.length) return empty("no shards listed", approved.length);
|
||||
|
||||
const shards = await Promise.all(shardNames.map((n) => fetchText(`data/${n}`)));
|
||||
// Nothing approved, and the dataset has not changed since the last sweep:
|
||||
// there is nothing new to observe, so skip the ~6 MB fetch. A manual run
|
||||
// always looks. Without the state table there is no memory between ticks,
|
||||
// so only approvals or a manual run justify the fetch.
|
||||
if (!approved.length && !options.manual) {
|
||||
if (!stateAvailable) return empty("nothing approved");
|
||||
const last = await readState(env.DB, WATERMARK_KEY);
|
||||
if (last === null || last === generation) return empty("dataset unchanged");
|
||||
}
|
||||
|
||||
const shards = await Promise.all(shardNames.map((n) => deps.fetchRaw(`data/${n}`, 0)));
|
||||
// A partial read would look exactly like "these games were never published"
|
||||
// and would strand every row it touched, so treat it as a failed tick.
|
||||
if (shards.some((s) => s === null)) return bail("shard unreachable");
|
||||
if (shards.some((s) => s === null)) return empty("shard unreachable", approved.length);
|
||||
const live = publishedAppids(shards as string[]);
|
||||
|
||||
// Kept separate from the "" fallback: an unreadable removals file is not the
|
||||
// same as an empty one. Without it we cannot tell a pipeline rejection from
|
||||
// an unprocessed row, so neither of those two judgements may be made on this
|
||||
// tick — promotions still can, because those only need the shards.
|
||||
const removedText = await fetchText("scripts/removed_games.jsonl");
|
||||
const removedText = await deps.fetchRaw("scripts/removed_games.jsonl", 0);
|
||||
const removed = removedText === null ? null : parseRemoved(removedText);
|
||||
|
||||
const nowMs = Date.now();
|
||||
const now = new Date(nowMs).toISOString();
|
||||
const nowDate = deps.now();
|
||||
const nowMs = nowDate.getTime();
|
||||
const now = nowDate.toISOString();
|
||||
const writes: D1PreparedStatement[] = [];
|
||||
let published = 0;
|
||||
let rejected = 0;
|
||||
let stale = 0;
|
||||
const outcome = { published: [] as string[], removed: [] as string[], stale: [] as string[], swept: [] as string[] };
|
||||
|
||||
for (const row of rows) {
|
||||
const needle = `/app/${row.appid}/`;
|
||||
|
||||
if (shards.some((s) => s!.includes(needle))) {
|
||||
published++;
|
||||
for (const row of approved) {
|
||||
if (live.has(row.appid)) {
|
||||
outcome.published.push(row.appid);
|
||||
writes.push(
|
||||
env.DB.prepare(
|
||||
`UPDATE ingest_queue SET status = 'committed' WHERE id = ? AND status = 'approved'`,
|
||||
).bind(row.id),
|
||||
// Only NOW is the appid genuinely decided, and only now may
|
||||
// /api/ingest/known tell the sweep to stop offering it.
|
||||
env.DB.prepare(
|
||||
`INSERT INTO ingest_decisions (appid, decision, reason, decided_by, decided_at)
|
||||
VALUES (?, 'approved', 'observed in data/', 'reconcile', ?)
|
||||
ON CONFLICT(appid) DO NOTHING`,
|
||||
).bind(row.appid, now),
|
||||
recordPublished(env, row.appid, now),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
@@ -200,7 +259,7 @@ async function runReconcile(env: Env): Promise<ReconcileResult> {
|
||||
|
||||
// Only a removal recorded AFTER this approval describes this approval.
|
||||
if (removedMs !== undefined && !Number.isNaN(decidedMs) && removedMs > decidedMs) {
|
||||
rejected++;
|
||||
outcome.removed.push(row.appid);
|
||||
// Deliberately NOT written to ingest_decisions: that would hide the appid
|
||||
// from every future sweep on the strength of one bad day (a delisting, a
|
||||
// temporary outage). Left as 'failed' so it is visible in the admin queue
|
||||
@@ -226,7 +285,7 @@ async function runReconcile(env: Env): Promise<ReconcileResult> {
|
||||
// would be permanently, silently lost. 'failed' is decidable, so the
|
||||
// reviewer can simply approve it again — which re-commits the link.
|
||||
if (!Number.isNaN(decidedMs) && nowMs - decidedMs > STALE_AFTER_MS) {
|
||||
stale++;
|
||||
outcome.stale.push(row.appid);
|
||||
writes.push(
|
||||
env.DB.prepare(
|
||||
`UPDATE ingest_queue
|
||||
@@ -238,18 +297,64 @@ async function runReconcile(env: Env): Promise<ReconcileResult> {
|
||||
}
|
||||
}
|
||||
|
||||
// The sweep. Reads every undecided row once; the published set is already
|
||||
// in memory, so this costs no further fetch.
|
||||
if (sweepable) {
|
||||
const rows = await env.DB.prepare(
|
||||
"SELECT id, appid FROM ingest_queue WHERE status IN ('pending','deferred','failed')",
|
||||
).all<{ id: string; appid: string }>();
|
||||
const seen = new Set<string>();
|
||||
for (const row of rows.results ?? []) {
|
||||
if (!live.has(row.appid)) continue;
|
||||
writes.push(
|
||||
env.DB.prepare(
|
||||
`UPDATE ingest_queue
|
||||
SET status = 'committed', decided_by = 'reconcile', decided_at = ?,
|
||||
reject_reason = 'observed in data/'
|
||||
WHERE id = ? AND status IN ('pending','deferred','failed')`,
|
||||
).bind(now, row.id),
|
||||
);
|
||||
if (!seen.has(row.appid)) {
|
||||
seen.add(row.appid);
|
||||
outcome.swept.push(row.appid);
|
||||
writes.push(recordPublished(env, row.appid, now));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < writes.length; i += BATCH) {
|
||||
await env.DB.batch(writes.slice(i, i + BATCH));
|
||||
}
|
||||
if (stateAvailable) await writeState(env.DB, WATERMARK_KEY, generation, nowDate);
|
||||
|
||||
const result: ReconcileResult = {
|
||||
checked: approved.length,
|
||||
published: outcome.published.length,
|
||||
removed: outcome.removed.length,
|
||||
stale: outcome.stale.length,
|
||||
swept: outcome.swept.length,
|
||||
};
|
||||
|
||||
if (writes.length) {
|
||||
await env.DB.batch(writes);
|
||||
const cap = (xs: string[]) => xs.slice(0, AUDIT_APPIDS);
|
||||
await env.DB.prepare(
|
||||
`INSERT INTO audit_log (actor, action, target, detail_json, created_at)
|
||||
VALUES ('reconcile', 'reconcile.approved', NULL, ?, ?)`,
|
||||
)
|
||||
.bind(
|
||||
JSON.stringify({ checked: rows.length, published, removed: rejected, stale }),
|
||||
JSON.stringify({
|
||||
...result,
|
||||
appids: {
|
||||
published: cap(outcome.published),
|
||||
removed: cap(outcome.removed),
|
||||
stale: cap(outcome.stale),
|
||||
swept: cap(outcome.swept),
|
||||
},
|
||||
}),
|
||||
now,
|
||||
)
|
||||
.run();
|
||||
}
|
||||
|
||||
return { checked: rows.length, published, removed: rejected, stale };
|
||||
return result;
|
||||
}
|
||||
@@ -30,7 +30,13 @@ export interface GameRecord {
|
||||
[k: string]: unknown;
|
||||
}
|
||||
|
||||
async function fetchText(path: string, ttl: number): Promise<string | null> {
|
||||
/**
|
||||
* One repository file as text, through Cloudflare's edge cache for `ttl`
|
||||
* seconds (0 bypasses it). Exported as a dependency so the admin handlers and
|
||||
* reconcile can be run against fixture data in tests and in `npm run
|
||||
* dev:admin`.
|
||||
*/
|
||||
export async function fetchRaw(path: string, ttl: number): Promise<string | null> {
|
||||
const res = await fetch(`${RAW_BASE}/${path}`, {
|
||||
cf: { cacheTtl: ttl, cacheEverything: true },
|
||||
headers: { Accept: "text/plain, application/json, */*" },
|
||||
@@ -41,7 +47,7 @@ async function fetchText(path: string, ttl: number): Promise<string | null> {
|
||||
async function shardNames(): Promise<string[]> {
|
||||
// index.json is ~170 bytes and is the catalogue's own manifest; hardcoding
|
||||
// five shard names would break the day the catalogue crosses 4,000 games.
|
||||
const text = await fetchText("data/index.json", 30);
|
||||
const text = await fetchRaw("data/index.json", 30);
|
||||
if (!text) return [];
|
||||
try {
|
||||
const parsed = JSON.parse(text) as { files?: { name?: unknown }[] };
|
||||
@@ -61,7 +67,7 @@ export async function findRecord(appid: string): Promise<GameRecord | null> {
|
||||
|
||||
const needle = `/app/${appid}/`;
|
||||
for (const name of names) {
|
||||
const text = await fetchText(`data/${name}`, SHARD_TTL);
|
||||
const text = await fetchRaw(`data/${name}`, SHARD_TTL);
|
||||
if (!text) continue;
|
||||
const hit = text.indexOf(needle);
|
||||
if (hit < 0) continue;
|
||||
@@ -87,7 +93,7 @@ export async function findOverride(appid: string): Promise<Record<string, unknow
|
||||
// TTL 0, unlike the shards: a stale override read would let the edit screen
|
||||
// build its next version on top of a superseded one and silently drop
|
||||
// somebody's `was` value.
|
||||
const text = await fetchText(`data/overrides/${appid}.json`, 0);
|
||||
const text = await fetchRaw(`data/overrides/${appid}.json`, 0);
|
||||
if (!text) return null;
|
||||
try {
|
||||
const doc = JSON.parse(text) as Record<string, unknown>;
|
||||
@@ -113,7 +119,7 @@ export async function genreCounts(): Promise<{ genre: string; count: number }[]>
|
||||
const re = /"genre"\s*:\s*"((?:[^"\\]|\\.)*)"/g;
|
||||
|
||||
for (const name of names) {
|
||||
const text = await fetchText(`data/${name}`, SHARD_TTL);
|
||||
const text = await fetchRaw(`data/${name}`, SHARD_TTL);
|
||||
if (!text) continue;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = re.exec(text)) !== null) {
|
||||
@@ -159,7 +165,7 @@ export async function listByGenre(
|
||||
let total = 0;
|
||||
|
||||
for (const name of names) {
|
||||
const text = await fetchText(`data/${name}`, SHARD_TTL);
|
||||
const text = await fetchRaw(`data/${name}`, SHARD_TTL);
|
||||
if (!text) continue;
|
||||
for (const line of text.split("\n")) {
|
||||
if (!line || (!line.includes(needle) && !line.includes(alt))) continue;
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
-- Admin coordination state. Apply BEFORE or AFTER deploying the Worker that
|
||||
-- uses it - the code tolerates both orders (see worker/lib/locks.ts):
|
||||
--
|
||||
-- npx wrangler d1 migrations apply f2p-admin --remote
|
||||
--
|
||||
-- Nothing here holds game data or visitor data.
|
||||
|
||||
-- ─────────────────────────────────────────────────────────────────────────────
|
||||
-- Leases, so a job that must not overlap itself cannot - across isolates, not
|
||||
-- just within one. The reconcile cron and the manual /admin button each fetch
|
||||
-- the whole ~6 MB dataset; before this, two isolates could run both at once.
|
||||
-- A lease that is never released (a crashed isolate) simply expires.
|
||||
-- ─────────────────────────────────────────────────────────────────────────────
|
||||
CREATE TABLE IF NOT EXISTS admin_locks (
|
||||
name TEXT PRIMARY KEY,
|
||||
owner TEXT NOT NULL,
|
||||
expires_at TEXT NOT NULL
|
||||
) STRICT;
|
||||
|
||||
-- ─────────────────────────────────────────────────────────────────────────────
|
||||
-- Small named values that must survive between cron ticks. Today: the dataset
|
||||
-- generation reconcile last swept against, so a tick with nothing new costs one
|
||||
-- ~600-byte index fetch instead of the whole dataset.
|
||||
-- ─────────────────────────────────────────────────────────────────────────────
|
||||
CREATE TABLE IF NOT EXISTS admin_state (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
) STRICT;
|
||||
@@ -0,0 +1,243 @@
|
||||
/**
|
||||
* The real admin handlers against the real schema (in-memory SQLite with the
|
||||
* migrations applied), with GitHub and the dataset faked.
|
||||
*
|
||||
* What matters most here is that a row which must not be decided is never
|
||||
* decided - and that the reviewer is told which rows were skipped and why,
|
||||
* instead of the old silent drop.
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { handleAdminApi } from "./admin";
|
||||
import {
|
||||
ADMIN,
|
||||
decisionOf,
|
||||
fakeDeps,
|
||||
insertDecision,
|
||||
insertRow,
|
||||
makeEnv,
|
||||
request,
|
||||
rowStatus,
|
||||
} from "../testing/fixtures";
|
||||
|
||||
async function call(env: Env, deps: ReturnType<typeof fakeDeps>, method: string, path: string, body?: unknown) {
|
||||
const { request: req, url } = request(method, path, body);
|
||||
const res = await handleAdminApi(req, url, env, ADMIN, deps);
|
||||
return { status: res.status, body: (await res.json()) as Record<string, any> };
|
||||
}
|
||||
|
||||
describe("GET /api/admin/queue", () => {
|
||||
it("marks rows whose game is already published, per game", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "a", appid: "10", status: "pending" });
|
||||
insertRow(env, { id: "b", appid: "10", status: "committed" });
|
||||
insertRow(env, { id: "c", appid: "20", status: "failed" });
|
||||
insertDecision(env, "20", "approved");
|
||||
insertRow(env, { id: "d", appid: "30", status: "pending" });
|
||||
|
||||
const { body } = await call(env, fakeDeps(), "GET", "/api/admin/queue?status=pending");
|
||||
const byId = Object.fromEntries(body.items.map((r: any) => [r.id, r.published]));
|
||||
expect(byId).toEqual({ a: true, d: false });
|
||||
|
||||
const failed = await call(env, fakeDeps(), "GET", "/api/admin/queue?status=failed");
|
||||
expect(failed.body.items[0].published).toBe(true);
|
||||
});
|
||||
|
||||
it("searches every status at once, but only with a query", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "p", appid: "4242", status: "pending" });
|
||||
insertRow(env, { id: "r", appid: "4242", status: "rejected" });
|
||||
insertRow(env, { id: "x", appid: "999", status: "pending" });
|
||||
|
||||
const { body } = await call(env, fakeDeps(), "GET", "/api/admin/queue?status=all&q=4242");
|
||||
expect(body.items.map((r: any) => r.id).sort()).toEqual(["p", "r"]);
|
||||
expect(body.total).toBe(2);
|
||||
|
||||
expect((await call(env, fakeDeps(), "GET", "/api/admin/queue?status=all")).status).toBe(400);
|
||||
expect((await call(env, fakeDeps(), "GET", "/api/admin/queue?status=bogus")).status).toBe(400);
|
||||
});
|
||||
});
|
||||
|
||||
describe("POST /api/admin/decide", () => {
|
||||
it("reports every skip reason and decides only what may be decided", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "ok", appid: "1", status: "pending" });
|
||||
insertRow(env, { id: "done", appid: "2", status: "committed" });
|
||||
insertRow(env, { id: "live", appid: "3", status: "pending" });
|
||||
insertDecision(env, "3", "approved");
|
||||
insertRow(env, { id: "dup-old", appid: "4", status: "failed", first_seen_at: "2026-09-01T00:00:00Z" });
|
||||
insertRow(env, { id: "dup-new", appid: "4", status: "pending", first_seen_at: "2026-09-05T00:00:00Z" });
|
||||
|
||||
const deps = fakeDeps();
|
||||
const { status, body } = await call(env, deps, "POST", "/api/admin/decide", {
|
||||
action: "reject",
|
||||
ids: ["ok", "done", "live", "dup-old", "dup-new", "ghost"],
|
||||
reason: "not free",
|
||||
});
|
||||
|
||||
expect(status).toBe(200);
|
||||
expect(body.decidedIds.sort()).toEqual(["dup-new", "ok"]);
|
||||
const reasons = Object.fromEntries(body.skipped.map((s: any) => [s.id, s.reason]));
|
||||
expect(reasons).toEqual({
|
||||
done: "not-decidable",
|
||||
live: "published",
|
||||
ghost: "not-found",
|
||||
"dup-old": "duplicate-in-request",
|
||||
});
|
||||
expect(rowStatus(env, "ok")).toBe("rejected");
|
||||
expect(rowStatus(env, "live")).toBe("pending");
|
||||
expect(decisionOf(env, "1")).toBe("rejected");
|
||||
// An approved decision is a publication record and is never replaced.
|
||||
expect(decisionOf(env, "3")).toBe("approved");
|
||||
});
|
||||
|
||||
it("refuses approve, reject and defer on a published game", async () => {
|
||||
for (const action of ["approve", "reject", "defer"]) {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "old", appid: "7", status: "committed" });
|
||||
insertRow(env, { id: "again", appid: "7", status: "failed" });
|
||||
const deps = fakeDeps();
|
||||
const { status, body } = await call(env, deps, "POST", "/api/admin/decide", { action, ids: ["again"] });
|
||||
expect(status, action).toBe(409);
|
||||
expect(body.skipped).toEqual([{ id: "again", appid: "7", reason: "published" }]);
|
||||
expect(rowStatus(env, "again")).toBe("failed");
|
||||
expect(deps.repo.commits).toEqual([]);
|
||||
}
|
||||
});
|
||||
|
||||
it("skips, instead of failing with a 500, a requeue that would collide with an open row", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "open", appid: "8", status: "pending" });
|
||||
insertRow(env, { id: "failed", appid: "8", status: "failed" });
|
||||
const { status, body } = await call(env, fakeDeps(), "POST", "/api/admin/decide", { action: "requeue", ids: ["failed"] });
|
||||
expect(status).toBe(409);
|
||||
expect(body.skipped[0].reason).toBe("open-row-exists");
|
||||
expect(rowStatus(env, "failed")).toBe("failed");
|
||||
});
|
||||
|
||||
it("reports a no-op move", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "p", appid: "9", status: "pending" });
|
||||
const { status, body } = await call(env, fakeDeps(), "POST", "/api/admin/decide", { action: "requeue", ids: ["p"] });
|
||||
expect(status).toBe(409);
|
||||
expect(body.skipped[0].reason).toBe("no-op");
|
||||
});
|
||||
|
||||
it("approves: commits the links, records the job, demotes a rival open row", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "rival", appid: "11", status: "pending" });
|
||||
insertRow(env, { id: "retry", appid: "11", status: "failed", first_seen_at: "2026-09-16T00:00:00Z" });
|
||||
const deps = fakeDeps();
|
||||
const { status, body } = await call(env, deps, "POST", "/api/admin/decide", {
|
||||
action: "approve",
|
||||
ids: ["retry"],
|
||||
reason: "checked by hand",
|
||||
});
|
||||
expect(status).toBe(200);
|
||||
expect(body.decidedIds).toEqual(["retry"]);
|
||||
expect(body.commit).toMatch(/^sha/);
|
||||
expect(rowStatus(env, "retry")).toBe("approved");
|
||||
expect(rowStatus(env, "rival")).toBe("failed");
|
||||
expect(deps.repo.appended).toEqual([["https://store.steampowered.com/app/11/"]]);
|
||||
|
||||
const job = env.sqlite.db.prepare("SELECT kind, status, commit_sha FROM commit_jobs").get() as any;
|
||||
expect(job).toMatchObject({ kind: "approve", status: "committed" });
|
||||
const audit = env.sqlite.db.prepare("SELECT detail_json FROM audit_log WHERE action='approve'").get() as any;
|
||||
expect(JSON.parse(audit.detail_json).reason).toBe("checked by hand");
|
||||
});
|
||||
|
||||
it("a commit that lands but whose queue update fails: job committed, honest 500", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "r", appid: "12", status: "pending" });
|
||||
// Make the queue batch fail after the commit, the way a D1 outage would.
|
||||
env.sqlite.db.exec(`CREATE TRIGGER boom BEFORE UPDATE ON ingest_queue BEGIN SELECT RAISE(ABORT, 'd1 down'); END;`);
|
||||
const { status, body } = await call(env, fakeDeps(), "POST", "/api/admin/decide", { action: "approve", ids: ["r"] });
|
||||
|
||||
expect(status).toBe(500);
|
||||
expect(body.error).toMatch(/landed but the queue could not be updated/);
|
||||
expect(body.error).toMatch(/approving them again is safe/);
|
||||
const job = env.sqlite.db.prepare("SELECT status FROM commit_jobs").get() as any;
|
||||
expect(job.status).toBe("committed");
|
||||
const desync = env.sqlite.db.prepare("SELECT COUNT(*) AS n FROM audit_log WHERE action='approve.desynced'").get() as any;
|
||||
expect(desync.n).toBe(1);
|
||||
});
|
||||
|
||||
it("a failed commit leaves the rows untouched and the job failed", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "r", appid: "13", status: "pending" });
|
||||
const { status } = await call(env, fakeDeps({ failCommit: "github down" }), "POST", "/api/admin/decide", {
|
||||
action: "approve",
|
||||
ids: ["r"],
|
||||
});
|
||||
expect(status).toBe(502);
|
||||
expect(rowStatus(env, "r")).toBe("pending");
|
||||
expect((env.sqlite.db.prepare("SELECT status FROM commit_jobs").get() as any).status).toBe("failed");
|
||||
});
|
||||
});
|
||||
|
||||
describe("POST /api/admin/reopen", () => {
|
||||
it("returns a rejected row to pending and forgets only the rejection", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "r", appid: "21", status: "rejected", decided_by: ADMIN.email, reject_reason: "dup" });
|
||||
insertDecision(env, "21", "rejected");
|
||||
const { status, body } = await call(env, fakeDeps(), "POST", "/api/admin/reopen", { id: "r" });
|
||||
expect(status).toBe(200);
|
||||
expect(body.appid).toBe("21");
|
||||
expect(rowStatus(env, "r")).toBe("pending");
|
||||
expect(decisionOf(env, "21")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("refuses when the game is published, and never deletes an approved decision", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "r", appid: "22", status: "rejected" });
|
||||
insertDecision(env, "22", "approved");
|
||||
const { status, body } = await call(env, fakeDeps(), "POST", "/api/admin/reopen", { id: "r" });
|
||||
expect(status).toBe(409);
|
||||
expect(body.reason).toBe("published");
|
||||
expect(decisionOf(env, "22")).toBe("approved");
|
||||
expect(rowStatus(env, "r")).toBe("rejected");
|
||||
});
|
||||
|
||||
it("refuses when another row for the game is open, and keeps the rejection", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "r", appid: "23", status: "rejected" });
|
||||
insertRow(env, { id: "open", appid: "23", status: "deferred" });
|
||||
insertDecision(env, "23", "rejected");
|
||||
const { status, body } = await call(env, fakeDeps(), "POST", "/api/admin/reopen", { id: "r" });
|
||||
expect(status).toBe(409);
|
||||
expect(body.reason).toBe("open-row-exists");
|
||||
expect(decisionOf(env, "23")).toBe("rejected");
|
||||
});
|
||||
|
||||
it("refuses a row that is not rejected, and 404s an unknown one", async () => {
|
||||
const env = makeEnv();
|
||||
insertRow(env, { id: "p", appid: "24", status: "pending" });
|
||||
expect((await call(env, fakeDeps(), "POST", "/api/admin/reopen", { id: "p" })).body.reason).toBe("not-rejected");
|
||||
expect((await call(env, fakeDeps(), "POST", "/api/admin/reopen", { id: "nope" })).status).toBe(404);
|
||||
});
|
||||
});
|
||||
|
||||
describe("jobs, audit and health", () => {
|
||||
it("pages with totals and filters", async () => {
|
||||
const env = makeEnv();
|
||||
for (let i = 0; i < 5; i++) {
|
||||
env.sqlite.db
|
||||
.prepare("INSERT INTO commit_jobs (id, kind, status, target_path, created_at) VALUES (?, ?, ?, 'x', ?)")
|
||||
.run(`j${i}`, i % 2 ? "override" : "approve", "committed", `2026-09-1${i}T00:00:00Z`);
|
||||
}
|
||||
const page = await call(env, fakeDeps(), "GET", "/api/admin/jobs?limit=2&offset=2");
|
||||
expect(page.body.total).toBe(5);
|
||||
expect(page.body.items.map((j: any) => j.id)).toEqual(["j2", "j1"]);
|
||||
const overrides = await call(env, fakeDeps(), "GET", "/api/admin/jobs?kind=override");
|
||||
expect(overrides.body.total).toBe(2);
|
||||
expect((await call(env, fakeDeps(), "GET", "/api/admin/jobs?status=weird")).status).toBe(400);
|
||||
});
|
||||
|
||||
it("health answers 503 WITH its body when GitHub is down", async () => {
|
||||
const env = makeEnv();
|
||||
const deps = fakeDeps({ gh: async () => new Response("no", { status: 401 }) });
|
||||
const { status, body } = await call(env, deps, "GET", "/api/admin/health");
|
||||
expect(status).toBe(503);
|
||||
expect(body).toMatchObject({ ok: false, d1: "ok", github: "401", retentionDays: 180 });
|
||||
expect(body.migrations).toEqual(["0001_init.sql", "0002_admin_state.sql"]);
|
||||
});
|
||||
});
|
||||
+387
-153
@@ -10,28 +10,24 @@
|
||||
import { jsonError, SECURITY_HEADERS, clampLimit, clampOffset } from "../lib/http";
|
||||
import { audit } from "../lib/audit";
|
||||
import type { AccessIdentity } from "../lib/access";
|
||||
import { gh } from "../lib/github-app";
|
||||
import { appendLinks, TEMP_INFO_PATH } from "../lib/git-commit";
|
||||
import { TEMP_INFO_PATH } from "../lib/git-commit";
|
||||
import { reconcileApproved } from "../lib/reconcile";
|
||||
import { readLease } from "../lib/locks";
|
||||
import { retentionDays } from "../lib/prune";
|
||||
import { defaultAdminDeps, type AdminDeps } from "../lib/deps";
|
||||
import { handleEditApi } from "./edit";
|
||||
|
||||
/**
|
||||
* Per-request cap on a bulk decision. Bounds three things at once: the D1
|
||||
* batch, the size of a single commit, and how much one mis-click can do.
|
||||
*/
|
||||
const MAX_DECIDE = 100;
|
||||
|
||||
/**
|
||||
* Statuses a row may be decided FROM. A committed row is finished.
|
||||
*
|
||||
* Enforced TWICE on purpose: once when the rows are read, and again in the
|
||||
* WHERE clause of every UPDATE. The read-then-write gap is real — an admin
|
||||
* page held open across a reconcile, or two reviewers acting at once — and
|
||||
* without the second check a stale page could reject a game whose approval had
|
||||
* already been committed to Git, or one the pipeline had already published.
|
||||
* Every branch below carries the guard; do not add one that skips it.
|
||||
*/
|
||||
const DECIDABLE = ["pending", "deferred", "failed"];
|
||||
import {
|
||||
DECIDABLE,
|
||||
DECIDE_ACTIONS,
|
||||
MAX_DECIDE,
|
||||
OPEN_STATUSES,
|
||||
QUEUE_STATUSES,
|
||||
isDecidable,
|
||||
isNoOp,
|
||||
sqlList,
|
||||
type DecideAction,
|
||||
type SkippedRow,
|
||||
} from "../../shared/queue-rules";
|
||||
|
||||
function json(body: unknown, status = 200): Response {
|
||||
return new Response(JSON.stringify(body), {
|
||||
@@ -45,19 +41,52 @@ function json(body: unknown, status = 200): Response {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the game a row names is already published, as SQL over `alias`.
|
||||
*
|
||||
* Per GAME, not per row: a committed row for the appid (reconcile saw it in
|
||||
* data/), or an approved decision (the durable record of the same fact, which
|
||||
* outlives queue history). Every decide UPDATE repeats this in its WHERE
|
||||
* clause, so a page held open across a reconcile cannot act on a game that
|
||||
* went live in the meantime.
|
||||
*/
|
||||
function publishedSql(alias: string): string {
|
||||
return `(EXISTS (SELECT 1 FROM ingest_decisions pd WHERE pd.appid = ${alias}.appid AND pd.decision = 'approved')
|
||||
OR EXISTS (SELECT 1 FROM ingest_queue pc WHERE pc.appid = ${alias}.appid AND pc.status = 'committed'))`;
|
||||
}
|
||||
|
||||
/** Another OPEN row for the same appid (the rows uq_queue_open_appid counts). */
|
||||
function openSiblingSql(alias: string): string {
|
||||
return `EXISTS (SELECT 1 FROM ingest_queue os WHERE os.appid = ${alias}.appid AND os.id <> ${alias}.id
|
||||
AND os.status IN (${sqlList(OPEN_STATUSES)}))`;
|
||||
}
|
||||
|
||||
const DECIDABLE_SQL = sqlList(DECIDABLE);
|
||||
|
||||
interface DecideRow {
|
||||
id: string;
|
||||
appid: string;
|
||||
link: string;
|
||||
name: string;
|
||||
status: string;
|
||||
first_seen_at: string;
|
||||
published: number;
|
||||
open_sibling: number;
|
||||
}
|
||||
|
||||
export async function handleAdminApi(
|
||||
request: Request,
|
||||
url: URL,
|
||||
env: Env,
|
||||
who: AccessIdentity,
|
||||
deps: AdminDeps = defaultAdminDeps,
|
||||
): Promise<Response> {
|
||||
const route = url.pathname.slice("/api/admin/".length);
|
||||
|
||||
// Corrections to already-published games live in their own module; they
|
||||
// write data/overrides/, never data/ and never the queue.
|
||||
if (route === "game" || route === "edit" || route === "genres" || route === "by-genre") {
|
||||
return handleEditApi(request, url, env, who);
|
||||
return handleEditApi(request, url, env, who, deps);
|
||||
}
|
||||
|
||||
// Who am I? Confirms the Access -> Worker identity chain end to end.
|
||||
@@ -65,10 +94,11 @@ export async function handleAdminApi(
|
||||
return json({ email: who.email, isServiceToken: who.isServiceToken });
|
||||
}
|
||||
|
||||
// Status of the moving parts. Deliberately reports each dependency
|
||||
// separately so a failure names itself instead of surfacing as "broken".
|
||||
// Status of the moving parts. Always a JSON body, 200 or 503: the old
|
||||
// response omitted `error` on a 503, so the page threw "HTTP 503" and never
|
||||
// drew the table - the health view was blank exactly when it was needed.
|
||||
if (route === "health" && request.method === "GET") {
|
||||
const out: Record<string, unknown> = { ok: true, actor: who.email };
|
||||
const out: Record<string, unknown> = { ok: true, actor: who.email, retentionDays: retentionDays(env) };
|
||||
|
||||
try {
|
||||
const row = await env.DB.prepare(
|
||||
@@ -78,25 +108,35 @@ export async function handleAdminApi(
|
||||
out.d1 = "ok";
|
||||
} catch (err) {
|
||||
out.ok = false;
|
||||
// "error", not the message. This is behind Access so the exposure is
|
||||
// small, but a D1 driver message can carry SQL fragments and column
|
||||
// names, and there is no reason for a health endpoint to be the place
|
||||
// they surface. The detail goes to Workers Logs instead.
|
||||
// "error", not the message: a D1 driver message can carry SQL fragments.
|
||||
// The detail goes to Workers Logs instead.
|
||||
out.d1 = "error";
|
||||
console.error("health: d1", err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
|
||||
// Applied migrations, so a Worker deployed ahead of its migration is
|
||||
// visible here rather than only as degraded behaviour.
|
||||
try {
|
||||
const rows = await env.DB.prepare("SELECT name FROM d1_migrations ORDER BY id").all<{ name: string }>();
|
||||
out.migrations = (rows.results ?? []).map((r) => r.name);
|
||||
} catch {
|
||||
out.migrations = null;
|
||||
}
|
||||
try {
|
||||
out.lock = await readLease(env.DB, "reconcile");
|
||||
} catch {
|
||||
out.lock = null;
|
||||
}
|
||||
|
||||
// Verifies the App private key, the installation, and its permissions in
|
||||
// one call — this is the check that catches a mis-scoped App.
|
||||
try {
|
||||
const res = await gh(env, "/installation/repositories?per_page=1");
|
||||
const res = await deps.gh(env, "/installation/repositories?per_page=1");
|
||||
out.github = res.ok ? "ok" : `${res.status}`;
|
||||
if (!res.ok) out.ok = false;
|
||||
} catch (err) {
|
||||
out.ok = false;
|
||||
// Same reasoning, and more pressing: github-app.ts embeds GitHub's raw
|
||||
// response body in this message, which on a mis-scoped App includes the
|
||||
// installation details.
|
||||
// github-app.ts embeds GitHub's raw response body in this message.
|
||||
out.github = "error";
|
||||
console.error("health: github", err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
@@ -124,32 +164,43 @@ export async function handleAdminApi(
|
||||
// The review queue.
|
||||
if (route === "queue" && request.method === "GET") {
|
||||
const status = url.searchParams.get("status") ?? "pending";
|
||||
if (!/^[a-z]{1,12}$/.test(status)) return jsonError(400, "bad status");
|
||||
const limit = clampLimit(url.searchParams.get("limit"), 50, 200);
|
||||
const offset = clampOffset(url.searchParams.get("offset"));
|
||||
|
||||
// Search is SERVER-side. The page used to filter only the 60 rows it had
|
||||
// loaded, so typing an appid that sat on page 3 reported "Nothing on this
|
||||
// page matches that filter" - indistinguishable from "not in the queue".
|
||||
const q = (url.searchParams.get("q") ?? "").trim().slice(0, 80);
|
||||
// status=all searches every tab at once - but only WITH a query, so it
|
||||
// cannot become a way to page through the whole table in one listing.
|
||||
const all = status === "all";
|
||||
if (all ? !q : !(QUEUE_STATUSES as readonly string[]).includes(status)) {
|
||||
return jsonError(400, all ? "status=all needs a search query" : "bad status");
|
||||
}
|
||||
const limit = clampLimit(url.searchParams.get("limit"), 50, 200);
|
||||
const offset = clampOffset(url.searchParams.get("offset"));
|
||||
|
||||
// LIKE with an escaped pattern, so a reviewer pasting a name containing %
|
||||
// or _ searches for those characters rather than wildcards.
|
||||
const like = q ? `%${q.replace(/[\\%_]/g, (c) => "\\" + c)}%` : null;
|
||||
const where = like
|
||||
? "status = ? AND (name LIKE ? ESCAPE '\\' OR appid LIKE ? ESCAPE '\\')"
|
||||
: "status = ?";
|
||||
const whereArgs: unknown[] = like ? [status, like, like] : [status];
|
||||
const clauses: string[] = [];
|
||||
const args: unknown[] = [];
|
||||
if (!all) {
|
||||
clauses.push("q.status = ?");
|
||||
args.push(status);
|
||||
}
|
||||
if (like) {
|
||||
clauses.push("(q.name LIKE ? ESCAPE '\\' OR q.appid LIKE ? ESCAPE '\\')");
|
||||
args.push(like, like);
|
||||
}
|
||||
const where = clauses.join(" AND ");
|
||||
|
||||
// ORDER BY is a closed set mapped to fixed SQL, never interpolated from
|
||||
// the caller. reviews_pct and current_players_num have been selected since
|
||||
// the schema was written - the comment there says they exist "so the
|
||||
// review screen can sort" - but nothing ever offered the sort.
|
||||
// the caller.
|
||||
const SORTS: Record<string, string> = {
|
||||
newest: "first_seen_at DESC, id",
|
||||
oldest: "first_seen_at ASC, id",
|
||||
players: "current_players_num DESC NULLS LAST, id",
|
||||
reviews: "reviews_pct DESC NULLS LAST, id",
|
||||
name: "name COLLATE NOCASE ASC, id",
|
||||
newest: "q.first_seen_at DESC, q.id",
|
||||
oldest: "q.first_seen_at ASC, q.id",
|
||||
players: "q.current_players_num DESC NULLS LAST, q.id",
|
||||
reviews: "q.reviews_pct DESC NULLS LAST, q.id",
|
||||
name: "q.name COLLATE NOCASE ASC, q.id",
|
||||
decided: "q.decided_at DESC NULLS LAST, q.id",
|
||||
};
|
||||
const sortKey = url.searchParams.get("sort") ?? "newest";
|
||||
const orderBy = SORTS[sortKey] ?? SORTS.newest;
|
||||
@@ -158,23 +209,22 @@ export async function handleAdminApi(
|
||||
// -- so every ordering above breaks the tie on id, or rows would swap
|
||||
// places between pages and a game could be shown twice or never.
|
||||
const rows = await env.DB.prepare(
|
||||
`SELECT id, appid, link, name, header_image, release_date, app_type,
|
||||
is_free, health_status, reviews_raw, reviews_pct,
|
||||
current_players_raw, current_players_num, source, status,
|
||||
decided_by, decided_at, reject_reason,
|
||||
first_seen_at, last_seen_at, seen_count
|
||||
FROM ingest_queue
|
||||
`SELECT q.id, q.appid, q.link, q.name, q.header_image, q.release_date, q.app_type,
|
||||
q.is_free, q.health_status, q.reviews_raw, q.reviews_pct,
|
||||
q.current_players_raw, q.current_players_num, q.source, q.status,
|
||||
q.decided_by, q.decided_at, q.reject_reason,
|
||||
q.first_seen_at, q.last_seen_at, q.seen_count,
|
||||
${publishedSql("q")} AS published
|
||||
FROM ingest_queue q
|
||||
WHERE ${where}
|
||||
ORDER BY ${orderBy}
|
||||
LIMIT ? OFFSET ?`,
|
||||
)
|
||||
.bind(...whereArgs, limit, offset)
|
||||
.all();
|
||||
.bind(...args, limit, offset)
|
||||
.all<Record<string, unknown>>();
|
||||
|
||||
const total = await env.DB.prepare(
|
||||
`SELECT COUNT(*) AS n FROM ingest_queue WHERE ${where}`,
|
||||
)
|
||||
.bind(...whereArgs)
|
||||
const total = await env.DB.prepare(`SELECT COUNT(*) AS n FROM ingest_queue q WHERE ${where}`)
|
||||
.bind(...args)
|
||||
.first<{ n: number }>();
|
||||
|
||||
return json({
|
||||
@@ -185,59 +235,116 @@ export async function handleAdminApi(
|
||||
sort: sortKey in SORTS ? sortKey : "newest",
|
||||
total: total?.n ?? 0,
|
||||
count: rows.results?.length ?? 0,
|
||||
items: rows.results ?? [],
|
||||
// So the page stops hardcoding 100 as a client-side literal.
|
||||
items: (rows.results ?? []).map((r) => ({ ...r, published: Boolean(r.published) })),
|
||||
maxDecide: MAX_DECIDE,
|
||||
});
|
||||
}
|
||||
|
||||
// The full candidate as it arrived, for one row. payload_json is stored
|
||||
// verbatim on every insert and was never surfaced anywhere, so the only way
|
||||
// to see why a game was proposed was `wrangler d1 execute`.
|
||||
// One row in full: the candidate as it arrived, its decision, and every
|
||||
// other row the queue holds for the same game.
|
||||
if (route === "candidate" && request.method === "GET") {
|
||||
const id = (url.searchParams.get("id") ?? "").slice(0, 64);
|
||||
if (!id) return jsonError(400, "id is required");
|
||||
const row = await env.DB.prepare(
|
||||
`SELECT id, appid, name, payload_json, first_seen_at, last_seen_at, seen_count,
|
||||
health_status, app_type, is_free, source
|
||||
FROM ingest_queue WHERE id = ?`,
|
||||
`SELECT q.id, q.appid, q.link, q.name, q.header_image, q.status, q.reject_reason,
|
||||
q.decided_by, q.decided_at, q.payload_json, q.first_seen_at, q.last_seen_at,
|
||||
q.seen_count, q.health_status, q.app_type, q.is_free, q.source,
|
||||
${publishedSql("q")} AS published
|
||||
FROM ingest_queue q WHERE q.id = ?`,
|
||||
)
|
||||
.bind(id)
|
||||
.first();
|
||||
.first<Record<string, unknown> & { appid: string }>();
|
||||
if (!row) return jsonError(404, "not found");
|
||||
return json(row);
|
||||
|
||||
const [decision, siblings] = await Promise.all([
|
||||
env.DB.prepare(
|
||||
"SELECT decision, reason, decided_by, decided_at FROM ingest_decisions WHERE appid = ?",
|
||||
)
|
||||
.bind(row.appid)
|
||||
.first(),
|
||||
env.DB.prepare(
|
||||
`SELECT id, status, reject_reason, decided_by, decided_at, first_seen_at
|
||||
FROM ingest_queue WHERE appid = ? AND id <> ? ORDER BY first_seen_at DESC LIMIT 20`,
|
||||
)
|
||||
.bind(row.appid, id)
|
||||
.all(),
|
||||
]);
|
||||
return json({
|
||||
...row,
|
||||
published: Boolean(row.published),
|
||||
decision: decision ?? null,
|
||||
siblings: siblings.results ?? [],
|
||||
});
|
||||
}
|
||||
|
||||
// Recent admin actions.
|
||||
// Recent admin actions, paged and filterable.
|
||||
if (route === "audit" && request.method === "GET") {
|
||||
const limit = clampLimit(url.searchParams.get("limit"), 50, 200);
|
||||
const rows = await env.DB.prepare(
|
||||
`SELECT id, actor, action, target, detail_json, created_at
|
||||
FROM audit_log ORDER BY id DESC LIMIT ?`,
|
||||
)
|
||||
.bind(limit)
|
||||
.all();
|
||||
return json({ items: rows.results ?? [] });
|
||||
const offset = clampOffset(url.searchParams.get("offset"));
|
||||
const clauses: string[] = [];
|
||||
const args: unknown[] = [];
|
||||
const action = url.searchParams.get("action") ?? "";
|
||||
if (action) {
|
||||
if (!/^[a-z][a-z.]{0,39}$/.test(action)) return jsonError(400, "bad action");
|
||||
clauses.push("action = ?");
|
||||
args.push(action);
|
||||
}
|
||||
const actor = (url.searchParams.get("actor") ?? "").slice(0, 200);
|
||||
if (actor) {
|
||||
clauses.push("actor = ?");
|
||||
args.push(actor);
|
||||
}
|
||||
const where = clauses.length ? `WHERE ${clauses.join(" AND ")}` : "";
|
||||
const [rows, total] = await Promise.all([
|
||||
env.DB.prepare(
|
||||
`SELECT id, actor, action, target, detail_json, created_at
|
||||
FROM audit_log ${where} ORDER BY id DESC LIMIT ? OFFSET ?`,
|
||||
)
|
||||
.bind(...args, limit, offset)
|
||||
.all(),
|
||||
env.DB.prepare(`SELECT COUNT(*) AS n FROM audit_log ${where}`)
|
||||
.bind(...args)
|
||||
.first<{ n: number }>(),
|
||||
]);
|
||||
return json({ items: rows.results ?? [], total: total?.n ?? 0, limit, offset });
|
||||
}
|
||||
|
||||
// commit_jobs, which had no UI at all. /api/admin/stats already returned
|
||||
// these counts and the page discarded them, so the one table that records a
|
||||
// commit attempt BEFORE it is made - the evidence that a commit died
|
||||
// mid-flight - was only readable through `wrangler d1 execute`.
|
||||
// commit_jobs: the record of every Worker commit attempt, written BEFORE the
|
||||
// attempt - the evidence that a commit died mid-flight.
|
||||
if (route === "jobs" && request.method === "GET") {
|
||||
const limit = clampLimit(url.searchParams.get("limit"), 50, 200);
|
||||
const rows = await env.DB.prepare(
|
||||
`SELECT id, kind, status, target_path, commit_sha, error,
|
||||
requested_by, created_at, finished_at
|
||||
FROM commit_jobs ORDER BY created_at DESC LIMIT ?`,
|
||||
)
|
||||
.bind(limit)
|
||||
.all();
|
||||
return json({ items: rows.results ?? [] });
|
||||
const offset = clampOffset(url.searchParams.get("offset"));
|
||||
const clauses: string[] = [];
|
||||
const args: unknown[] = [];
|
||||
const status = url.searchParams.get("status") ?? "";
|
||||
if (status) {
|
||||
if (!["pending", "committed", "conflict", "failed"].includes(status)) return jsonError(400, "bad status");
|
||||
clauses.push("status = ?");
|
||||
args.push(status);
|
||||
}
|
||||
const kind = url.searchParams.get("kind") ?? "";
|
||||
if (kind) {
|
||||
if (!/^[a-z][a-z.]{0,31}$/.test(kind)) return jsonError(400, "bad kind");
|
||||
clauses.push("kind = ?");
|
||||
args.push(kind);
|
||||
}
|
||||
const where = clauses.length ? `WHERE ${clauses.join(" AND ")}` : "";
|
||||
const [rows, total] = await Promise.all([
|
||||
env.DB.prepare(
|
||||
`SELECT id, kind, status, target_path, commit_sha, error, requested_by, created_at, finished_at
|
||||
FROM commit_jobs ${where} ORDER BY created_at DESC, id LIMIT ? OFFSET ?`,
|
||||
)
|
||||
.bind(...args, limit, offset)
|
||||
.all(),
|
||||
env.DB.prepare(`SELECT COUNT(*) AS n FROM commit_jobs ${where}`)
|
||||
.bind(...args)
|
||||
.first<{ n: number }>(),
|
||||
]);
|
||||
return json({ items: rows.results ?? [], total: total?.n ?? 0, limit, offset });
|
||||
}
|
||||
|
||||
if (route === "ping" && request.method === "POST") {
|
||||
await audit(env, who.email, "ping", null, { at: new Date().toISOString() });
|
||||
await audit(env, who.email, "ping", null, { at: deps.now().toISOString() });
|
||||
return json({ ok: true });
|
||||
}
|
||||
|
||||
@@ -249,6 +356,10 @@ export async function handleAdminApi(
|
||||
// approve only REQUESTS publication. It commits to scripts/temp_info.jsonl
|
||||
// and stops; ingest_new.py decides whether the game is real, and
|
||||
// lib/reconcile.ts records the outcome once it can observe it.
|
||||
//
|
||||
// Every named row is either decided or reported back in `skipped` with a
|
||||
// reason. The old handler dropped non-decidable rows silently, so a mixed
|
||||
// selection reported "approve: 3 rows" for five selected.
|
||||
if (route === "decide" && request.method === "POST") {
|
||||
let body: { ids?: unknown; action?: unknown; reason?: unknown };
|
||||
try {
|
||||
@@ -257,33 +368,81 @@ export async function handleAdminApi(
|
||||
return jsonError(400, "invalid json");
|
||||
}
|
||||
|
||||
const action = typeof body.action === "string" ? body.action : "";
|
||||
if (!["approve", "reject", "defer", "requeue"].includes(action)) {
|
||||
const action = (typeof body.action === "string" ? body.action : "") as DecideAction;
|
||||
if (!(DECIDE_ACTIONS as readonly string[]).includes(action)) {
|
||||
return jsonError(400, "action must be approve, reject, defer or requeue");
|
||||
}
|
||||
|
||||
const ids = Array.isArray(body.ids)
|
||||
? body.ids.filter((i): i is string => typeof i === "string" && i.length <= 64)
|
||||
? [...new Set(body.ids.filter((i): i is string => typeof i === "string" && i.length > 0 && i.length <= 64))]
|
||||
: [];
|
||||
if (!ids.length) return jsonError(400, "ids must be a non-empty array");
|
||||
if (ids.length > MAX_DECIDE) return jsonError(400, "at most " + MAX_DECIDE + " ids");
|
||||
|
||||
const reason = typeof body.reason === "string" ? body.reason.slice(0, 300) : "";
|
||||
const reason = typeof body.reason === "string" ? body.reason.trim().slice(0, 300) : "";
|
||||
|
||||
// Re-read server-side. The client sends ids and nothing else: a request
|
||||
// that could name its own link would let this endpoint queue an arbitrary
|
||||
// URL into a repository file.
|
||||
const placeholders = ids.map(() => "?").join(",");
|
||||
const found = await env.DB.prepare(
|
||||
"SELECT id, appid, link, name, status FROM ingest_queue WHERE id IN (" + placeholders + ")",
|
||||
`SELECT q.id, q.appid, q.link, q.name, q.status, q.first_seen_at,
|
||||
${publishedSql("q")} AS published,
|
||||
${openSiblingSql("q")} AS open_sibling
|
||||
FROM ingest_queue q
|
||||
WHERE q.id IN (SELECT value FROM json_each(?))`,
|
||||
)
|
||||
.bind(...ids)
|
||||
.all<{ id: string; appid: string; link: string; name: string; status: string }>();
|
||||
.bind(JSON.stringify(ids))
|
||||
.all<DecideRow>();
|
||||
|
||||
const rows = (found.results ?? []).filter((r) => DECIDABLE.includes(r.status));
|
||||
if (!rows.length) return jsonError(409, "no decidable rows in that selection");
|
||||
const byId = new Map((found.results ?? []).map((r) => [r.id, r]));
|
||||
const skipped: SkippedRow[] = [];
|
||||
const candidates: DecideRow[] = [];
|
||||
for (const id of ids) {
|
||||
const row = byId.get(id);
|
||||
if (!row) skipped.push({ id, appid: null, reason: "not-found" });
|
||||
else if (!isDecidable(row.status)) skipped.push({ id, appid: row.appid, reason: "not-decidable" });
|
||||
else if (row.published) skipped.push({ id, appid: row.appid, reason: "published" });
|
||||
else if (isNoOp(action, row.status)) skipped.push({ id, appid: row.appid, reason: "no-op" });
|
||||
// Moving a failed row back into an open state while the appid already
|
||||
// has an open row would violate uq_queue_open_appid - which used to
|
||||
// throw inside the batch and fail the whole request with a 500.
|
||||
else if ((action === "defer" || action === "requeue") && row.open_sibling) {
|
||||
skipped.push({ id, appid: row.appid, reason: "open-row-exists" });
|
||||
} else candidates.push(row);
|
||||
}
|
||||
|
||||
const now = new Date().toISOString();
|
||||
// One row per game: the newest wins, the rest are reported.
|
||||
candidates.sort((a, b) => (a.first_seen_at < b.first_seen_at ? 1 : a.first_seen_at > b.first_seen_at ? -1 : 0));
|
||||
const rows: DecideRow[] = [];
|
||||
const appids = new Set<string>();
|
||||
for (const row of candidates) {
|
||||
if (appids.has(row.appid)) skipped.push({ id: row.id, appid: row.appid, reason: "duplicate-in-request" });
|
||||
else {
|
||||
appids.add(row.appid);
|
||||
rows.push(row);
|
||||
}
|
||||
}
|
||||
|
||||
if (!rows.length) {
|
||||
return new Response(JSON.stringify({ error: "nothing in that selection can be decided", skipped }), {
|
||||
status: 409,
|
||||
headers: { "Content-Type": "application/json; charset=utf-8", "Cache-Control": "no-store", ...SECURITY_HEADERS },
|
||||
});
|
||||
}
|
||||
|
||||
const now = deps.now().toISOString();
|
||||
// Rows that were decidable when read but changed before the write.
|
||||
const settle = (results: D1Result[], perRow: number, offsetInBatch: number) => {
|
||||
const decidedIds: string[] = [];
|
||||
rows.forEach((row, i) => {
|
||||
const res = results[offsetInBatch + i * perRow + (perRow - 1)];
|
||||
if ((res?.meta?.changes ?? 0) > 0) decidedIds.push(row.id);
|
||||
else skipped.push({ id: row.id, appid: row.appid, reason: "changed-concurrently" });
|
||||
});
|
||||
return decidedIds;
|
||||
};
|
||||
|
||||
const notPublished = `AND NOT ${publishedSql("ingest_queue")}`;
|
||||
|
||||
if (action === "approve") {
|
||||
const jobId = crypto.randomUUID();
|
||||
@@ -298,60 +457,70 @@ export async function handleAdminApi(
|
||||
|
||||
let result;
|
||||
try {
|
||||
result = await appendLinks(env, rows.map((r) => r.link), who.email);
|
||||
result = await deps.appendLinks(env, rows.map((r) => r.link), who.email, reason);
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
await env.DB.prepare(
|
||||
"UPDATE commit_jobs SET status='failed', error=?, finished_at=? WHERE id=?",
|
||||
)
|
||||
.bind(message.slice(0, 500), new Date().toISOString(), jobId)
|
||||
.bind(message.slice(0, 500), deps.now().toISOString(), jobId)
|
||||
.run();
|
||||
await audit(env, who.email, "approve.failed", jobId, { message, count: rows.length });
|
||||
await audit(env, who.email, "approve.failed", jobId, { message, count: rows.length, reason });
|
||||
// Rows are deliberately left untouched. A row marked approved with no
|
||||
// commit behind it is invisible to the queue AND to the reconciler.
|
||||
return jsonError(502, "commit failed: " + message);
|
||||
}
|
||||
|
||||
// The job's outcome is recorded ON ITS OWN, before the queue batch. It
|
||||
// used to share that batch, so when the batch failed the job was rolled
|
||||
// back to 'pending' forever despite a commit having landed.
|
||||
try {
|
||||
await env.DB.batch([
|
||||
env.DB.prepare(
|
||||
"UPDATE commit_jobs SET status='committed', commit_sha=?, finished_at=? WHERE id=?",
|
||||
).bind(result.sha, new Date().toISOString(), jobId),
|
||||
...rows.flatMap((r) => [
|
||||
await env.DB.prepare(
|
||||
"UPDATE commit_jobs SET status='committed', commit_sha=?, error=?, finished_at=? WHERE id=?",
|
||||
)
|
||||
.bind(result.sha, result.sha ? null : "nothing to commit: every link was already queued", deps.now().toISOString(), jobId)
|
||||
.run();
|
||||
} catch (err) {
|
||||
console.error("approve: commit_jobs update failed", err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
|
||||
let decidedIds: string[];
|
||||
try {
|
||||
const results = await env.DB.batch(
|
||||
rows.flatMap((r) => [
|
||||
// Stand any RIVAL open row for this appid down FIRST.
|
||||
//
|
||||
// uq_queue_open_appid covers ('pending','deferred','approved') but
|
||||
// not 'failed', so a row reconcile marked failed can coexist with a
|
||||
// fresh pending row for the same appid. Promoting this one into
|
||||
// 'approved' while the other is still open violates the index and
|
||||
// throws - and by then the commit has already landed, so the batch
|
||||
// would abort leaving rows pending that are queued in Git. D1 runs
|
||||
// a batch sequentially in one transaction, so demoting the rival
|
||||
// not 'failed', so a failed row can coexist with a fresh pending
|
||||
// row for the same appid. Promoting this one into 'approved' while
|
||||
// the other is still open violates the index and throws. D1 runs a
|
||||
// batch sequentially in one transaction, so demoting the rival
|
||||
// before the promotion keeps the index satisfied at every step.
|
||||
env.DB.prepare(
|
||||
`UPDATE ingest_queue
|
||||
SET status='failed', reject_reason='superseded by a duplicate approval'
|
||||
WHERE appid=? AND id<>? AND status IN ('pending','deferred','approved')`,
|
||||
WHERE appid=? AND id<>? AND status IN (${sqlList(OPEN_STATUSES)})`,
|
||||
).bind(r.appid, r.id),
|
||||
env.DB.prepare(
|
||||
`UPDATE ingest_queue
|
||||
SET status='approved', decided_by=?, decided_at=?, reject_reason=NULL
|
||||
WHERE id=? AND status IN ('pending','deferred','failed')`,
|
||||
WHERE id=? AND status IN (${DECIDABLE_SQL}) ${notPublished}`,
|
||||
).bind(who.email, now, r.id),
|
||||
]),
|
||||
]);
|
||||
);
|
||||
decidedIds = settle(results, 2, 0);
|
||||
} catch (err) {
|
||||
// The commit LANDED; only the bookkeeping failed. Say so loudly rather
|
||||
// than reporting a clean failure: the links are in Git and the pipeline
|
||||
// will publish them, but these rows still read as undecided.
|
||||
// The commit LANDED; only the bookkeeping failed. Say exactly that,
|
||||
// and what happens next.
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
await audit(env, who.email, "approve.desynced", jobId, {
|
||||
message, sha: result.sha, appids: rows.map((r) => r.appid),
|
||||
message, sha: result.sha, appids: rows.map((r) => r.appid), reason,
|
||||
});
|
||||
return jsonError(
|
||||
500,
|
||||
"commit " + (result.sha ?? "(none)") + " landed but the queue was not updated: " +
|
||||
message + " - reconcile will still promote these rows once they appear in data/",
|
||||
`commit ${result.sha ?? "(none - already queued)"} landed but the queue could not be updated (${message}). ` +
|
||||
"The rows are unchanged, so approving them again is safe and will not queue them twice. " +
|
||||
"Once the pipeline publishes the games, reconcile marks these rows committed on its own.",
|
||||
);
|
||||
}
|
||||
|
||||
@@ -359,12 +528,16 @@ export async function handleAdminApi(
|
||||
appids: rows.map((r) => r.appid),
|
||||
sha: result.sha,
|
||||
appended: result.appended,
|
||||
skipped: result.skipped.length,
|
||||
already_queued: result.skipped.length,
|
||||
skipped: skipped.length,
|
||||
reason,
|
||||
});
|
||||
|
||||
return json({
|
||||
action,
|
||||
decided: rows.length,
|
||||
decided: decidedIds.length,
|
||||
decidedIds,
|
||||
skipped,
|
||||
// null when every link was already queued - the UI reads this to avoid
|
||||
// announcing a commit that was never made.
|
||||
commit: result.sha,
|
||||
@@ -374,62 +547,123 @@ export async function handleAdminApi(
|
||||
}
|
||||
|
||||
if (action === "reject") {
|
||||
await env.DB.batch(
|
||||
const results = await env.DB.batch(
|
||||
rows.flatMap((r) => [
|
||||
env.DB.prepare(
|
||||
`UPDATE ingest_queue
|
||||
SET status='rejected', decided_by=?, decided_at=?, reject_reason=?
|
||||
WHERE id=? AND status IN ('pending','deferred','failed')`,
|
||||
).bind(who.email, now, reason || null, r.id),
|
||||
// Durable, so tomorrow's sweep does not offer it again. The partial
|
||||
// unique index only covers OPEN rows, so this table is what makes a
|
||||
// rejection stick.
|
||||
//
|
||||
// The trailing WHERE is what stops a reject DOWNGRADING an approval.
|
||||
// reconcile.ts writes decision='approved' only after observing the
|
||||
// appid in data/ - i.e. the game is published, which is a fact rather
|
||||
// than a preference. Its insert uses DO NOTHING; this one used a bare
|
||||
// DO UPDATE, so the two disagreed about who may overwrite whom, and a
|
||||
// reject landing on a re-queued row for an already-published appid
|
||||
// silently replaced that record and its attribution. Refuse instead:
|
||||
// to actually un-publish a game, remove it from data/.
|
||||
// rejection stick. Written only for a row that is really being
|
||||
// rejected (the same guards as the UPDATE below), and never over an
|
||||
// 'approved' decision: that one records a publication, which is a
|
||||
// fact rather than a preference.
|
||||
env.DB.prepare(
|
||||
`INSERT INTO ingest_decisions (appid, decision, reason, decided_by, decided_at)
|
||||
VALUES (?, 'rejected', ?, ?, ?)
|
||||
SELECT q.appid, 'rejected', ?, ?, ? FROM ingest_queue q
|
||||
WHERE q.id = ? AND q.status IN (${DECIDABLE_SQL}) AND NOT ${publishedSql("q")}
|
||||
ON CONFLICT(appid) DO UPDATE SET
|
||||
decision=excluded.decision, reason=excluded.reason,
|
||||
decided_by=excluded.decided_by, decided_at=excluded.decided_at
|
||||
WHERE ingest_decisions.decision <> 'approved'`,
|
||||
).bind(r.appid, reason || null, who.email, now),
|
||||
).bind(reason || null, who.email, now, r.id),
|
||||
env.DB.prepare(
|
||||
`UPDATE ingest_queue
|
||||
SET status='rejected', decided_by=?, decided_at=?, reject_reason=?
|
||||
WHERE id=? AND status IN (${DECIDABLE_SQL}) ${notPublished}`,
|
||||
).bind(who.email, now, reason || null, r.id),
|
||||
]),
|
||||
);
|
||||
const decidedIds = settle(results, 2, 0);
|
||||
await audit(env, who.email, "reject", null, {
|
||||
appids: rows.map((r) => r.appid),
|
||||
appids: rows.filter((r) => decidedIds.includes(r.id)).map((r) => r.appid),
|
||||
reason,
|
||||
skipped: skipped.length,
|
||||
});
|
||||
return json({ action, decided: rows.length });
|
||||
return json({ action, decided: decidedIds.length, decidedIds, skipped });
|
||||
}
|
||||
|
||||
// defer / requeue both just move a row between OPEN states, so neither
|
||||
// touches ingest_decisions - the appid stays blocked by the partial unique
|
||||
// index and stays known to /api/ingest/known.
|
||||
const target = action === "defer" ? "deferred" : "pending";
|
||||
await env.DB.batch(
|
||||
const results = await env.DB.batch(
|
||||
rows.map((r) =>
|
||||
env.DB.prepare(
|
||||
`UPDATE ingest_queue SET status=?, reject_reason=NULL
|
||||
WHERE id=? AND status IN ('pending','deferred','failed')`,
|
||||
).bind(target, r.id),
|
||||
WHERE id=? AND status IN (${DECIDABLE_SQL}) AND status <> ? ${notPublished}
|
||||
AND NOT ${openSiblingSql("ingest_queue")}`,
|
||||
).bind(target, r.id, target),
|
||||
),
|
||||
);
|
||||
await audit(env, who.email, action, null, { appids: rows.map((r) => r.appid) });
|
||||
return json({ action, decided: rows.length });
|
||||
const decidedIds = settle(results, 1, 0);
|
||||
await audit(env, who.email, action, null, {
|
||||
appids: rows.filter((r) => decidedIds.includes(r.id)).map((r) => r.appid),
|
||||
reason,
|
||||
skipped: skipped.length,
|
||||
});
|
||||
return json({ action, decided: decidedIds.length, decidedIds, skipped });
|
||||
}
|
||||
|
||||
// Reopen ONE rejected row: back to pending, and its rejection forgotten, so
|
||||
// the discovery sweep and the reviewer can see the game again. This replaced
|
||||
// a raw-SQL runbook step.
|
||||
if (route === "reopen" && request.method === "POST") {
|
||||
let body: { id?: unknown };
|
||||
try {
|
||||
body = (await request.json()) as typeof body;
|
||||
} catch {
|
||||
return jsonError(400, "invalid json");
|
||||
}
|
||||
const id = typeof body.id === "string" ? body.id.slice(0, 64) : "";
|
||||
if (!id) return jsonError(400, "id is required");
|
||||
|
||||
// The same three conditions guard both statements, evaluated against the
|
||||
// same state inside one transaction, so the decision is deleted if and
|
||||
// only if the row is reopened:
|
||||
// - the row is rejected,
|
||||
// - no other row for the appid is open or committed,
|
||||
// - the game has no approved decision (it is not published).
|
||||
// Only a 'rejected' decision is ever deleted.
|
||||
const guard = (alias: string) => `
|
||||
${alias}.status = 'rejected'
|
||||
AND NOT EXISTS (SELECT 1 FROM ingest_queue o WHERE o.appid = ${alias}.appid AND o.id <> ${alias}.id
|
||||
AND o.status IN (${sqlList([...OPEN_STATUSES, "committed"])}))
|
||||
AND NOT EXISTS (SELECT 1 FROM ingest_decisions d WHERE d.appid = ${alias}.appid AND d.decision = 'approved')`;
|
||||
|
||||
const results = await env.DB.batch([
|
||||
env.DB.prepare(
|
||||
`DELETE FROM ingest_decisions
|
||||
WHERE decision = 'rejected'
|
||||
AND appid = (SELECT q.appid FROM ingest_queue q WHERE q.id = ?1 AND ${guard("q")})`,
|
||||
).bind(id),
|
||||
env.DB.prepare(
|
||||
`UPDATE ingest_queue
|
||||
SET status = 'pending', decided_by = NULL, decided_at = NULL, reject_reason = NULL
|
||||
WHERE id = ?1 AND ${guard("ingest_queue")}`,
|
||||
).bind(id),
|
||||
]);
|
||||
|
||||
if ((results[1]?.meta?.changes ?? 0) === 0) {
|
||||
const row = await env.DB.prepare(
|
||||
`SELECT q.status, q.appid, ${publishedSql("q")} AS published FROM ingest_queue q WHERE q.id = ?`,
|
||||
)
|
||||
.bind(id)
|
||||
.first<{ status: string; appid: string; published: number }>();
|
||||
if (!row) return jsonError(404, "not found");
|
||||
const reason = row.status !== "rejected" ? "not-rejected" : row.published ? "published" : "open-row-exists";
|
||||
return new Response(JSON.stringify({ error: `cannot reopen: ${reason}`, reason }), {
|
||||
status: 409,
|
||||
headers: { "Content-Type": "application/json; charset=utf-8", "Cache-Control": "no-store", ...SECURITY_HEADERS },
|
||||
});
|
||||
}
|
||||
|
||||
const reopened = await env.DB.prepare("SELECT appid FROM ingest_queue WHERE id = ?").bind(id).first<{ appid: string }>();
|
||||
await audit(env, who.email, "reopen", id, { appid: reopened?.appid ?? null });
|
||||
return json({ reopened: id, appid: reopened?.appid ?? null });
|
||||
}
|
||||
|
||||
// Manual reconcile. The cron runs this on a schedule; the button exists so a
|
||||
// reviewer who just watched the pipeline finish need not wait for it.
|
||||
if (route === "reconcile" && request.method === "POST") {
|
||||
const out = await reconcileApproved(env);
|
||||
const out = await reconcileApproved(env, deps, { manual: true });
|
||||
await audit(env, who.email, "reconcile.manual", null, out);
|
||||
return json(out);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { deletable, handleEditApi } from "./edit";
|
||||
import { ADMIN, fakeDeps, makeEnv, request } from "../testing/fixtures";
|
||||
|
||||
const RECORD = { link: "https://store.steampowered.com/app/730/", name: "Counter-Strike 2", genre: "FPS", safe: "?" };
|
||||
|
||||
async function call(env: Env, deps: ReturnType<typeof fakeDeps>, method: string, path: string, body?: unknown) {
|
||||
const { request: req, url } = request(method, path, body);
|
||||
const res = await handleEditApi(req, url, env, ADMIN, deps);
|
||||
return { status: res.status, body: (await res.json()) as Record<string, any> };
|
||||
}
|
||||
|
||||
describe("deletable", () => {
|
||||
it("refuses while any field is still overridden", () => {
|
||||
expect(deletable({ fields: { genre: { value: "Shooter" } }, retired: {} }, RECORD).ok).toBe(false);
|
||||
});
|
||||
|
||||
it("refuses while a retired value has not been restored yet", () => {
|
||||
const doc = { fields: {}, retired: { genre: { value: "Shooter", was: "FPS" } } };
|
||||
const r = deletable(doc, { ...RECORD, genre: "Shooter" });
|
||||
expect(r.ok).toBe(false);
|
||||
expect(r.reason).toMatch(/not restored yet: genre/);
|
||||
});
|
||||
|
||||
it("allows it once every retirement has done its job", () => {
|
||||
const doc = { fields: {}, retired: { genre: { value: "Shooter", was: "FPS" } } };
|
||||
expect(deletable(doc, RECORD).ok).toBe(true);
|
||||
// An override that never changed anything has nothing to restore.
|
||||
expect(deletable({ fields: {}, retired: { safe: { value: "?", was: "?" } } }, RECORD).ok).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("POST /api/admin/edit", () => {
|
||||
it("writes an override and records a commit job", async () => {
|
||||
const env = makeEnv();
|
||||
const deps = fakeDeps({ records: { "730": RECORD } });
|
||||
const { status, body } = await call(env, deps, "POST", "/api/admin/edit", {
|
||||
appid: "730",
|
||||
set: { genre: "Tactical Shooter" },
|
||||
reason: "Steam's own tag",
|
||||
});
|
||||
expect(status).toBe(200);
|
||||
expect(body.commit).toMatch(/^sha/);
|
||||
const doc = JSON.parse(deps.repo.files.get("data/overrides/730.json")!);
|
||||
expect(doc.fields.genre).toMatchObject({ value: "Tactical Shooter", was: "FPS" });
|
||||
const job = env.sqlite.db.prepare("SELECT kind, status, target_path FROM commit_jobs").get() as any;
|
||||
expect(job).toEqual({ kind: "override", status: "committed", target_path: "data/overrides/730.json" });
|
||||
});
|
||||
|
||||
it("deletes a settled override file in its own commit", async () => {
|
||||
const env = makeEnv();
|
||||
const deps = fakeDeps({ records: { "730": RECORD } });
|
||||
deps.repo.files.set(
|
||||
"data/overrides/730.json",
|
||||
JSON.stringify({ schema: 1, appid: "730", fields: {}, retired: { genre: { value: "Shooter", was: "FPS" } } }),
|
||||
);
|
||||
|
||||
const game = await call(env, deps, "GET", "/api/admin/game?appid=730");
|
||||
expect(game.body.deletable).toEqual({ ok: true, reason: "" });
|
||||
|
||||
const { status, body } = await call(env, deps, "POST", "/api/admin/edit", { appid: "730", delete: true });
|
||||
expect(status).toBe(200);
|
||||
expect(body.deleted).toBe(true);
|
||||
expect(deps.repo.files.has("data/overrides/730.json")).toBe(false);
|
||||
expect(deps.repo.commits.at(-1)?.deletions).toEqual(["data/overrides/730.json"]);
|
||||
const job = env.sqlite.db.prepare("SELECT kind FROM commit_jobs").get() as any;
|
||||
expect(job.kind).toBe("override.delete");
|
||||
});
|
||||
|
||||
it("refuses to delete a file that still pins a value", async () => {
|
||||
const env = makeEnv();
|
||||
const deps = fakeDeps({ records: { "730": RECORD } });
|
||||
deps.repo.files.set(
|
||||
"data/overrides/730.json",
|
||||
JSON.stringify({ schema: 1, appid: "730", fields: { genre: { value: "FPS", was: "Action" } }, retired: {} }),
|
||||
);
|
||||
const { status, body } = await call(env, deps, "POST", "/api/admin/edit", { appid: "730", delete: true });
|
||||
expect(status).toBe(409);
|
||||
expect(body.error).toMatch(/retire them first/);
|
||||
expect(deps.repo.files.has("data/overrides/730.json")).toBe(true);
|
||||
});
|
||||
|
||||
it("refuses to mix delete with other changes, or with several games", async () => {
|
||||
const env = makeEnv();
|
||||
const deps = fakeDeps({ records: { "730": RECORD } });
|
||||
expect((await call(env, deps, "POST", "/api/admin/edit", { appid: "730", delete: true, retire: ["genre"] })).status).toBe(400);
|
||||
expect((await call(env, deps, "POST", "/api/admin/edit", { appids: ["730", "570"], delete: true })).status).toBe(400);
|
||||
});
|
||||
|
||||
it("marks the job failed when the commit fails", async () => {
|
||||
const env = makeEnv();
|
||||
const deps = fakeDeps({ records: { "730": RECORD }, failCommit: "branch protected" });
|
||||
const { status } = await call(env, deps, "POST", "/api/admin/edit", { appid: "730", set: { safe: "y" } });
|
||||
expect(status).toBe(502);
|
||||
expect((env.sqlite.db.prepare("SELECT status FROM commit_jobs").get() as any).status).toBe("failed");
|
||||
});
|
||||
});
|
||||
+180
-47
@@ -18,17 +18,10 @@ import { jsonError, SECURITY_HEADERS, clampLimit, clampOffset } from "../lib/htt
|
||||
import { audit } from "../lib/audit";
|
||||
import { serialiseOverride } from "../lib/override-doc";
|
||||
import type { AccessIdentity } from "../lib/access";
|
||||
import { commitFiles } from "../lib/git-commit";
|
||||
import { findOverride, findRecord, genreCounts, listByGenre } from "../lib/records";
|
||||
|
||||
/**
|
||||
* Games one request may change.
|
||||
*
|
||||
* Bounds the GraphQL read (one aliased field per file), the size of a single
|
||||
* commit, and how much one mis-click can do. Ten is the maintainer's number,
|
||||
* and it is small enough that a reviewer can still read what they selected.
|
||||
*/
|
||||
const MAX_EDIT = 10;
|
||||
import { DELETE_FILE } from "../lib/git-commit";
|
||||
import { defaultAdminDeps, type AdminDeps } from "../lib/deps";
|
||||
import type { GameRecord } from "../lib/records";
|
||||
import { MAX_EDIT } from "../../shared/queue-rules";
|
||||
|
||||
/** Mirrors MANUAL_FIELDS in scripts/core/constants.py. */
|
||||
const MANUAL_FIELDS = [
|
||||
@@ -98,6 +91,43 @@ interface OverrideDoc {
|
||||
retired: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether an override document may be deleted, and if not, why.
|
||||
*
|
||||
* Deleting an override is NOT the same as retiring one (CLAUDE.md): a deleted
|
||||
* file can no longer restore `was`, so any value it still pins stays pinned
|
||||
* forever. It is therefore allowed only once the document is SETTLED:
|
||||
*
|
||||
* - no active field is left (retire them first), and
|
||||
* - every retired value has already been restored by the pipeline - the
|
||||
* catalogue no longer holds the overridden value, or the override never
|
||||
* changed anything (`was` equals `value`).
|
||||
*
|
||||
* scripts/edit_game.py removes a file only when it has no entries at all; this
|
||||
* is the same principle, extended to a file whose retirements have finished
|
||||
* their one job.
|
||||
*/
|
||||
export function deletable(
|
||||
doc: { fields?: unknown; retired?: unknown } | null,
|
||||
record: Record<string, unknown>,
|
||||
): { ok: boolean; reason: string } {
|
||||
if (!doc) return { ok: false, reason: "no override file" };
|
||||
const fields = doc.fields && typeof doc.fields === "object" ? Object.keys(doc.fields) : [];
|
||||
if (fields.length) return { ok: false, reason: `still overriding: ${fields.join(", ")} - retire them first` };
|
||||
const retired = (doc.retired && typeof doc.retired === "object" ? doc.retired : {}) as Record<string, OverrideEntry>;
|
||||
const pending = Object.entries(retired)
|
||||
.filter(([field, entry]) => {
|
||||
if (!entry || typeof entry !== "object") return false;
|
||||
const same = JSON.stringify(entry.was ?? null) === JSON.stringify(entry.value ?? null);
|
||||
return !same && JSON.stringify(record[field] ?? null) === JSON.stringify(entry.value ?? null);
|
||||
})
|
||||
.map(([field]) => field);
|
||||
if (pending.length) {
|
||||
return { ok: false, reason: `not restored yet: ${pending.join(", ")} - wait for the pipeline to run` };
|
||||
}
|
||||
return { ok: true, reason: "" };
|
||||
}
|
||||
|
||||
function asDoc(raw: unknown, appid: string, link: string, name: string): OverrideDoc {
|
||||
const d = (raw ?? {}) as Partial<OverrideDoc>;
|
||||
return {
|
||||
@@ -115,6 +145,7 @@ export async function handleEditApi(
|
||||
url: URL,
|
||||
env: Env,
|
||||
who: AccessIdentity,
|
||||
deps: AdminDeps = defaultAdminDeps,
|
||||
): Promise<Response> {
|
||||
const route = url.pathname.slice("/api/admin/".length);
|
||||
|
||||
@@ -123,9 +154,9 @@ export async function handleEditApi(
|
||||
const appid = url.searchParams.get("appid") ?? "";
|
||||
if (!/^\d{1,10}$/.test(appid)) return jsonError(400, "appid must be 1-10 digits");
|
||||
|
||||
const record = await findRecord(appid);
|
||||
const record = await deps.findRecord(appid);
|
||||
if (!record) return jsonError(404, "not in the catalogue");
|
||||
const override = await findOverride(appid);
|
||||
const override = await deps.findOverride(appid);
|
||||
|
||||
// Only the editable fields plus enough to identify the game. The whole
|
||||
// record is not the admin screen's business and would be a much larger
|
||||
@@ -143,6 +174,7 @@ export async function handleEditApi(
|
||||
is_dead: record.is_dead ?? false,
|
||||
fields,
|
||||
override,
|
||||
deletable: deletable(override, record),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -150,7 +182,7 @@ export async function handleEditApi(
|
||||
// counts are the point: they are how the maintainer spots the vague buckets
|
||||
// worth breaking up.
|
||||
if (route === "genres" && request.method === "GET") {
|
||||
return json({ genres: await genreCounts() });
|
||||
return json({ genres: await deps.genreCounts() });
|
||||
}
|
||||
|
||||
// The games carrying one genre, for bulk retagging.
|
||||
@@ -159,14 +191,14 @@ export async function handleEditApi(
|
||||
if (!genre || genre.length > 100) return jsonError(400, "genre required");
|
||||
const limit = clampLimit(url.searchParams.get("limit"), 60, 200);
|
||||
const offset = clampOffset(url.searchParams.get("offset"));
|
||||
const out = await listByGenre(genre, limit, offset);
|
||||
const out = await deps.listByGenre(genre, limit, offset);
|
||||
return json({ genre, limit, offset, ...out });
|
||||
}
|
||||
|
||||
if (route === "edit" && request.method === "POST") {
|
||||
let body: {
|
||||
appid?: unknown; appids?: unknown;
|
||||
set?: unknown; retire?: unknown; reason?: unknown;
|
||||
set?: unknown; retire?: unknown; reason?: unknown; delete?: unknown;
|
||||
};
|
||||
try {
|
||||
body = (await request.json()) as typeof body;
|
||||
@@ -188,6 +220,53 @@ export async function handleEditApi(
|
||||
if (!/^\d{1,10}$/.test(a)) return jsonError(400, `${a}: appid must be 1-10 digits`);
|
||||
}
|
||||
|
||||
const reason = typeof body.reason === "string" ? body.reason.slice(0, 300) : "";
|
||||
|
||||
// Deleting a settled override file: one game at a time, nothing else in
|
||||
// the same request, re-checked against the file as it is at commit time.
|
||||
if (body.delete === true) {
|
||||
if (appids.length !== 1) return jsonError(400, "delete takes exactly one appid");
|
||||
if (body.set !== undefined || body.retire !== undefined) {
|
||||
return jsonError(400, "delete cannot be combined with set or retire");
|
||||
}
|
||||
const appid = appids[0];
|
||||
const record = await deps.findRecord(appid);
|
||||
if (!record) return jsonError(404, "not in the catalogue");
|
||||
const check = deletable(await deps.findOverride(appid), record);
|
||||
if (!check.ok) return jsonError(409, check.reason);
|
||||
|
||||
return commitOverride(env, deps, who, {
|
||||
kind: "override.delete",
|
||||
appids: [appid],
|
||||
targetPath: `data/overrides/${appid}.json`,
|
||||
headline: `override: remove settled file for ${record.name || appid}`,
|
||||
body:
|
||||
`Removed in /admin by ${who.email}.${reason ? `\n\nReason: ${reason}` : ""}\n\n` +
|
||||
"Every entry in it was retired and already restored, so it no longer changes anything.",
|
||||
edits: [
|
||||
{
|
||||
path: `data/overrides/${appid}.json`,
|
||||
build: (current: string, exists: boolean) => {
|
||||
if (!exists) return null;
|
||||
let doc: unknown;
|
||||
try {
|
||||
doc = JSON.parse(current);
|
||||
} catch {
|
||||
throw new Error(`existing override for ${appid} is not valid JSON`);
|
||||
}
|
||||
// Re-checked on the text being deleted, not the text read above:
|
||||
// a field set in between must not be deleted with the file.
|
||||
const again = deletable(doc as { fields?: unknown; retired?: unknown }, record);
|
||||
if (!again.ok) throw new Error(`not deletable any more: ${again.reason}`);
|
||||
return DELETE_FILE;
|
||||
},
|
||||
},
|
||||
],
|
||||
auditDetail: { appids: [appid], delete: true, reason },
|
||||
response: { appids: [appid], deleted: true },
|
||||
});
|
||||
}
|
||||
|
||||
const set = (body.set && typeof body.set === "object" ? body.set : {}) as Record<string, unknown>;
|
||||
const retire = Array.isArray(body.retire)
|
||||
? body.retire.filter((f): f is string => typeof f === "string")
|
||||
@@ -195,7 +274,6 @@ export async function handleEditApi(
|
||||
if (!Object.keys(set).length && !retire.length) {
|
||||
return jsonError(400, "nothing to set or retire");
|
||||
}
|
||||
const reason = typeof body.reason === "string" ? body.reason.slice(0, 300) : "";
|
||||
|
||||
for (const [field, value] of Object.entries(set)) {
|
||||
const why = validateValue(field, value);
|
||||
@@ -212,7 +290,7 @@ export async function handleEditApi(
|
||||
// what a later retire restores, so a client-supplied `was` would let a
|
||||
// crafted request rewrite history. It is also why a bulk edit cannot just
|
||||
// reuse one `was` across games.
|
||||
const records = await Promise.all(appids.map((a) => findRecord(a)));
|
||||
const records = await Promise.all(appids.map((a) => deps.findRecord(a)));
|
||||
const missing = appids.filter((_, i) => !records[i]);
|
||||
if (missing.length) {
|
||||
return jsonError(404, `not in the catalogue: ${missing.join(", ")}`);
|
||||
@@ -227,9 +305,16 @@ export async function handleEditApi(
|
||||
? `override: ${first.name || appids[0]} (${touched.join(", ")})`
|
||||
: `override: ${appids.length} games (${touched.join(", ")})`;
|
||||
|
||||
const sha = await commitFiles(
|
||||
env,
|
||||
appids.map((appid, i) => {
|
||||
return commitOverride(env, deps, who, {
|
||||
kind: "override",
|
||||
appids,
|
||||
targetPath: appids.length === 1 ? `data/overrides/${appids[0]}.json` : `data/overrides/{${appids.join(",")}}.json`,
|
||||
headline,
|
||||
body:
|
||||
`Edited in /admin by ${who.email}.${reason ? `\n\nReason: ${reason}` : ""}\n\n` +
|
||||
`Standing instruction re-applied by save_main() on every write to data/. ` +
|
||||
`See scripts/core/overrides.py.`,
|
||||
edits: appids.map((appid, i) => {
|
||||
const record = records[i]!;
|
||||
return {
|
||||
path: `data/overrides/${appid}.json`,
|
||||
@@ -245,7 +330,7 @@ export async function handleEditApi(
|
||||
throw new Error(`existing override for ${appid} is not valid JSON`);
|
||||
}
|
||||
}
|
||||
const doc = asDoc(existing, appid, record.link, record.name ?? "");
|
||||
const doc = asDoc(existing, appid, record.link, (record as GameRecord).name ?? "");
|
||||
|
||||
for (const [field, value] of Object.entries(set)) {
|
||||
// `was` is captured the FIRST time a field is overridden and
|
||||
@@ -280,33 +365,81 @@ export async function handleEditApi(
|
||||
},
|
||||
};
|
||||
}),
|
||||
headline,
|
||||
`Edited in /admin by ${who.email}.${reason ? `\n\nReason: ${reason}` : ""}\n\n` +
|
||||
`Standing instruction re-applied by save_main() on every write to data/. ` +
|
||||
`See scripts/core/overrides.py.`,
|
||||
);
|
||||
|
||||
// Shared helper, and best-effort on purpose: this runs after commitFiles
|
||||
// has already landed a commit, so a D1 failure here must not report the
|
||||
// override as failed. See lib/audit.ts.
|
||||
await audit(env, who.email, "override", appids.join(","), {
|
||||
appids,
|
||||
set: Object.keys(set),
|
||||
retire,
|
||||
sha,
|
||||
reason,
|
||||
});
|
||||
|
||||
return json({
|
||||
appids,
|
||||
commit: sha,
|
||||
set: Object.keys(set),
|
||||
retired: retire,
|
||||
// data/ is untouched until the pipeline runs; say so rather than letting
|
||||
// the reviewer assume the catalogue changed.
|
||||
applied: false,
|
||||
auditDetail: { appids, set: Object.keys(set), retire, reason },
|
||||
response: {
|
||||
appids,
|
||||
set: Object.keys(set),
|
||||
retired: retire,
|
||||
// data/ is untouched until the pipeline runs; say so rather than
|
||||
// letting the reviewer assume the catalogue changed.
|
||||
applied: false,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
return jsonError(404, "not found");
|
||||
}
|
||||
|
||||
/**
|
||||
* One override commit, with its commit_jobs row.
|
||||
*
|
||||
* Override commits used to leave no job at all - only an audit row written
|
||||
* AFTER the commit - so an edit whose Worker died mid-flight left no trace, and
|
||||
* the admin "Commit jobs" view never showed an override. The job is written
|
||||
* before the attempt, like an approval's.
|
||||
*/
|
||||
async function commitOverride(
|
||||
env: Env,
|
||||
deps: AdminDeps,
|
||||
who: AccessIdentity,
|
||||
plan: {
|
||||
kind: "override" | "override.delete";
|
||||
appids: string[];
|
||||
targetPath: string;
|
||||
headline: string;
|
||||
body: string;
|
||||
edits: { path: string; build: (current: string, exists: boolean) => string | null | typeof DELETE_FILE }[];
|
||||
auditDetail: Record<string, unknown>;
|
||||
response: Record<string, unknown>;
|
||||
},
|
||||
): Promise<Response> {
|
||||
const jobId = crypto.randomUUID();
|
||||
const startedAt = deps.now().toISOString();
|
||||
try {
|
||||
await env.DB.prepare(
|
||||
`INSERT INTO commit_jobs (id, kind, status, target_path, requested_by, created_at)
|
||||
VALUES (?, ?, 'pending', ?, ?, ?)`,
|
||||
)
|
||||
.bind(jobId, plan.kind, plan.targetPath.slice(0, 500), who.email, startedAt)
|
||||
.run();
|
||||
} catch (err) {
|
||||
// The commit is the point; a bookkeeping failure must not block it.
|
||||
console.error("override: commit_jobs insert failed", err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
|
||||
let sha: string | null;
|
||||
try {
|
||||
sha = await deps.commitFiles(env, plan.edits, plan.headline, plan.body);
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
await env.DB.prepare("UPDATE commit_jobs SET status='failed', error=?, finished_at=? WHERE id=?")
|
||||
.bind(message.slice(0, 500), deps.now().toISOString(), jobId)
|
||||
.run()
|
||||
.catch(() => undefined);
|
||||
await audit(env, who.email, `${plan.kind}.failed`, plan.appids.join(","), { ...plan.auditDetail, message });
|
||||
return jsonError(502, "commit failed: " + message);
|
||||
}
|
||||
|
||||
await env.DB.prepare("UPDATE commit_jobs SET status='committed', commit_sha=?, error=?, finished_at=? WHERE id=?")
|
||||
.bind(sha, sha ? null : "nothing to commit: the files already said this", deps.now().toISOString(), jobId)
|
||||
.run()
|
||||
.catch((err: unknown) =>
|
||||
console.error("override: commit_jobs update failed", err instanceof Error ? err.message : String(err)),
|
||||
);
|
||||
|
||||
// Best-effort on purpose: the commit has already landed, so a D1 failure
|
||||
// here must not report the override as failed. See lib/audit.ts.
|
||||
await audit(env, who.email, plan.kind, plan.appids.join(","), { ...plan.auditDetail, sha, job: jobId });
|
||||
|
||||
return json({ ...plan.response, commit: sha, job: jobId });
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
/**
|
||||
* Builders shared by the admin handler tests and the local admin dev server:
|
||||
* an Env over SqliteD1, fake dependencies that record what they were asked to
|
||||
* do, and queue rows with sensible defaults.
|
||||
*/
|
||||
import type { AdminDeps } from "../lib/deps";
|
||||
import type { CommitResult, FileEdit } from "../lib/git-commit";
|
||||
import { DELETE_FILE } from "../lib/git-commit";
|
||||
import type { GameRecord } from "../lib/records";
|
||||
import { SqliteD1 } from "./sqlite-d1";
|
||||
|
||||
export const ADMIN = { email: "reviewer@example.com", isServiceToken: false };
|
||||
|
||||
export function makeEnv(db = new SqliteD1()): Env & { sqlite: SqliteD1 } {
|
||||
return {
|
||||
DB: db.asD1(),
|
||||
sqlite: db,
|
||||
SITE_ORIGIN: "https://free-steam-games.win",
|
||||
IMG_TRANSFORM: "false",
|
||||
ADMIN_RETENTION_DAYS: "180",
|
||||
ACCESS_TEAM_DOMAIN: "example.cloudflareaccess.com",
|
||||
ACCESS_AUD_ADMIN: "aud",
|
||||
ACCESS_AUD_INGEST: "aud-ingest",
|
||||
} as unknown as Env & { sqlite: SqliteD1 };
|
||||
}
|
||||
|
||||
export interface QueueRowInput {
|
||||
id?: string;
|
||||
appid: string;
|
||||
name?: string;
|
||||
status?: string;
|
||||
first_seen_at?: string;
|
||||
decided_by?: string | null;
|
||||
decided_at?: string | null;
|
||||
reject_reason?: string | null;
|
||||
}
|
||||
|
||||
let counter = 0;
|
||||
|
||||
export function insertRow(env: Env, row: QueueRowInput): string {
|
||||
const id = row.id ?? `row-${++counter}`;
|
||||
const seen = row.first_seen_at ?? `2026-09-${String(10 + (counter % 10)).padStart(2, "0")}T00:00:00Z`;
|
||||
(env as unknown as { sqlite: SqliteD1 }).sqlite.db
|
||||
.prepare(
|
||||
`INSERT INTO ingest_queue (id, appid, link, name, status, decided_by, decided_at, reject_reason,
|
||||
first_seen_at, last_seen_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
)
|
||||
.run(
|
||||
id,
|
||||
row.appid,
|
||||
`https://store.steampowered.com/app/${row.appid}/`,
|
||||
row.name ?? `Game ${row.appid}`,
|
||||
row.status ?? "pending",
|
||||
row.decided_by ?? null,
|
||||
row.decided_at ?? null,
|
||||
row.reject_reason ?? null,
|
||||
seen,
|
||||
seen,
|
||||
);
|
||||
return id;
|
||||
}
|
||||
|
||||
export function insertDecision(env: Env, appid: string, decision: "approved" | "rejected"): void {
|
||||
(env as unknown as { sqlite: SqliteD1 }).sqlite.db
|
||||
.prepare("INSERT INTO ingest_decisions (appid, decision, decided_at) VALUES (?, ?, '2026-09-01T00:00:00Z')")
|
||||
.run(appid, decision);
|
||||
}
|
||||
|
||||
export function rowStatus(env: Env, id: string): string | undefined {
|
||||
const row = (env as unknown as { sqlite: SqliteD1 }).sqlite.db
|
||||
.prepare("SELECT status FROM ingest_queue WHERE id = ?")
|
||||
.get(id) as { status: string } | undefined;
|
||||
return row?.status;
|
||||
}
|
||||
|
||||
export function decisionOf(env: Env, appid: string): string | undefined {
|
||||
const row = (env as unknown as { sqlite: SqliteD1 }).sqlite.db
|
||||
.prepare("SELECT decision FROM ingest_decisions WHERE appid = ?")
|
||||
.get(appid) as { decision: string } | undefined;
|
||||
return row?.decision;
|
||||
}
|
||||
|
||||
/** A shard line for a published game. */
|
||||
export function shardLine(appid: string, extra: Record<string, unknown> = {}): string {
|
||||
return JSON.stringify({ link: `https://store.steampowered.com/app/${appid}/`, name: `Game ${appid}`, ...extra });
|
||||
}
|
||||
|
||||
export interface FakeRepo {
|
||||
/** Files in the fake repository, by path. */
|
||||
files: Map<string, string>;
|
||||
commits: { headline: string; body: string; additions: string[]; deletions: string[] }[];
|
||||
appended: string[][];
|
||||
}
|
||||
|
||||
/**
|
||||
* Dependencies with an in-memory "repository" and "dataset". Nothing reaches
|
||||
* the network or GitHub.
|
||||
*/
|
||||
export function fakeDeps(options: {
|
||||
records?: Record<string, GameRecord>;
|
||||
raw?: Record<string, string>;
|
||||
failCommit?: string;
|
||||
now?: Date;
|
||||
gh?: AdminDeps["gh"];
|
||||
} = {}): AdminDeps & { repo: FakeRepo } {
|
||||
const repo: FakeRepo = { files: new Map(), commits: [], appended: [] };
|
||||
let sha = 0;
|
||||
const now = options.now ?? new Date("2026-09-17T12:00:00Z");
|
||||
|
||||
const appendLinks = async (_env: Env, links: string[], actor: string, reason = ""): Promise<CommitResult> => {
|
||||
if (options.failCommit) throw new Error(options.failCommit);
|
||||
const queued = new Set(repo.appended.flat());
|
||||
const fresh = [...new Set(links)].filter((l) => !queued.has(l));
|
||||
const skipped = [...new Set(links)].filter((l) => queued.has(l));
|
||||
if (!fresh.length) return { sha: null, appended: 0, skipped };
|
||||
repo.appended.push(fresh);
|
||||
repo.commits.push({ headline: `queue: ${fresh.length}`, body: `by ${actor} ${reason}`, additions: ["scripts/temp_info.jsonl"], deletions: [] });
|
||||
return { sha: `sha${++sha}`.padEnd(40, "0"), appended: fresh.length, skipped };
|
||||
};
|
||||
|
||||
const commitFiles = async (_env: Env, edits: FileEdit[], headline: string, body: string) => {
|
||||
if (options.failCommit) throw new Error(options.failCommit);
|
||||
const additions: string[] = [];
|
||||
const deletions: string[] = [];
|
||||
for (const e of edits) {
|
||||
const exists = repo.files.has(e.path);
|
||||
const next = e.build(repo.files.get(e.path) ?? "", exists);
|
||||
if (next === DELETE_FILE) {
|
||||
if (exists) {
|
||||
repo.files.delete(e.path);
|
||||
deletions.push(e.path);
|
||||
}
|
||||
} else if (next !== null && next !== repo.files.get(e.path)) {
|
||||
repo.files.set(e.path, next);
|
||||
additions.push(e.path);
|
||||
}
|
||||
}
|
||||
if (!additions.length && !deletions.length) return null;
|
||||
repo.commits.push({ headline, body, additions, deletions });
|
||||
return `sha${++sha}`.padEnd(40, "0");
|
||||
};
|
||||
|
||||
return {
|
||||
repo,
|
||||
appendLinks: appendLinks as AdminDeps["appendLinks"],
|
||||
commitFiles: commitFiles as AdminDeps["commitFiles"],
|
||||
gh: options.gh ?? (async () => new Response("[]", { status: 200 })),
|
||||
findRecord: async (appid) => options.records?.[appid] ?? null,
|
||||
findOverride: async (appid) => {
|
||||
const text = repo.files.get(`data/overrides/${appid}.json`);
|
||||
return text ? (JSON.parse(text) as Record<string, unknown>) : null;
|
||||
},
|
||||
genreCounts: async () => [],
|
||||
listByGenre: async () => ({ total: 0, items: [] }),
|
||||
fetchRaw: async (path) => options.raw?.[path] ?? null,
|
||||
now: () => now,
|
||||
};
|
||||
}
|
||||
|
||||
/** Call a handler with a JSON body the way the Worker would route it. */
|
||||
export function request(method: string, path: string, body?: unknown): { request: Request; url: URL } {
|
||||
const url = new URL(`https://free-steam-games.win${path}`);
|
||||
return {
|
||||
url,
|
||||
request: new Request(url, {
|
||||
method,
|
||||
...(body === undefined ? {} : { body: JSON.stringify(body), headers: { "Content-Type": "application/json" } }),
|
||||
}),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
/**
|
||||
* An in-memory D1 for tests and for `npm run dev:admin`, over node:sqlite.
|
||||
*
|
||||
* It runs the REAL migrations from worker/migrations, so a handler under test
|
||||
* meets the same CHECK constraints, STRICT typing and partial unique index as
|
||||
* production - the index is exactly what the admin decide/reopen guards exist
|
||||
* to respect, and a mock that ignored it would prove nothing.
|
||||
*
|
||||
* Only the surface the Worker uses is implemented: prepare/bind, first, all,
|
||||
* run (with meta.changes), batch (one transaction, rolled back on any throw,
|
||||
* as D1 does) and exec. Node-only: excluded from tsconfig.worker.json.
|
||||
*/
|
||||
import { DatabaseSync } from "node:sqlite";
|
||||
import { readdirSync, readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
type Param = string | number | bigint | null | Uint8Array;
|
||||
|
||||
function normalise(params: unknown[]): Param[] {
|
||||
return params.map((p) => {
|
||||
if (p === undefined) throw new Error("D1_TYPE_ERROR: undefined is not a supported bind value");
|
||||
if (typeof p === "boolean") return p ? 1 : 0;
|
||||
return p as Param;
|
||||
});
|
||||
}
|
||||
|
||||
/** SELECT-shaped statements return rows; everything else reports changes. */
|
||||
function isReader(sql: string): boolean {
|
||||
const s = sql.trim().toUpperCase();
|
||||
return s.startsWith("SELECT") || s.startsWith("WITH") || /\bRETURNING\b/.test(s);
|
||||
}
|
||||
|
||||
class Statement {
|
||||
constructor(
|
||||
private readonly db: DatabaseSync,
|
||||
readonly sql: string,
|
||||
private readonly params: unknown[] = [],
|
||||
) {}
|
||||
|
||||
bind(...params: unknown[]): Statement {
|
||||
return new Statement(this.db, this.sql, params);
|
||||
}
|
||||
|
||||
async first<T = Record<string, unknown>>(column?: string): Promise<T | null> {
|
||||
const row = this.db.prepare(this.sql).get(...normalise(this.params)) as Record<string, unknown> | undefined;
|
||||
if (!row) return null;
|
||||
return (column ? row[column] : row) as T;
|
||||
}
|
||||
|
||||
async all<T = Record<string, unknown>>() {
|
||||
return this.execute<T>(true);
|
||||
}
|
||||
|
||||
async run() {
|
||||
return this.execute(false);
|
||||
}
|
||||
|
||||
/** What batch() needs from each statement. */
|
||||
execute<T>(preferRows: boolean) {
|
||||
const stmt = this.db.prepare(this.sql);
|
||||
if (preferRows || isReader(this.sql)) {
|
||||
const results = stmt.all(...normalise(this.params)) as T[];
|
||||
return { success: true, results, meta: { changes: 0, last_row_id: 0, rows_read: results.length, rows_written: 0 } };
|
||||
}
|
||||
const info = stmt.run(...normalise(this.params));
|
||||
return {
|
||||
success: true,
|
||||
results: [] as T[],
|
||||
meta: { changes: Number(info.changes), last_row_id: Number(info.lastInsertRowid), rows_read: 0, rows_written: Number(info.changes) },
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export class SqliteD1 {
|
||||
readonly db: DatabaseSync;
|
||||
|
||||
constructor(options: { migrations?: boolean } = {}) {
|
||||
this.db = new DatabaseSync(":memory:");
|
||||
if (options.migrations !== false) this.applyMigrations();
|
||||
}
|
||||
|
||||
/** The migrations directory, applied in order and recorded like wrangler does. */
|
||||
applyMigrations(dir = join(process.cwd(), "worker", "migrations")): void {
|
||||
this.db.exec(
|
||||
"CREATE TABLE IF NOT EXISTS d1_migrations (id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT UNIQUE, applied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP)",
|
||||
);
|
||||
for (const name of readdirSync(dir).filter((n) => n.endsWith(".sql")).sort()) {
|
||||
this.db.exec(readFileSync(join(dir, name), "utf-8"));
|
||||
this.db.prepare("INSERT INTO d1_migrations (name) VALUES (?)").run(name);
|
||||
}
|
||||
}
|
||||
|
||||
prepare(sql: string): Statement {
|
||||
return new Statement(this.db, sql);
|
||||
}
|
||||
|
||||
async batch(statements: Statement[]) {
|
||||
this.db.exec("BEGIN");
|
||||
try {
|
||||
const out = statements.map((s) => s.execute(false));
|
||||
this.db.exec("COMMIT");
|
||||
return out;
|
||||
} catch (err) {
|
||||
this.db.exec("ROLLBACK");
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
async exec(sql: string) {
|
||||
this.db.exec(sql);
|
||||
return { count: 1, duration: 0 };
|
||||
}
|
||||
|
||||
/** Typed as D1Database for handlers; the subset above is what they use. */
|
||||
asD1(): D1Database {
|
||||
return this as unknown as D1Database;
|
||||
}
|
||||
}
|
||||
@@ -86,6 +86,11 @@
|
||||
// decide. Passthrough still gets edge caching.
|
||||
"IMG_TRANSFORM": "false",
|
||||
|
||||
// Days audit_log and commit_jobs rows are kept (worker/lib/prune.ts, run
|
||||
// once a day from the cron). docs/PRIVACY_POLICY.md states this number -
|
||||
// change both together. Values under 7 are ignored as a safety floor.
|
||||
"ADMIN_RETENTION_DAYS": "180",
|
||||
|
||||
// Cloudflare Access. Neither value is a secret: the team domain appears in
|
||||
// the login redirect and the AUD tag identifies the application. They are
|
||||
// vars so the Worker can verify tokens itself rather than trusting that a
|
||||
|
||||
Reference in new issue
Block a user