docs: the pipeline-scale assessment, CLAUDE.md, README, CHANGELOG

docs/plan/18-pipeline-scale.md holds the measurements (9,847 games,
+~280 a day), what each job cost, what GetItems answers, the direction
and what is left. CLAUDE.md gains the rules for reviews, the purge,
budgets and the data-write group; README the new schedules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 committed 2026-10-23 22:22:09 +07:00
1 parent 529b8603c3
commit 0897443bd6
4 files changed
+311 -31

No files matched your search

+37
View File
@@ -4,6 +4,43 @@ All notable changes to this awesome noob repo will be documented here.
## [Unreleased]
### 🧰 Pipeline
- **The data jobs scale with the catalogue again.** With almost 10,000 games
and a few hundred more every day, three jobs that asked Steam about one
game at a time had reached GitHub's six-hour limit: on 2026-10-23 Update
Reviews was stopped at exactly six hours with nothing saved. The full
assessment, with every measurement, is `docs/plan/18-pipeline-scale.md`.
- **Reviews come from Steam's store API, for every game, every day.** Fifty
games a request, about two minutes for the catalogue, instead of every two
days and five and a half hours. The percentage is now Steam's own, which
rounds down: about 4,000 games show one point less once (Counter-Strike 2
"85%" instead of "86%"), exactly as on their store pages. A summary that
says 0 reviews is double-checked game by game before it replaces anything.
- **The daily purge needs two answers to remove a game.** Steam's store API
screens every game in a few minutes, and a game is removed only when
appdetails agrees; "unavailable" is asked twice, five minutes apart. A run
that would remove more than 100 games, or flags more than 500, stops before
writing anything. It also lists earlier removals that Steam has since
undone. It runs daily now, instead of a five-hour sweep on Mondays.
- **Check Dead Links is retired.** In its whole life it never found a dead
link: Steam sends a delisted game's page to the store front, which answers
200. The purge's screen sees delisted games (`unlisted`) instead.
- **No run is cancelled for waiting.** The data jobs share one queue that
used to hold a single waiting run: a third job cancelled it, and Update
Reviews lost five of ten runs that way. The queue now holds them all, first
in, first out, and every job checks out the latest `main` when it starts -
Auto Update JSON failed four days in ten from working on the data as it was
when the run was queued.
- **Every long job stops in time to save.** Each workflow gives its job a time
budget below its timeout: past it the job saves and commits what it did
instead of being stopped with nothing, and it warns before the budget gets
tight. The leaderboards skip the pauses meant for store pages (Top Online
~27 minutes instead of 48). The store-data and all-time-peak rotations
stretch their cycle as the catalogue grows instead of their daily runs.
- **Force Re-fetch runs in parts** (`1/8` … `8/8`, queued one after another),
and a game whose fetch fails keeps its old data.
### 🧰 Tooling
- **Desktop releases keep reaching the updater.** tauri-action v1 writes each
+93 -19
View File
@@ -222,7 +222,13 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
(tags, languages, `language_details`, `has_paid_dlc`) every 7 days,
appdetails (name, header image, description, developer/publisher, release
date, platforms, Metacritic, DRM notes) every 30. Never `appid % cycle`:
appids are mostly multiples of 10, which leaves most slices empty. It
appids are mostly multiples of 10, which leaves most slices empty. A cycle
grows with the catalogue instead of the run (`rotation.cycle_for()`: past
`PAGE_DAILY` 2,000 pages or `DETAILS_DAILY` 500 appdetails a day it becomes
14, 21, ... or 60, 90, ...) and is always a MULTIPLE of its base, so a game
keeps its weekday and its day of the month and visits only ever thin out.
The slice is visited in `rotation.daily_order()`, so a run that meets its
time budget leaves a different tail each day. It
replaces values only with NON-EMPTY ones, never writes `MANUAL_FIELDS` or
`notes`, trusts `has_paid_dlc` only from a real store page (an age gate has
no language table or tags) or a gated game's DLC prices, compares every
@@ -231,6 +237,9 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
counting as news), and bumps `last_updated` only on a real change. A game
with empty tags, languages or `language_details` is in the page group EVERY
day, first, capped at `MISSING_DAILY_CAP`; `--missing-only` runs just those.
A game whose appdetails no longer call it free is LISTED in a `::warning::`,
never written: purge removes a game only when the store API flags it too,
and this is where the opposite case shows.
A new field added to it must obey the same rules;
`scripts/tests/test_refresh_store_data.py` holds them.
- **A store page behind Steam's age gate is never parsed.** An 'Adult Only'
@@ -252,7 +261,8 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
with `l=english`: without it, a runner was once answered in Russian.
- **`all_time_peak` only ever goes UP, and `refresh_peaks.py` is what raises
it** (daily 19:30 UTC, "Refresh All-Time Peaks"). A `crc32(appid) % 30`
slice (`core/rotation.py`, shared with `refresh_store_data.py`) reads
slice (`core/rotation.py`, shared with `refresh_store_data.py`; 60, 90, ...
past `PEAK_DAILY` 400 pages a day, never shorter) reads
SteamCharts' recorded all-time peak - the only source older than our own
sampling: SteamDB forbids scraping and Valve's APIs know only today. Every
run also raises it from our samples (`peak_today`, which despite its name is
@@ -260,8 +270,9 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
`apply_players()` raises it on every sample, never from a 0. SteamCharts has
no API, terms or robots.txt, so: one page per game a month, 2-3 s apart, a
User-Agent that names the project; 500 is an app it does not track (most of
the long tail), 404 one it has never seen; a 403, 429 or challenge page
stops the run with a warning, not a failure. `refetch_all.py` must never
the long tail), 404 one it has never seen; a 403, 429 or challenge page,
or the job's time budget, stops the run with a warning, not a failure.
`refetch_all.py` must never
clear the field - it cannot be fetched again - and nothing may lower it.
The backfill is a manual run with `backfill` ticked (~4 h, start it in the
evening UTC). DISCLAIMER section 6 names the source, so rewording it reopens
@@ -293,6 +304,42 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
name and the new key are the same family, and a known label stays as it was
written (so "Easy Anti-Cheat" and "EAC" both occur; the site groups them).
`scripts/tests/test_anticheat.py` and `test_refresh_anticheat.py` hold it.
- **Reviews are Steam's own summary, read for EVERY game daily**
(`update_reviews.py`, "Update Reviews" 06:00): `GetItems` with
`include_reviews`, fifty games a request (`SteamClient.fetch_items()`, the
sweep the anti-cheat and purge jobs share), ~2 minutes. It asked
`appreviews` one game at a time until v4.4.0 and, at 9,800 games, ran into
GitHub's six-hour limit with nothing saved. Rules: the percentage is Steam's,
FLOORED (`reviews_from_summary()` floors appreviews in integers too, so an
ingest and the refresh agree; until v4.4.0 it was rounded, one point higher
on ~40% of games); a GetItems count of 0 is never written - its summary has
said 0 for a game with thousands (845070) - so a stored percentage or "N/A"
is asked on appreviews (`ASK_CAP` 300 a run, daily order) and only that
answer is written; no item, no usable summary or a failed batch keeps the
value. A record's `last_updated` moves on a change and on a HEARTBEAT: a
game read whose stamp is 14-20 days old (crc32 jitter) is stamped again.
The site calls a record older than 30 days stale (`web/src/lib/validation.ts`,
`/health`), and this job used to stamp every game on every run; the
heartbeat keeps "the pipeline confirmed this lately" true without rewriting
every record daily. `scripts/tests/test_update_reviews.py` holds it.
- **A game is purged only when two of Steam's answers agree**
(`purge_unhealthy.py`, "Purge Unhealthy Games" daily 07:00). GetItems
screens the whole catalogue (`store_items.screen_item()`: unknown to Steam,
`visible` false, `unlisted`, `release.is_coming_soon`, no `is_free`; never
`type` - a free mod is listed on purpose), and `check_game_health()`
(appdetails) must confirm each flag. An `unavailable`/404 answer is asked
again `RECHECK_AFTER` (5 min) later: appdetails answered `success:false` for
14 live games in two bursts on 2026-09-27. A failed batch or a network error
is never "gone"; blocklisted games are not asked about. More than 500 flags
or 100 confirmed removals exits non-zero BEFORE anything is written
(`--max-candidates`, `--max-removals` to override). Removals are logged to
`removed_games.jsonl` exactly as before - `reconcile.ts` reads them. A
`::notice::` names earlier removals GetItems now sees free, listed and
released (re-queue the link to bring one back: a purged game keeps its D1
decision and discovery never proposes it again). Check Dead Links is gone:
HEAD on a delisted app's page ends on the store front with 200, so in its
whole life it never found a dead link, and it took 5.5 hours.
`scripts/tests/test_purge_unhealthy.py` holds it.
- `ingest_new.py` only reads `scripts/temp_info.jsonl`, and a line is a store
LINK and nothing else: every other key is ignored and the game is built from
Steam (`merge_extension_data()` went with the archived browser extension;
@@ -305,14 +352,15 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
appdetails calls other than `type: "game"` (`status_code: not_a_game` in
`removed_games.jsonl`) - the check that makes auto-approve safe, and why a
queued link cannot add a free DLC, demo or tool. The check lives in ingest
only: in `health_checker` it would make Purge delete old records every week.
only: in `health_checker` it would make Purge delete old records every day.
- **`ingest-new.yml` is also the removal job.** `bash/ingest.sh` runs
`remove_games.py` (applies `data/blocklist.jsonl`) and then
`ingest_new.py`, and commits once ("Apply removals" when `temp_info` did not
change). A separate removal workflow would keep evicting the ingest from
`data-write`'s single pending slot, and the ingest it. The workflow checks
out `ref: main`, not the triggering SHA: a run that waited its turn would
otherwise work on the data as it was when it was queued.
change), so one commit to either file starts one run that settles both.
The workflow checks out `ref: main`, not the triggering SHA, like every
data-write workflow: a run that waited its turn would otherwise work on the
data as it was when it was queued. It has no time budget: `clear_temp()`
empties the whole queue file, so a game it had not reached would be lost.
- New games ARE discovered automatically now, by `scripts/discover_new.py`
into the D1 review queue — but discovery only PROPOSES. Publication is
decided in `/admin`: by a reviewer, batch by batch, or by the auto-approve
@@ -329,8 +377,10 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
page previews is the SQL that runs). The empty-file gate is checked again
inside the commit (`requireEmptyQueue`), so a batch never lands mid-ingest;
"no row is approved" would have stalled it for the 12 hours a skipped row
stays approved. The idle gate exists because a push while a data job waits
EVICTS that job. `shared/pipeline.ts` lists every `data-write` workflow by
stays approved. The idle gate dates from when a push while a data job waited
EVICTED that job; with `queue: max` the ingest would only queue behind it,
and the gate stays so a batch lands when its ingest can start at once.
`shared/pipeline.ts` lists every `data-write` workflow by
file, and `pipeline.test.ts` fails when a workflow joins the group without
joining the list. Settings, limits and their validation are
`shared/automation.ts`, imported by the Worker and the page. A scheduled
@@ -393,16 +443,40 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
(wrangler.jsonc, 180). `docs/PRIVACY_POLICY.md` states the number: change
both together. `ingest_queue` and `ingest_decisions` are never pruned — the
decisions are what keep a rejected game out of the discovery sweep.
- Every workflow that writes to `data/` shares `concurrency: {group: data-write}`.
A new data-writing workflow without it will race jobs that run ~2 hours.
The group holds ONE pending run - a second queueing run cancels the first -
and scheduled runs here start 2-5 hours after their cron, so a new schedule
goes where nothing long is running (the long jobs - reviews, dead links,
purge - start mid-morning UTC and run 2-4 hours; Refresh Store Data is at
16:00 for that reason).
- **Every workflow that writes to `data/` shares `concurrency: {group:
data-write, cancel-in-progress: false, queue: max}`, checks out `ref: main`,
and declares `timeout-minutes`.** `pipeline.test.ts` fails on a data-write
workflow without any of them. A new data-writing workflow outside the group
would race the others.
- `queue: max` (GitHub, 2026-05-07) keeps up to 100 runs waiting, first in,
first out. The default keeps ONE: a run queueing behind a running job
cancelled the one already waiting - Update Reviews lost five of ten runs
that way in October 2026. Mixing the two settings in one group is
undocumented, so every member has `queue: max`.
- `ref: main`: a run's commit is fixed when the run is CREATED. A scheduled
run that waited behind another data job wrote shards from before that
job's commit, and `commit_push.sh`'s rebase then conflicted on
`data/index.json` - Auto Update JSON failed four days in ten that way.
- Scheduled runs start up to an hour after their cron (2-5 hours in
September). Waiting costs nothing now, but a long new job still goes where
nothing long runs (Refresh Store Data is at 16:00 for that reason).
- **Every long loop has a time budget** (`scripts/core/budget.py`). The
workflow sets `JOB_BUDGET_MINUTES` at least 20 below its `timeout-minutes`
(`pipeline.test.ts` checks); past it a loop starts no further game, the
script saves what it did and the job commits it, and a loop that used more
than three quarters says so in a `::warning::`. Loops visit their games in
`rotation.daily_order()`, so a cut falls elsewhere the next day. A job
stopped at its timeout commits nothing and concludes `cancelled`, which
`notify-ci-failure.yml` (a reusable workflow in `poli0981/.github`) does not
report: the budget is what keeps a growing catalogue from failing silently.
`process_batch(pause=False)` for the keyed Web API (player counts); the
15-30 s rest between batches stays for store pages and appdetails. Never a
budget for `ingest_new.py`. `refetch_all.py` runs in parts (`--part K/N`)
and clears each game on a copy that replaces it only once appdetails has
answered, so a cut part leaves every other game as it was.
- **Every data job commits through `bash/commit_push.sh`.** The group
serialises workflows, not `main`: a merged PR or an /admin commit can land
during a four-hour job, and a bare `git push` is then rejected and the run
during a long job, and a bare `git push` is then rejected and the run
thrown away (purge-unhealthy, 2026-09-21). The helper rebases onto
`origin/main` and retries; none of those writers touch the shards, so the
rebase is clean. A new wrapper must use it, once per commit (Refresh Store
+11 -12
View File
@@ -5,7 +5,7 @@
[![Web App](https://img.shields.io/badge/Web%20App-Live-2563eb?style=flat&logo=svelte)](https://free-steam-games.win/)
[![Desktop](https://img.shields.io/badge/Desktop-Tauri%202-FFC131?style=flat&logo=tauri)](https://github.com/poli0981/free-steam-games-list/releases)
[![Android](https://img.shields.io/badge/Android-11%2B%20APK-3DDC84?style=flat&logo=android)](https://github.com/poli0981/free-steam-games-list/releases)
[![Health Check](https://img.shields.io/badge/Health%20Check-Weekly-orange?style=flat&logo=github-actions)](.github/workflows/purge-unhealthy.yml)
[![Health Check](https://img.shields.io/badge/Health%20Check-Daily-orange?style=flat&logo=github-actions)](.github/workflows/purge-unhealthy.yml)
![Visitors](https://visitor-badge.laobi.icu/badge?page_id=poli0981.free-steam-games-list)
[![Sponsor](https://img.shields.io/badge/Sponsor-Buy%20me%20a%20coffee-ff5f5f?style=flat&logo=buy-me-a-coffee)](.github/FUNDING.yml)
@@ -89,8 +89,8 @@ The Python pipeline that powered v2 is untouched and runs daily as before:
- Schema v2.2 (23 + manual fields per record), CC BY 4.0 (see [LICENSE-DATA](LICENSE-DATA)), sharded data in `data/data_*.jsonl` (cap 800 records/shard) plus a `data/index.json` manifest with a SHA-256 per shard.
- HTML scraper for accurate language tables, DLC pricing, user tags and the developer's anti-cheat disclosure — single GET per game.
- Health-checked: dead-link sweep every 5 days, full purge every Monday.
- Reviews-only refresh every 2 days; full refresh daily.
- Health-checked daily: Steam's store API screens every game, and a game is removed only when appdetails agrees.
- Reviews refreshed daily from Steam's own summaries; full refresh daily. Every data job queues for its turn and has a time budget, so a growing catalogue slows a refresh down instead of breaking it ([plan](docs/plan/18-pipeline-scale.md)).
</details>
@@ -151,7 +151,8 @@ scripts/
│ ├── anticheat.py # Steam's anti-cheat disclosure and the rule that applies it
│ ├── steamcharts.py # SteamCharts all-time peak reader (polite, stops on refusal)
│ ├── peaks.py # all_time_peak: only ever raised
│ ├── rotation.py # crc32 daily slices shared by the refresh jobs
│ ├── rotation.py # crc32 daily slices shared by the refresh jobs, growing with the catalogue
│ ├── budget.py # A job's time budget: stop in time to save, warn before it runs out
│ └── health_checker.py # Game status detection with cached API data
├── discover_new.py # Propose new games into the /admin review queue
├── ingest_new.py # Add queued games (health check + fetch from Steam)
@@ -160,13 +161,12 @@ scripts/
├── refresh_store_data.py # Daily rotation: tags/languages weekly, store details monthly
├── refresh_anticheat.py # Daily: every game's anti-cheat, as its developer declares it on Steam
├── refresh_peaks.py # Daily rotation: all-time peak from SteamCharts, monthly per game
├── update_reviews.py # Reviews-only (every 2 days)
├── update_reviews.py # Daily: every game's review summary from Steam, ~2 minutes
├── mark_dead_games.py # Flag online games with no players
├── check_dead_links.py # HEAD-request 404/410 scanner
├── purge_unhealthy.py # Remove delisted/not-free/invalid
├── purge_unhealthy.py # Daily: remove delisted/not-free/unreleased (store API screen + appdetails)
├── top_online.py / top_offline.py # Leaderboards
├── edit_game.py # Overrides from the command line
├── refetch_all.py # Force re-fetch ALL (manual only)
├── refetch_all.py # Force re-fetch, in parts (manual only)
└── delete_game.py # Remove games (blocklist) from the command line
```
@@ -181,14 +181,13 @@ scripts/
| Ingest New | On `scripts/temp_info.jsonl` or `data/blocklist.jsonl` push, and every 3 h | Apply removals, then add approved games |
| Top Online | Sun/Wed/Fri 03:00 UTC | Live player leaderboard |
| Top Offline | 1st + 15th of month 05:00 UTC | Leaderboard for single-player F2P concurrents |
| Update Reviews | Every 2 days 06:00 UTC | Reviews-only refresh |
| Update Reviews | Daily 06:00 UTC | Every game's review summary from Steam's store API |
| Refresh Store Data | Daily 16:00 UTC | Rotating re-fetch: tags, languages, DLC flag weekly; name, image, description, developer/publisher, release date, platforms, Metacritic, DRM monthly. Then every game's anti-cheat from Steam's disclosure (committed separately; manual run with `only: anticheat` does just that) |
| Refresh All-Time Peaks | Daily 19:30 UTC | `all_time_peak` from SteamCharts, every game once a month, plus our own samples; manual run with `backfill` asks about every game |
| Check Dead Links | Every 5 days 04:00 UTC | HEAD-request 404/410 scanner |
| Purge Unhealthy | Weekly Mon 05:00 UTC | Full health scan + removal |
| Purge Unhealthy | Daily 07:00 UTC | Store API screen of every game; removal once appdetails agrees |
| Mark Dead Games | Mon + Thu 04:30 UTC | Online + >1y + 0 players ≥14d → `is_dead=true` + 💀 note |
| Daily Snapshot | Daily 23:00 UTC | Per-game players and reviews into `data/snapshots/` |
| Force Re-fetch | Manual only | Re-fetch ALL games |
| Force Re-fetch | Manual only | Re-fetch games from scratch, one part (`1/8` … `8/8`) a run |
| Web CI / Python Tests | On `web/**` / `scripts/**` change | Type check, tests, build and output checks |
| Release Desktop / Android | On `desktop-v*` / `android-v*` tag | Build the apps into a draft release |
| Announce Discussion | On release `published` | Auto-create a GitHub Discussion in `Announcements` |
+170
View File
@@ -0,0 +1,170 @@
# Pipeline scale — the data workflows as the catalogue grows
## Why this note exists
On 2026-10-23 the maintainer asked for an assessment of the GitHub Actions
data workflows now that the catalogue grows by hundreds of games a day, a
direction for them, and the change. This is that assessment and the decisions
it led to. Every number below was measured on the day (`main` at 01a631bb,
9,847 games), with `gh run list` over each workflow's last 40 runs and live,
read-only requests to Steam.
The short version: three jobs asked Steam about every game one request at a
time, two seconds a game, and had reached GitHub's six-hour limit; the
`data-write` group cancelled half of the runs that waited for one of them;
and the runs that did wait worked on stale data. Steam answers fifty games a
request in its store API, and GitHub can now queue runs. The pipeline uses
both, and every long loop now stops in time to save.
## What the catalogue does to each job
The catalogue held 9,847 games and grew by about 280 a day (auto-approve is
capped at 500); Steam lists about 17,000 free games matching its filters, so
the plan is sized for 20,000.
A per-game loop cost about 2.05 s a game against store endpoints (a
1.2-2.0 s throttle, plus `process_batch`'s 15-30 s rest after every 50 games)
and about 1.0 s against the keyed Web API, of which the rests were half.
| Workflow | Before | Measured | At 20,000 games |
| --- | --- | --- | --- |
| Update Reviews (every 2 days) | `appreviews`, one game a request | 2026-10-23: stopped at exactly 6:00:00, nothing saved; 5 of the last 10 runs cancelled while waiting | impossible |
| Check Dead Links (every 5 days) | HEAD on every store page, flags 404/410 | 5.5 h on 2026-10-21; `dead_links.log` never existed and every game is `active`: Steam sends a delisted app's page to the store front, 200 | impossible |
| Purge Unhealthy (Mondays) | appdetails for every game, one answer enough to remove | 5.3 h on 2026-10-19 | impossible |
| Auto Update JSON (daily) | incomplete records only | 1 minute, but failed 4 days in 10 (below) | fine |
| Top Online (3 a week) | player count for each online game | 48 min (2,898 games) | ~1.7 h |
| Top Offline (twice a month) | player count for each offline game | 88 min (6,949 games) | ~4 h |
| Refresh Store Data (daily) | a 7-day / 30-day rotation | 65 min of a 120-minute timeout | ~2.2 h: over |
| Refresh All-Time Peaks (daily) | a 30-day rotation on SteamCharts | 14 min of a 30-minute timeout | ~29 min: at the edge |
| Force Re-fetch (manual) | everything in one run | ~6.5 s a game, 18 hours | impossible |
Three problems were not about speed:
- **A stopped job is silent.** A job killed at its timeout commits nothing and
concludes `cancelled`. `notify-ci-failure.yml` hands runs to a reusable
workflow in `poli0981/.github` that reports only `failure`, so nobody heard
that Update Reviews had stopped working.
- **The `data-write` group kept ONE pending run.** A run that queued behind a
running job cancelled the one already waiting. The ingest's three-hourly
schedule made that happen every morning: Update Reviews was cancelled on
10-05, 10-11, 10-15, 10-19 and 10-21.
- **Queued runs worked on stale data.** A run's commit is fixed when the run
is created. Auto Update JSON is created at 00:00 together with the ingest's
schedule; when the ingest ran first and committed, Auto Update JSON wrote
shards from before that commit, and `commit_push.sh`'s rebase failed on
`data/index.json` (10-15, 10-16, 10-17, 10-18). Only three workflows checked
out `ref: main`.
## What Steam's store API answers
`IStoreBrowseService/GetItems` (keyless, already used for gated games and the
anti-cheat sweep) answered for the whole catalogue in 99 requests and 105
seconds (100 ids a request; 200 also worked, 500 is refused at the edge for
the URL's length). Each item carries what the slow jobs asked for:
- `reviews.summary_filtered`: `review_count`, `percent_positive`,
`review_score_label` - the summary the store shows, in the vocabulary
`appreviews` uses ("3 user reviews", "Mixed").
- `visible`, `unlisted` (true for delisted apps), `is_free` (absent when not
free), `release.is_coming_soon`, `type`.
Compared with the stored reviews (read from `appreviews` on 10-17, rounded):
5,296 identical, 3,795 exactly one point lower - Steam floors its percentage
(85 for Counter-Strike 2, which `round()` had made 86) - 287 with a new label,
and 4 where GetItems said 0 reviews for a game the store page shows with
thousands (845070) or a few. Its 0 is not to be trusted alone.
Screening every game the way purge would: 3 `unlisted` (appdetails calls them
not free: removed), 5 without `is_free` (appdetails calls two of them free:
kept), nothing hidden, unknown or coming soon. The dry run of the new purge on
the real catalogue matched: 8 flagged, 3 removed, 5 kept by appdetails.
Of the 142 games removed before, GetItems now sees 27 as free and listed
again. Most were never really gone: 14 were rejected by the ingest in two
bursts on 2026-09-27, when appdetails answered `success:false` for live games,
and were approved again later. Eight are still out; the new purge names them.
GitHub Actions has supported `concurrency.queue: max` since 2026-05-07: up to
100 waiting runs, first in, first out, with `cancel-in-progress: false`.
## The direction
1. **Queue, don't evict.** Every data-write workflow has `queue: max` and
checks out `ref: main`, so a run that waits loses nothing and works on what
the run before it committed.
2. **Batch, don't crawl.** Reviews and the purge screen come from GetItems:
about 200 requests for the catalogue, whatever its size within reason.
Requests per game are kept for confirming a removal and for the rare
review count of 0.
3. **Bound every run.** A job sets `JOB_BUDGET_MINUTES` below its timeout;
a loop past it starts no further game and the script saves what it did.
Loops visit games in an order that changes daily, rotations grow their
cycle (a multiple of the base) instead of their daily slice, and a
`::warning::` says when a loop used more than three quarters of its budget.
4. **Drop what does nothing.** Check Dead Links goes, and so do the rests
between batches of the keyed Web API.
Considered and left: a matrix of runners per job (still a request per game,
parts racing each other's pushes, and SonarCloud's S8233 on workflow-level
write permissions with more than one job), a self-hosted runner, the Worker's
cron (the Worker may never write a shard), and a second opinion from GetItems
for the ingest's rejections (a skipped link and a rejected one both end as a
`failed` queue row; only the public log line differs).
## What changed
| Job | Now | Time at 9,847 | At 20,000 |
| --- | --- | --- | --- |
| Update Reviews | daily; GetItems, `appreviews` only for a 0 (at most 300) | ~2-5 min | ~5 min |
| Purge Unhealthy | daily; GetItems screen, appdetails confirms each flag | ~3-8 min | ~5-10 min |
| Check Dead Links | retired | 0 | 0 |
| Top Online / Offline | no rests between batches | ~27 / ~64 min | ~55 min / ~2.1 h |
| Refresh Store Data | cycles 7 → 14 past 2,000 pages a day, 30 → 60 past 500 appdetails | ~65 min | ~65-90 min |
| Refresh All-Time Peaks | cycle 30 → 60 past 400 pages a day | ~14 min | ~15 min |
| Force Re-fetch | parts `1/8` … `8/8`, queued | ~2.2 h a part | ~2.2 h a part (`1/16`…) |
Rules worth keeping (CLAUDE.md has them in full):
- **Reviews.** Steam's floored percentage, in integers (`29/100*100` is 28.99…
in floating point). A GetItems count of 0 never writes; a stored percentage
or "N/A" is asked on `appreviews` instead. No answer keeps the value.
`last_updated` moves on a change, and on a heartbeat every 14-20 days for a
game that was read: the site's "stale" badge (30 days) used to be kept quiet
by this job stamping every game every run, which also rewrote every shard.
- **Purge.** Remove only when GetItems flags a game AND `check_game_health()`
agrees; ask an `unavailable` again five minutes later; never treat a failed
batch or a network error as "gone"; stop before writing anything past 500
flags or 100 removals; log as before; name earlier removals Steam has since
undone. A purged game keeps its D1 decision, so discovery never proposes it
again - the notice is the way back.
- **Budgets.** Never for the ingest: `clear_temp()` empties its whole queue.
Refetch clears a game on a copy and swaps it in only once appdetails has
answered, oldest first, so a part cut short resumes where it stopped.
`web/shared/pipeline.test.ts` now fails on a data-write workflow without
`queue: max`, `cancel-in-progress: false`, `ref: main` on every checkout, a
`timeout-minutes`, or a budget less than 20 minutes below it.
## Load, before and after
| | Before | After |
| --- | --- | --- |
| Time the data-write group is busy | ~46 h a week | ~12 h a week |
| Requests to Steam | ~82,000 a week | ~25,000 a week |
| Runs cancelled while waiting | about one a day | none |
## Left for later
- **A timed-out run is still silent.** The reusable notifier in
`poli0981/.github` reports `failure` only; reporting `cancelled` for the
data workflows (now that waiting no longer cancels anything) belongs there.
- **Workers Builds deploys on data commits** (`docs/DEPLOYMENT.md`, "Which
commits deploy"). Daily reviews and purges add about one deploy a day until
the watch paths are set.
- **`data/snapshots/`** grows by 1.9 MB a day and nothing in the repository
reads it. A leaner format (no name, no zero rows) would quarter it.
- **The keyed player counts** are the last per-game loop. At 20,000 games
Top Offline needs about two hours; past that, sample games with no players
on a rotation instead of every run.
- **Re-queue what Steam has undone**: the purge's notice lists the removed
games that are free and on the store again.