docs: the pipeline-scale assessment, CLAUDE.md, README, CHANGELOG
docs/plan/18-pipeline-scale.md holds the measurements (9,847 games, +~280 a day), what each job cost, what GetItems answers, the direction and what is left. CLAUDE.md gains the rules for reviews, the purge, budgets and the data-write group; README the new schedules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
529b8603c3
commit
0897443bd6
4 files changed
+311
-31
No files matched your search
@@ -4,6 +4,43 @@ All notable changes to this awesome noob repo will be documented here.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### 🧰 Pipeline
|
||||
|
||||
- **The data jobs scale with the catalogue again.** With almost 10,000 games
|
||||
and a few hundred more every day, three jobs that asked Steam about one
|
||||
game at a time had reached GitHub's six-hour limit: on 2026-10-23 Update
|
||||
Reviews was stopped at exactly six hours with nothing saved. The full
|
||||
assessment, with every measurement, is `docs/plan/18-pipeline-scale.md`.
|
||||
- **Reviews come from Steam's store API, for every game, every day.** Fifty
|
||||
games a request, about two minutes for the catalogue, instead of every two
|
||||
days and five and a half hours. The percentage is now Steam's own, which
|
||||
rounds down: about 4,000 games show one point less once (Counter-Strike 2
|
||||
"85%" instead of "86%"), exactly as on their store pages. A summary that
|
||||
says 0 reviews is double-checked game by game before it replaces anything.
|
||||
- **The daily purge needs two answers to remove a game.** Steam's store API
|
||||
screens every game in a few minutes, and a game is removed only when
|
||||
appdetails agrees; "unavailable" is asked twice, five minutes apart. A run
|
||||
that would remove more than 100 games, or flags more than 500, stops before
|
||||
writing anything. It also lists earlier removals that Steam has since
|
||||
undone. It runs daily now, instead of a five-hour sweep on Mondays.
|
||||
- **Check Dead Links is retired.** In its whole life it never found a dead
|
||||
link: Steam sends a delisted game's page to the store front, which answers
|
||||
200. The purge's screen sees delisted games (`unlisted`) instead.
|
||||
- **No run is cancelled for waiting.** The data jobs share one queue that
|
||||
used to hold a single waiting run: a third job cancelled it, and Update
|
||||
Reviews lost five of ten runs that way. The queue now holds them all, first
|
||||
in, first out, and every job checks out the latest `main` when it starts -
|
||||
Auto Update JSON failed four days in ten from working on the data as it was
|
||||
when the run was queued.
|
||||
- **Every long job stops in time to save.** Each workflow gives its job a time
|
||||
budget below its timeout: past it the job saves and commits what it did
|
||||
instead of being stopped with nothing, and it warns before the budget gets
|
||||
tight. The leaderboards skip the pauses meant for store pages (Top Online
|
||||
~27 minutes instead of 48). The store-data and all-time-peak rotations
|
||||
stretch their cycle as the catalogue grows instead of their daily runs.
|
||||
- **Force Re-fetch runs in parts** (`1/8` … `8/8`, queued one after another),
|
||||
and a game whose fetch fails keeps its old data.
|
||||
|
||||
### 🧰 Tooling
|
||||
|
||||
- **Desktop releases keep reaching the updater.** tauri-action v1 writes each
|
||||
|
||||
@@ -222,7 +222,13 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
(tags, languages, `language_details`, `has_paid_dlc`) every 7 days,
|
||||
appdetails (name, header image, description, developer/publisher, release
|
||||
date, platforms, Metacritic, DRM notes) every 30. Never `appid % cycle`:
|
||||
appids are mostly multiples of 10, which leaves most slices empty. It
|
||||
appids are mostly multiples of 10, which leaves most slices empty. A cycle
|
||||
grows with the catalogue instead of the run (`rotation.cycle_for()`: past
|
||||
`PAGE_DAILY` 2,000 pages or `DETAILS_DAILY` 500 appdetails a day it becomes
|
||||
14, 21, ... or 60, 90, ...) and is always a MULTIPLE of its base, so a game
|
||||
keeps its weekday and its day of the month and visits only ever thin out.
|
||||
The slice is visited in `rotation.daily_order()`, so a run that meets its
|
||||
time budget leaves a different tail each day. It
|
||||
replaces values only with NON-EMPTY ones, never writes `MANUAL_FIELDS` or
|
||||
`notes`, trusts `has_paid_dlc` only from a real store page (an age gate has
|
||||
no language table or tags) or a gated game's DLC prices, compares every
|
||||
@@ -231,6 +237,9 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
counting as news), and bumps `last_updated` only on a real change. A game
|
||||
with empty tags, languages or `language_details` is in the page group EVERY
|
||||
day, first, capped at `MISSING_DAILY_CAP`; `--missing-only` runs just those.
|
||||
A game whose appdetails no longer call it free is LISTED in a `::warning::`,
|
||||
never written: purge removes a game only when the store API flags it too,
|
||||
and this is where the opposite case shows.
|
||||
A new field added to it must obey the same rules;
|
||||
`scripts/tests/test_refresh_store_data.py` holds them.
|
||||
- **A store page behind Steam's age gate is never parsed.** An 'Adult Only'
|
||||
@@ -252,7 +261,8 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
with `l=english`: without it, a runner was once answered in Russian.
|
||||
- **`all_time_peak` only ever goes UP, and `refresh_peaks.py` is what raises
|
||||
it** (daily 19:30 UTC, "Refresh All-Time Peaks"). A `crc32(appid) % 30`
|
||||
slice (`core/rotation.py`, shared with `refresh_store_data.py`) reads
|
||||
slice (`core/rotation.py`, shared with `refresh_store_data.py`; 60, 90, ...
|
||||
past `PEAK_DAILY` 400 pages a day, never shorter) reads
|
||||
SteamCharts' recorded all-time peak - the only source older than our own
|
||||
sampling: SteamDB forbids scraping and Valve's APIs know only today. Every
|
||||
run also raises it from our samples (`peak_today`, which despite its name is
|
||||
@@ -260,8 +270,9 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
`apply_players()` raises it on every sample, never from a 0. SteamCharts has
|
||||
no API, terms or robots.txt, so: one page per game a month, 2-3 s apart, a
|
||||
User-Agent that names the project; 500 is an app it does not track (most of
|
||||
the long tail), 404 one it has never seen; a 403, 429 or challenge page
|
||||
stops the run with a warning, not a failure. `refetch_all.py` must never
|
||||
the long tail), 404 one it has never seen; a 403, 429 or challenge page,
|
||||
or the job's time budget, stops the run with a warning, not a failure.
|
||||
`refetch_all.py` must never
|
||||
clear the field - it cannot be fetched again - and nothing may lower it.
|
||||
The backfill is a manual run with `backfill` ticked (~4 h, start it in the
|
||||
evening UTC). DISCLAIMER section 6 names the source, so rewording it reopens
|
||||
@@ -293,6 +304,42 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
name and the new key are the same family, and a known label stays as it was
|
||||
written (so "Easy Anti-Cheat" and "EAC" both occur; the site groups them).
|
||||
`scripts/tests/test_anticheat.py` and `test_refresh_anticheat.py` hold it.
|
||||
- **Reviews are Steam's own summary, read for EVERY game daily**
|
||||
(`update_reviews.py`, "Update Reviews" 06:00): `GetItems` with
|
||||
`include_reviews`, fifty games a request (`SteamClient.fetch_items()`, the
|
||||
sweep the anti-cheat and purge jobs share), ~2 minutes. It asked
|
||||
`appreviews` one game at a time until v4.4.0 and, at 9,800 games, ran into
|
||||
GitHub's six-hour limit with nothing saved. Rules: the percentage is Steam's,
|
||||
FLOORED (`reviews_from_summary()` floors appreviews in integers too, so an
|
||||
ingest and the refresh agree; until v4.4.0 it was rounded, one point higher
|
||||
on ~40% of games); a GetItems count of 0 is never written - its summary has
|
||||
said 0 for a game with thousands (845070) - so a stored percentage or "N/A"
|
||||
is asked on appreviews (`ASK_CAP` 300 a run, daily order) and only that
|
||||
answer is written; no item, no usable summary or a failed batch keeps the
|
||||
value. A record's `last_updated` moves on a change and on a HEARTBEAT: a
|
||||
game read whose stamp is 14-20 days old (crc32 jitter) is stamped again.
|
||||
The site calls a record older than 30 days stale (`web/src/lib/validation.ts`,
|
||||
`/health`), and this job used to stamp every game on every run; the
|
||||
heartbeat keeps "the pipeline confirmed this lately" true without rewriting
|
||||
every record daily. `scripts/tests/test_update_reviews.py` holds it.
|
||||
- **A game is purged only when two of Steam's answers agree**
|
||||
(`purge_unhealthy.py`, "Purge Unhealthy Games" daily 07:00). GetItems
|
||||
screens the whole catalogue (`store_items.screen_item()`: unknown to Steam,
|
||||
`visible` false, `unlisted`, `release.is_coming_soon`, no `is_free`; never
|
||||
`type` - a free mod is listed on purpose), and `check_game_health()`
|
||||
(appdetails) must confirm each flag. An `unavailable`/404 answer is asked
|
||||
again `RECHECK_AFTER` (5 min) later: appdetails answered `success:false` for
|
||||
14 live games in two bursts on 2026-09-27. A failed batch or a network error
|
||||
is never "gone"; blocklisted games are not asked about. More than 500 flags
|
||||
or 100 confirmed removals exits non-zero BEFORE anything is written
|
||||
(`--max-candidates`, `--max-removals` to override). Removals are logged to
|
||||
`removed_games.jsonl` exactly as before - `reconcile.ts` reads them. A
|
||||
`::notice::` names earlier removals GetItems now sees free, listed and
|
||||
released (re-queue the link to bring one back: a purged game keeps its D1
|
||||
decision and discovery never proposes it again). Check Dead Links is gone:
|
||||
HEAD on a delisted app's page ends on the store front with 200, so in its
|
||||
whole life it never found a dead link, and it took 5.5 hours.
|
||||
`scripts/tests/test_purge_unhealthy.py` holds it.
|
||||
- `ingest_new.py` only reads `scripts/temp_info.jsonl`, and a line is a store
|
||||
LINK and nothing else: every other key is ignored and the game is built from
|
||||
Steam (`merge_extension_data()` went with the archived browser extension;
|
||||
@@ -305,14 +352,15 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
appdetails calls other than `type: "game"` (`status_code: not_a_game` in
|
||||
`removed_games.jsonl`) - the check that makes auto-approve safe, and why a
|
||||
queued link cannot add a free DLC, demo or tool. The check lives in ingest
|
||||
only: in `health_checker` it would make Purge delete old records every week.
|
||||
only: in `health_checker` it would make Purge delete old records every day.
|
||||
- **`ingest-new.yml` is also the removal job.** `bash/ingest.sh` runs
|
||||
`remove_games.py` (applies `data/blocklist.jsonl`) and then
|
||||
`ingest_new.py`, and commits once ("Apply removals" when `temp_info` did not
|
||||
change). A separate removal workflow would keep evicting the ingest from
|
||||
`data-write`'s single pending slot, and the ingest it. The workflow checks
|
||||
out `ref: main`, not the triggering SHA: a run that waited its turn would
|
||||
otherwise work on the data as it was when it was queued.
|
||||
change), so one commit to either file starts one run that settles both.
|
||||
The workflow checks out `ref: main`, not the triggering SHA, like every
|
||||
data-write workflow: a run that waited its turn would otherwise work on the
|
||||
data as it was when it was queued. It has no time budget: `clear_temp()`
|
||||
empties the whole queue file, so a game it had not reached would be lost.
|
||||
- New games ARE discovered automatically now, by `scripts/discover_new.py`
|
||||
into the D1 review queue — but discovery only PROPOSES. Publication is
|
||||
decided in `/admin`: by a reviewer, batch by batch, or by the auto-approve
|
||||
@@ -329,8 +377,10 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
page previews is the SQL that runs). The empty-file gate is checked again
|
||||
inside the commit (`requireEmptyQueue`), so a batch never lands mid-ingest;
|
||||
"no row is approved" would have stalled it for the 12 hours a skipped row
|
||||
stays approved. The idle gate exists because a push while a data job waits
|
||||
EVICTS that job. `shared/pipeline.ts` lists every `data-write` workflow by
|
||||
stays approved. The idle gate dates from when a push while a data job waited
|
||||
EVICTED that job; with `queue: max` the ingest would only queue behind it,
|
||||
and the gate stays so a batch lands when its ingest can start at once.
|
||||
`shared/pipeline.ts` lists every `data-write` workflow by
|
||||
file, and `pipeline.test.ts` fails when a workflow joins the group without
|
||||
joining the list. Settings, limits and their validation are
|
||||
`shared/automation.ts`, imported by the Worker and the page. A scheduled
|
||||
@@ -393,16 +443,40 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
(wrangler.jsonc, 180). `docs/PRIVACY_POLICY.md` states the number: change
|
||||
both together. `ingest_queue` and `ingest_decisions` are never pruned — the
|
||||
decisions are what keep a rejected game out of the discovery sweep.
|
||||
- Every workflow that writes to `data/` shares `concurrency: {group: data-write}`.
|
||||
A new data-writing workflow without it will race jobs that run ~2 hours.
|
||||
The group holds ONE pending run - a second queueing run cancels the first -
|
||||
and scheduled runs here start 2-5 hours after their cron, so a new schedule
|
||||
goes where nothing long is running (the long jobs - reviews, dead links,
|
||||
purge - start mid-morning UTC and run 2-4 hours; Refresh Store Data is at
|
||||
16:00 for that reason).
|
||||
- **Every workflow that writes to `data/` shares `concurrency: {group:
|
||||
data-write, cancel-in-progress: false, queue: max}`, checks out `ref: main`,
|
||||
and declares `timeout-minutes`.** `pipeline.test.ts` fails on a data-write
|
||||
workflow without any of them. A new data-writing workflow outside the group
|
||||
would race the others.
|
||||
- `queue: max` (GitHub, 2026-05-07) keeps up to 100 runs waiting, first in,
|
||||
first out. The default keeps ONE: a run queueing behind a running job
|
||||
cancelled the one already waiting - Update Reviews lost five of ten runs
|
||||
that way in October 2026. Mixing the two settings in one group is
|
||||
undocumented, so every member has `queue: max`.
|
||||
- `ref: main`: a run's commit is fixed when the run is CREATED. A scheduled
|
||||
run that waited behind another data job wrote shards from before that
|
||||
job's commit, and `commit_push.sh`'s rebase then conflicted on
|
||||
`data/index.json` - Auto Update JSON failed four days in ten that way.
|
||||
- Scheduled runs start up to an hour after their cron (2-5 hours in
|
||||
September). Waiting costs nothing now, but a long new job still goes where
|
||||
nothing long runs (Refresh Store Data is at 16:00 for that reason).
|
||||
- **Every long loop has a time budget** (`scripts/core/budget.py`). The
|
||||
workflow sets `JOB_BUDGET_MINUTES` at least 20 below its `timeout-minutes`
|
||||
(`pipeline.test.ts` checks); past it a loop starts no further game, the
|
||||
script saves what it did and the job commits it, and a loop that used more
|
||||
than three quarters says so in a `::warning::`. Loops visit their games in
|
||||
`rotation.daily_order()`, so a cut falls elsewhere the next day. A job
|
||||
stopped at its timeout commits nothing and concludes `cancelled`, which
|
||||
`notify-ci-failure.yml` (a reusable workflow in `poli0981/.github`) does not
|
||||
report: the budget is what keeps a growing catalogue from failing silently.
|
||||
`process_batch(pause=False)` for the keyed Web API (player counts); the
|
||||
15-30 s rest between batches stays for store pages and appdetails. Never a
|
||||
budget for `ingest_new.py`. `refetch_all.py` runs in parts (`--part K/N`)
|
||||
and clears each game on a copy that replaces it only once appdetails has
|
||||
answered, so a cut part leaves every other game as it was.
|
||||
- **Every data job commits through `bash/commit_push.sh`.** The group
|
||||
serialises workflows, not `main`: a merged PR or an /admin commit can land
|
||||
during a four-hour job, and a bare `git push` is then rejected and the run
|
||||
during a long job, and a bare `git push` is then rejected and the run
|
||||
thrown away (purge-unhealthy, 2026-09-21). The helper rebases onto
|
||||
`origin/main` and retries; none of those writers touch the shards, so the
|
||||
rebase is clean. A new wrapper must use it, once per commit (Refresh Store
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
[](https://free-steam-games.win/)
|
||||
[](https://github.com/poli0981/free-steam-games-list/releases)
|
||||
[](https://github.com/poli0981/free-steam-games-list/releases)
|
||||
[](.github/workflows/purge-unhealthy.yml)
|
||||
[](.github/workflows/purge-unhealthy.yml)
|
||||

|
||||
[](.github/FUNDING.yml)
|
||||
|
||||
@@ -89,8 +89,8 @@ The Python pipeline that powered v2 is untouched and runs daily as before:
|
||||
|
||||
- Schema v2.2 (23 + manual fields per record), CC BY 4.0 (see [LICENSE-DATA](LICENSE-DATA)), sharded data in `data/data_*.jsonl` (cap 800 records/shard) plus a `data/index.json` manifest with a SHA-256 per shard.
|
||||
- HTML scraper for accurate language tables, DLC pricing, user tags and the developer's anti-cheat disclosure — single GET per game.
|
||||
- Health-checked: dead-link sweep every 5 days, full purge every Monday.
|
||||
- Reviews-only refresh every 2 days; full refresh daily.
|
||||
- Health-checked daily: Steam's store API screens every game, and a game is removed only when appdetails agrees.
|
||||
- Reviews refreshed daily from Steam's own summaries; full refresh daily. Every data job queues for its turn and has a time budget, so a growing catalogue slows a refresh down instead of breaking it ([plan](docs/plan/18-pipeline-scale.md)).
|
||||
|
||||
</details>
|
||||
|
||||
@@ -151,7 +151,8 @@ scripts/
|
||||
│ ├── anticheat.py # Steam's anti-cheat disclosure and the rule that applies it
|
||||
│ ├── steamcharts.py # SteamCharts all-time peak reader (polite, stops on refusal)
|
||||
│ ├── peaks.py # all_time_peak: only ever raised
|
||||
│ ├── rotation.py # crc32 daily slices shared by the refresh jobs
|
||||
│ ├── rotation.py # crc32 daily slices shared by the refresh jobs, growing with the catalogue
|
||||
│ ├── budget.py # A job's time budget: stop in time to save, warn before it runs out
|
||||
│ └── health_checker.py # Game status detection with cached API data
|
||||
├── discover_new.py # Propose new games into the /admin review queue
|
||||
├── ingest_new.py # Add queued games (health check + fetch from Steam)
|
||||
@@ -160,13 +161,12 @@ scripts/
|
||||
├── refresh_store_data.py # Daily rotation: tags/languages weekly, store details monthly
|
||||
├── refresh_anticheat.py # Daily: every game's anti-cheat, as its developer declares it on Steam
|
||||
├── refresh_peaks.py # Daily rotation: all-time peak from SteamCharts, monthly per game
|
||||
├── update_reviews.py # Reviews-only (every 2 days)
|
||||
├── update_reviews.py # Daily: every game's review summary from Steam, ~2 minutes
|
||||
├── mark_dead_games.py # Flag online games with no players
|
||||
├── check_dead_links.py # HEAD-request 404/410 scanner
|
||||
├── purge_unhealthy.py # Remove delisted/not-free/invalid
|
||||
├── purge_unhealthy.py # Daily: remove delisted/not-free/unreleased (store API screen + appdetails)
|
||||
├── top_online.py / top_offline.py # Leaderboards
|
||||
├── edit_game.py # Overrides from the command line
|
||||
├── refetch_all.py # Force re-fetch ALL (manual only)
|
||||
├── refetch_all.py # Force re-fetch, in parts (manual only)
|
||||
└── delete_game.py # Remove games (blocklist) from the command line
|
||||
```
|
||||
|
||||
@@ -181,14 +181,13 @@ scripts/
|
||||
| Ingest New | On `scripts/temp_info.jsonl` or `data/blocklist.jsonl` push, and every 3 h | Apply removals, then add approved games |
|
||||
| Top Online | Sun/Wed/Fri 03:00 UTC | Live player leaderboard |
|
||||
| Top Offline | 1st + 15th of month 05:00 UTC | Leaderboard for single-player F2P concurrents |
|
||||
| Update Reviews | Every 2 days 06:00 UTC | Reviews-only refresh |
|
||||
| Update Reviews | Daily 06:00 UTC | Every game's review summary from Steam's store API |
|
||||
| Refresh Store Data | Daily 16:00 UTC | Rotating re-fetch: tags, languages, DLC flag weekly; name, image, description, developer/publisher, release date, platforms, Metacritic, DRM monthly. Then every game's anti-cheat from Steam's disclosure (committed separately; manual run with `only: anticheat` does just that) |
|
||||
| Refresh All-Time Peaks | Daily 19:30 UTC | `all_time_peak` from SteamCharts, every game once a month, plus our own samples; manual run with `backfill` asks about every game |
|
||||
| Check Dead Links | Every 5 days 04:00 UTC | HEAD-request 404/410 scanner |
|
||||
| Purge Unhealthy | Weekly Mon 05:00 UTC | Full health scan + removal |
|
||||
| Purge Unhealthy | Daily 07:00 UTC | Store API screen of every game; removal once appdetails agrees |
|
||||
| Mark Dead Games | Mon + Thu 04:30 UTC | Online + >1y + 0 players ≥14d → `is_dead=true` + 💀 note |
|
||||
| Daily Snapshot | Daily 23:00 UTC | Per-game players and reviews into `data/snapshots/` |
|
||||
| Force Re-fetch | Manual only | Re-fetch ALL games |
|
||||
| Force Re-fetch | Manual only | Re-fetch games from scratch, one part (`1/8` … `8/8`) a run |
|
||||
| Web CI / Python Tests | On `web/**` / `scripts/**` change | Type check, tests, build and output checks |
|
||||
| Release Desktop / Android | On `desktop-v*` / `android-v*` tag | Build the apps into a draft release |
|
||||
| Announce Discussion | On release `published` | Auto-create a GitHub Discussion in `Announcements` |
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
# Pipeline scale — the data workflows as the catalogue grows
|
||||
|
||||
## Why this note exists
|
||||
|
||||
On 2026-10-23 the maintainer asked for an assessment of the GitHub Actions
|
||||
data workflows now that the catalogue grows by hundreds of games a day, a
|
||||
direction for them, and the change. This is that assessment and the decisions
|
||||
it led to. Every number below was measured on the day (`main` at 01a631bb,
|
||||
9,847 games), with `gh run list` over each workflow's last 40 runs and live,
|
||||
read-only requests to Steam.
|
||||
|
||||
The short version: three jobs asked Steam about every game one request at a
|
||||
time, two seconds a game, and had reached GitHub's six-hour limit; the
|
||||
`data-write` group cancelled half of the runs that waited for one of them;
|
||||
and the runs that did wait worked on stale data. Steam answers fifty games a
|
||||
request in its store API, and GitHub can now queue runs. The pipeline uses
|
||||
both, and every long loop now stops in time to save.
|
||||
|
||||
## What the catalogue does to each job
|
||||
|
||||
The catalogue held 9,847 games and grew by about 280 a day (auto-approve is
|
||||
capped at 500); Steam lists about 17,000 free games matching its filters, so
|
||||
the plan is sized for 20,000.
|
||||
|
||||
A per-game loop cost about 2.05 s a game against store endpoints (a
|
||||
1.2-2.0 s throttle, plus `process_batch`'s 15-30 s rest after every 50 games)
|
||||
and about 1.0 s against the keyed Web API, of which the rests were half.
|
||||
|
||||
| Workflow | Before | Measured | At 20,000 games |
|
||||
| --- | --- | --- | --- |
|
||||
| Update Reviews (every 2 days) | `appreviews`, one game a request | 2026-10-23: stopped at exactly 6:00:00, nothing saved; 5 of the last 10 runs cancelled while waiting | impossible |
|
||||
| Check Dead Links (every 5 days) | HEAD on every store page, flags 404/410 | 5.5 h on 2026-10-21; `dead_links.log` never existed and every game is `active`: Steam sends a delisted app's page to the store front, 200 | impossible |
|
||||
| Purge Unhealthy (Mondays) | appdetails for every game, one answer enough to remove | 5.3 h on 2026-10-19 | impossible |
|
||||
| Auto Update JSON (daily) | incomplete records only | 1 minute, but failed 4 days in 10 (below) | fine |
|
||||
| Top Online (3 a week) | player count for each online game | 48 min (2,898 games) | ~1.7 h |
|
||||
| Top Offline (twice a month) | player count for each offline game | 88 min (6,949 games) | ~4 h |
|
||||
| Refresh Store Data (daily) | a 7-day / 30-day rotation | 65 min of a 120-minute timeout | ~2.2 h: over |
|
||||
| Refresh All-Time Peaks (daily) | a 30-day rotation on SteamCharts | 14 min of a 30-minute timeout | ~29 min: at the edge |
|
||||
| Force Re-fetch (manual) | everything in one run | ~6.5 s a game, 18 hours | impossible |
|
||||
|
||||
Three problems were not about speed:
|
||||
|
||||
- **A stopped job is silent.** A job killed at its timeout commits nothing and
|
||||
concludes `cancelled`. `notify-ci-failure.yml` hands runs to a reusable
|
||||
workflow in `poli0981/.github` that reports only `failure`, so nobody heard
|
||||
that Update Reviews had stopped working.
|
||||
- **The `data-write` group kept ONE pending run.** A run that queued behind a
|
||||
running job cancelled the one already waiting. The ingest's three-hourly
|
||||
schedule made that happen every morning: Update Reviews was cancelled on
|
||||
10-05, 10-11, 10-15, 10-19 and 10-21.
|
||||
- **Queued runs worked on stale data.** A run's commit is fixed when the run
|
||||
is created. Auto Update JSON is created at 00:00 together with the ingest's
|
||||
schedule; when the ingest ran first and committed, Auto Update JSON wrote
|
||||
shards from before that commit, and `commit_push.sh`'s rebase failed on
|
||||
`data/index.json` (10-15, 10-16, 10-17, 10-18). Only three workflows checked
|
||||
out `ref: main`.
|
||||
|
||||
## What Steam's store API answers
|
||||
|
||||
`IStoreBrowseService/GetItems` (keyless, already used for gated games and the
|
||||
anti-cheat sweep) answered for the whole catalogue in 99 requests and 105
|
||||
seconds (100 ids a request; 200 also worked, 500 is refused at the edge for
|
||||
the URL's length). Each item carries what the slow jobs asked for:
|
||||
|
||||
- `reviews.summary_filtered`: `review_count`, `percent_positive`,
|
||||
`review_score_label` - the summary the store shows, in the vocabulary
|
||||
`appreviews` uses ("3 user reviews", "Mixed").
|
||||
- `visible`, `unlisted` (true for delisted apps), `is_free` (absent when not
|
||||
free), `release.is_coming_soon`, `type`.
|
||||
|
||||
Compared with the stored reviews (read from `appreviews` on 10-17, rounded):
|
||||
5,296 identical, 3,795 exactly one point lower - Steam floors its percentage
|
||||
(85 for Counter-Strike 2, which `round()` had made 86) - 287 with a new label,
|
||||
and 4 where GetItems said 0 reviews for a game the store page shows with
|
||||
thousands (845070) or a few. Its 0 is not to be trusted alone.
|
||||
|
||||
Screening every game the way purge would: 3 `unlisted` (appdetails calls them
|
||||
not free: removed), 5 without `is_free` (appdetails calls two of them free:
|
||||
kept), nothing hidden, unknown or coming soon. The dry run of the new purge on
|
||||
the real catalogue matched: 8 flagged, 3 removed, 5 kept by appdetails.
|
||||
|
||||
Of the 142 games removed before, GetItems now sees 27 as free and listed
|
||||
again. Most were never really gone: 14 were rejected by the ingest in two
|
||||
bursts on 2026-09-27, when appdetails answered `success:false` for live games,
|
||||
and were approved again later. Eight are still out; the new purge names them.
|
||||
|
||||
GitHub Actions has supported `concurrency.queue: max` since 2026-05-07: up to
|
||||
100 waiting runs, first in, first out, with `cancel-in-progress: false`.
|
||||
|
||||
## The direction
|
||||
|
||||
1. **Queue, don't evict.** Every data-write workflow has `queue: max` and
|
||||
checks out `ref: main`, so a run that waits loses nothing and works on what
|
||||
the run before it committed.
|
||||
2. **Batch, don't crawl.** Reviews and the purge screen come from GetItems:
|
||||
about 200 requests for the catalogue, whatever its size within reason.
|
||||
Requests per game are kept for confirming a removal and for the rare
|
||||
review count of 0.
|
||||
3. **Bound every run.** A job sets `JOB_BUDGET_MINUTES` below its timeout;
|
||||
a loop past it starts no further game and the script saves what it did.
|
||||
Loops visit games in an order that changes daily, rotations grow their
|
||||
cycle (a multiple of the base) instead of their daily slice, and a
|
||||
`::warning::` says when a loop used more than three quarters of its budget.
|
||||
4. **Drop what does nothing.** Check Dead Links goes, and so do the rests
|
||||
between batches of the keyed Web API.
|
||||
|
||||
Considered and left: a matrix of runners per job (still a request per game,
|
||||
parts racing each other's pushes, and SonarCloud's S8233 on workflow-level
|
||||
write permissions with more than one job), a self-hosted runner, the Worker's
|
||||
cron (the Worker may never write a shard), and a second opinion from GetItems
|
||||
for the ingest's rejections (a skipped link and a rejected one both end as a
|
||||
`failed` queue row; only the public log line differs).
|
||||
|
||||
## What changed
|
||||
|
||||
| Job | Now | Time at 9,847 | At 20,000 |
|
||||
| --- | --- | --- | --- |
|
||||
| Update Reviews | daily; GetItems, `appreviews` only for a 0 (at most 300) | ~2-5 min | ~5 min |
|
||||
| Purge Unhealthy | daily; GetItems screen, appdetails confirms each flag | ~3-8 min | ~5-10 min |
|
||||
| Check Dead Links | retired | 0 | 0 |
|
||||
| Top Online / Offline | no rests between batches | ~27 / ~64 min | ~55 min / ~2.1 h |
|
||||
| Refresh Store Data | cycles 7 → 14 past 2,000 pages a day, 30 → 60 past 500 appdetails | ~65 min | ~65-90 min |
|
||||
| Refresh All-Time Peaks | cycle 30 → 60 past 400 pages a day | ~14 min | ~15 min |
|
||||
| Force Re-fetch | parts `1/8` … `8/8`, queued | ~2.2 h a part | ~2.2 h a part (`1/16`…) |
|
||||
|
||||
Rules worth keeping (CLAUDE.md has them in full):
|
||||
|
||||
- **Reviews.** Steam's floored percentage, in integers (`29/100*100` is 28.99…
|
||||
in floating point). A GetItems count of 0 never writes; a stored percentage
|
||||
or "N/A" is asked on `appreviews` instead. No answer keeps the value.
|
||||
`last_updated` moves on a change, and on a heartbeat every 14-20 days for a
|
||||
game that was read: the site's "stale" badge (30 days) used to be kept quiet
|
||||
by this job stamping every game every run, which also rewrote every shard.
|
||||
- **Purge.** Remove only when GetItems flags a game AND `check_game_health()`
|
||||
agrees; ask an `unavailable` again five minutes later; never treat a failed
|
||||
batch or a network error as "gone"; stop before writing anything past 500
|
||||
flags or 100 removals; log as before; name earlier removals Steam has since
|
||||
undone. A purged game keeps its D1 decision, so discovery never proposes it
|
||||
again - the notice is the way back.
|
||||
- **Budgets.** Never for the ingest: `clear_temp()` empties its whole queue.
|
||||
Refetch clears a game on a copy and swaps it in only once appdetails has
|
||||
answered, oldest first, so a part cut short resumes where it stopped.
|
||||
|
||||
`web/shared/pipeline.test.ts` now fails on a data-write workflow without
|
||||
`queue: max`, `cancel-in-progress: false`, `ref: main` on every checkout, a
|
||||
`timeout-minutes`, or a budget less than 20 minutes below it.
|
||||
|
||||
## Load, before and after
|
||||
|
||||
| | Before | After |
|
||||
| --- | --- | --- |
|
||||
| Time the data-write group is busy | ~46 h a week | ~12 h a week |
|
||||
| Requests to Steam | ~82,000 a week | ~25,000 a week |
|
||||
| Runs cancelled while waiting | about one a day | none |
|
||||
|
||||
## Left for later
|
||||
|
||||
- **A timed-out run is still silent.** The reusable notifier in
|
||||
`poli0981/.github` reports `failure` only; reporting `cancelled` for the
|
||||
data workflows (now that waiting no longer cancels anything) belongs there.
|
||||
- **Workers Builds deploys on data commits** (`docs/DEPLOYMENT.md`, "Which
|
||||
commits deploy"). Daily reviews and purges add about one deploy a day until
|
||||
the watch paths are set.
|
||||
- **`data/snapshots/`** grows by 1.9 MB a day and nothing in the repository
|
||||
reads it. A leaner format (no name, no zero rows) would quarter it.
|
||||
- **The keyed player counts** are the last per-game loop. At 20,000 games
|
||||
Top Offline needs about two hours; past that, sample games with no players
|
||||
on a rotation instead of every run.
|
||||
- **Re-queue what Steam has undone**: the purge's notice lists the removed
|
||||
games that are free and on the store again.
|
||||
Reference in new issue
Block a user