Files
poli0981andClaude Opus 5.5 11fb7a5b43 fix(reviews): stop depending on appreviews, which now redirects
The first run of the new Update Reviews on main asked appreviews about the
12 games GetItems counted at 0, and all 12 failed: since 2026-10-23
store.steampowered.com/appreviews/<id>?json=1 answers 302 to the store
front, from GitHub's runners as from anywhere else (it worked on 10-17).
The 38 games the ingest added that day were listed with "N/A".

- fetch_full() falls back to the GetItems summary when appreviews gives
  none (reviews_from_store_api()), so an ingest gets the value the daily
  refresh would write.
- update_reviews: a game never read ("N/A", nothing) with a count of 0
  becomes "No reviews" without a question; the first count GetItems has
  replaces it. A stored percentage is still never replaced by a 0 without a
  second source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 22:44:57 +07:00

183 lines
10 KiB
Markdown

# Pipeline scale — the data workflows as the catalogue grows
## Why this note exists
On 2026-10-23 the maintainer asked for an assessment of the GitHub Actions
data workflows now that the catalogue grows by hundreds of games a day, a
direction for them, and the change. This is that assessment and the decisions
it led to. Every number below was measured on the day (`main` at 01a631bb,
9,847 games), with `gh run list` over each workflow's last 40 runs and live,
read-only requests to Steam.
The short version: three jobs asked Steam about every game one request at a
time, two seconds a game, and had reached GitHub's six-hour limit; the
`data-write` group cancelled half of the runs that waited for one of them;
and the runs that did wait worked on stale data. Steam answers fifty games a
request in its store API, and GitHub can now queue runs. The pipeline uses
both, and every long loop now stops in time to save.
## What the catalogue does to each job
The catalogue held 9,847 games and grew by about 280 a day (auto-approve is
capped at 500); Steam lists about 17,000 free games matching its filters, so
the plan is sized for 20,000.
A per-game loop cost about 2.05 s a game against store endpoints (a
1.2-2.0 s throttle, plus `process_batch`'s 15-30 s rest after every 50 games)
and about 1.0 s against the keyed Web API, of which the rests were half.
| Workflow | Before | Measured | At 20,000 games |
| --- | --- | --- | --- |
| Update Reviews (every 2 days) | `appreviews`, one game a request | 2026-10-23: stopped at exactly 6:00:00, nothing saved; 5 of the last 10 runs cancelled while waiting | impossible |
| Check Dead Links (every 5 days) | HEAD on every store page, flags 404/410 | 5.5 h on 2026-10-21; `dead_links.log` never existed and every game is `active`: Steam sends a delisted app's page to the store front, 200 | impossible |
| Purge Unhealthy (Mondays) | appdetails for every game, one answer enough to remove | 5.3 h on 2026-10-19 | impossible |
| Auto Update JSON (daily) | incomplete records only | 1 minute, but failed 4 days in 10 (below) | fine |
| Top Online (3 a week) | player count for each online game | 48 min (2,898 games) | ~1.7 h |
| Top Offline (twice a month) | player count for each offline game | 88 min (6,949 games) | ~4 h |
| Refresh Store Data (daily) | a 7-day / 30-day rotation | 65 min of a 120-minute timeout | ~2.2 h: over |
| Refresh All-Time Peaks (daily) | a 30-day rotation on SteamCharts | 14 min of a 30-minute timeout | ~29 min: at the edge |
| Force Re-fetch (manual) | everything in one run | ~6.5 s a game, 18 hours | impossible |
Three problems were not about speed:
- **A stopped job is silent.** A job killed at its timeout commits nothing and
concludes `cancelled`. `notify-ci-failure.yml` hands runs to a reusable
workflow in `poli0981/.github` that reports only `failure`, so nobody heard
that Update Reviews had stopped working.
- **The `data-write` group kept ONE pending run.** A run that queued behind a
running job cancelled the one already waiting. The ingest's three-hourly
schedule made that happen every morning: Update Reviews was cancelled on
10-05, 10-11, 10-15, 10-19 and 10-21.
- **Queued runs worked on stale data.** A run's commit is fixed when the run
is created. Auto Update JSON is created at 00:00 together with the ingest's
schedule; when the ingest ran first and committed, Auto Update JSON wrote
shards from before that commit, and `commit_push.sh`'s rebase failed on
`data/index.json` (10-15, 10-16, 10-17, 10-18). Only three workflows checked
out `ref: main`.
## What Steam's store API answers
`IStoreBrowseService/GetItems` (keyless, already used for gated games and the
anti-cheat sweep) answered for the whole catalogue in 99 requests and 105
seconds (100 ids a request; 200 also worked, 500 is refused at the edge for
the URL's length). Each item carries what the slow jobs asked for:
- `reviews.summary_filtered`: `review_count`, `percent_positive`,
`review_score_label` - the summary the store shows, in the vocabulary
`appreviews` uses ("3 user reviews", "Mixed").
- `visible`, `unlisted` (true for delisted apps), `is_free` (absent when not
free), `release.is_coming_soon`, `type`.
Compared with the stored reviews (read from `appreviews` on 10-17, rounded):
5,296 identical, 3,795 exactly one point lower - Steam floors its percentage
(85 for Counter-Strike 2, which `round()` had made 86) - 287 with a new label,
and 4 where GetItems said 0 reviews for a game the store page shows with
thousands (845070) or a few. Its 0 is not to be trusted alone.
Screening every game the way purge would: 3 `unlisted` (appdetails calls them
not free: removed), 5 without `is_free` (appdetails calls two of them free:
kept), nothing hidden, unknown or coming soon. The dry run of the new purge on
the real catalogue matched: 8 flagged, 3 removed, 5 kept by appdetails.
Of the 142 games removed before, GetItems now sees 27 as free and listed
again. Most were never really gone: 14 were rejected by the ingest in two
bursts on 2026-09-27, when appdetails answered `success:false` for live games,
and were approved again later. Eight are still out; the new purge names them.
GitHub Actions has supported `concurrency.queue: max` since 2026-05-07: up to
100 waiting runs, first in, first out, with `cancel-in-progress: false`.
## The direction
1. **Queue, don't evict.** Every data-write workflow has `queue: max` and
checks out `ref: main`, so a run that waits loses nothing and works on what
the run before it committed.
2. **Batch, don't crawl.** Reviews and the purge screen come from GetItems:
about 200 requests for the catalogue, whatever its size within reason.
Requests per game are kept for confirming a removal and for the rare
review count of 0.
3. **Bound every run.** A job sets `JOB_BUDGET_MINUTES` below its timeout;
a loop past it starts no further game and the script saves what it did.
Loops visit games in an order that changes daily, rotations grow their
cycle (a multiple of the base) instead of their daily slice, and a
`::warning::` says when a loop used more than three quarters of its budget.
4. **Drop what does nothing.** Check Dead Links goes, and so do the rests
between batches of the keyed Web API.
Considered and left: a matrix of runners per job (still a request per game,
parts racing each other's pushes, and SonarCloud's S8233 on workflow-level
write permissions with more than one job), a self-hosted runner, the Worker's
cron (the Worker may never write a shard), and a second opinion from GetItems
for the ingest's rejections (a skipped link and a rejected one both end as a
`failed` queue row; only the public log line differs).
## What changed
| Job | Now | Time at 9,847 | At 20,000 |
| --- | --- | --- | --- |
| Update Reviews | daily; GetItems, `appreviews` only for a 0 (at most 300) | ~2-5 min | ~5 min |
| Purge Unhealthy | daily; GetItems screen, appdetails confirms each flag | ~3-8 min | ~5-10 min |
| Check Dead Links | retired | 0 | 0 |
| Top Online / Offline | no rests between batches | ~27 / ~64 min | ~55 min / ~2.1 h |
| Refresh Store Data | cycles 7 → 14 past 2,000 pages a day, 30 → 60 past 500 appdetails | ~65 min | ~65-90 min |
| Refresh All-Time Peaks | cycle 30 → 60 past 400 pages a day | ~14 min | ~15 min |
| Force Re-fetch | parts `1/8` … `8/8`, queued | ~2.2 h a part | ~2.2 h a part (`1/16`…) |
Rules worth keeping (CLAUDE.md has them in full):
- **Reviews.** Steam's floored percentage, in integers (`29/100*100` is 28.99…
in floating point). A GetItems count of 0 never writes; a stored percentage
or "N/A" is asked on `appreviews` instead. No answer keeps the value.
`last_updated` moves on a change, and on a heartbeat every 14-20 days for a
game that was read: the site's "stale" badge (30 days) used to be kept quiet
by this job stamping every game every run, which also rewrote every shard.
- **Purge.** Remove only when GetItems flags a game AND `check_game_health()`
agrees; ask an `unavailable` again five minutes later; never treat a failed
batch or a network error as "gone"; stop before writing anything past 500
flags or 100 removals; log as before; name earlier removals Steam has since
undone. A purged game keeps its D1 decision, so discovery never proposes it
again - the notice is the way back.
- **Budgets.** Never for the ingest: `clear_temp()` empties its whole queue.
Refetch clears a game on a copy and swaps it in only once appdetails has
answered, oldest first, so a part cut short resumes where it stopped.
`web/shared/pipeline.test.ts` now fails on a data-write workflow without
`queue: max`, `cancel-in-progress: false`, `ref: main` on every checkout, a
`timeout-minutes`, or a budget less than 20 minutes below it.
## Found after the merge: appreviews redirects
The first run of the new Update Reviews on `main` (2026-10-23, 2 m 31 s, 4,540
games changed - exactly the ones whose `last_updated` moved) asked `appreviews`
about the 12 games GetItems counted at 0, and all 12 failed. From this machine
and from GitHub's runners alike, `store.steampowered.com/appreviews/<id>?json=1`
now answers 302 to the store front, whatever the headers; it answered on
10-17. The 38 games the ingest added on 10-23 were the first to get "N/A".
So the ingest falls back to the GetItems summary, and a game never read
("N/A") with a count of 0 becomes "No reviews"; a stored percentage is still
kept until a second source confirms the 0.
## Load, before and after
| | Before | After |
| --- | --- | --- |
| Time the data-write group is busy | ~46 h a week | ~12 h a week |
| Requests to Steam | ~82,000 a week | ~25,000 a week |
| Runs cancelled while waiting | about one a day | none |
## Left for later
- **A timed-out run is still silent.** The reusable notifier in
`poli0981/.github` reports `failure` only; reporting `cancelled` for the
data workflows (now that waiting no longer cancels anything) belongs there.
- **Workers Builds deploys on data commits** (`docs/DEPLOYMENT.md`, "Which
commits deploy"). Daily reviews and purges add about one deploy a day until
the watch paths are set.
- **`data/snapshots/`** grows by 1.9 MB a day and nothing in the repository
reads it. A leaner format (no name, no zero rows) would quarter it.
- **The keyed player counts** are the last per-game loop. At 20,000 games
Top Offline needs about two hours; past that, sample games with no players
on a rotation instead of every run.
- **Re-queue what Steam has undone**: the purge's notice lists the removed
games that are free and on the store again.