feat(pipeline): keep the store, peaks and re-fetch runs bounded

- Refresh Store Data: the page and appdetails cycles grow past 2,000
  pages / 500 appdetails a day (7 -> 14, 30 -> 60) instead of the run,
  the slice comes in a daily order, and the store step has a budget. A
  game appdetails no longer calls free is listed in a ::warning::, the
  case the purge's two-signal rule cannot see; nothing is written.
- Refresh All-Time Peaks: 30 -> 60 days past 400 pages a day, never
  shorter, so SteamCharts is still asked at most once a month a game;
  the budget stops it with a warning like a refusal does.
- Force Re-fetch: --part K/N, oldest last_updated first, each game
  cleared on a copy that replaces it only once appdetails has answered;
  no final loop stamping every game. Parts queue one after another.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 committed 2026-10-23 22:22:08 +07:00
1 parent 74e1845fc4
commit 1da4d5940e
10 files changed
+406 -81

No files matched your search

+22 -6
View File
@@ -1,16 +1,25 @@
name : Force Re-fetch All (v2.1 Schema)
# ⚠ MANUAL ONLY – this re-fetches ALL games from Steam.
# Takes a long time for large datasets (300+ games ≈ 15-30 min).
# Creates a backup at data.jsonl.bak before starting.
# ⚠ MANUAL ONLY – this re-fetches games from Steam from scratch, about six
# seconds a game: far longer than one job may run once the catalogue is in
# the thousands. Run it in PARTS: part "1/8" takes an eighth of the games
# (crc32 slices, scripts/refetch_all.py), and the runs for 1/8 ... 8/8 can be
# started together - the data-write group queues them (queue: max). A part cut
# short by its budget keeps what it did; starting it again carries on with the
# games it did not reach. Creates a backup at data.bak before starting.
on :
workflow_dispatch :
inputs :
confirm :
description : 'Type "yes" to confirm force re-fetch of ALL games'
description : 'Type "yes" to confirm force re-fetch'
required : true
type : string
part :
description : 'Which part, K/N (1/8 ... 8/8 for the whole catalogue)'
required : true
default : '1/8'
type : string
permissions :
contents : write
@@ -18,16 +27,21 @@ permissions :
concurrency :
group : data-write
cancel-in-progress : false
queue : max
jobs :
refetch :
if : ${{ github.event.inputs.confirm == 'yes' }}
runs-on : ubuntu-latest
timeout-minutes : 120
timeout-minutes : 350
steps :
-
name : Checkout repo
uses : actions/checkout@v6
# main as it is when the job STARTS: a part that waited its turn must
# work on what the part before it committed.
with :
ref : main
-
name : Setup Python
@@ -36,9 +50,11 @@ jobs :
python-version : '3.15'
-
name : Force re-fetch all games
name : Force re-fetch one part
env :
STEAM_API_KEY : ${{ secrets.STEAM_API_KEY }}
REFETCH_PART : ${{ inputs.part }}
JOB_BUDGET_MINUTES : '320'
run : |
chmod +x ./bash/refetch_all.sh
./bash/refetch_all.sh
+17 -10
View File
@@ -1,16 +1,18 @@
name: Refresh All-Time Peaks
# all_time_peak - the most players a game has had at once - from SteamCharts'
# recorded peak, a thirtieth of the catalogue a day (~186 pages, 2-3 s apart:
# about ten minutes), so every game is asked once a month. Every run also
# raises the field from our own Steam API samples, which needs no requests.
# scripts/refresh_peaks.py has the rules: the field only ever goes up, and a
# refusal from SteamCharts ends the run early with a warning, not a failure.
# recorded peak, a thirtieth of the catalogue a day (~330 pages, 2-3 s apart:
# about a quarter of an hour), so every game is asked once a month. Past 400
# pages a day the cycle grows to 60 days and more instead: never more often
# than once a month a game. Every run also raises the field from our own
# Steam API samples, which needs no requests. scripts/refresh_peaks.py has
# the rules: the field only ever goes up, and a refusal from SteamCharts - or
# the time budget running out - ends the run early with a warning, not a
# failure.
#
# 19:30 UTC sits between Refresh Store Data (16:00, ~40 minutes) and the Daily
# Snapshot (23:00), where only the short Ingest runs are due. Scheduled runs
# here start up to a few hours late, and the data-write group keeps only ONE
# pending run, so this stays clear of the long morning jobs.
# 19:30 UTC sits between Refresh Store Data (16:00, about an hour) and the
# Daily Snapshot (23:00). A data job still running only delays this one: the
# data-write group queues runs (queue: max) instead of evicting them.
#
# The one-time backfill is a manual run with the box ticked: every game, about
# four hours. Start it in the evening UTC, after Refresh Store Data.
@@ -30,15 +32,20 @@ permissions: {contents: write}
concurrency:
group: data-write
cancel-in-progress: false
queue: max
jobs:
refresh:
runs-on: ubuntu-latest
timeout-minutes: ${{ inputs.backfill && 330 || 30 }}
timeout-minutes: ${{ inputs.backfill && 340 || 45 }}
steps:
# main as it is when the job STARTS: a run that waited its turn must
# work on what the job before it committed.
- uses: actions/checkout@v6
with: {ref: main}
- uses: actions/setup-python@v6
with: {python-version: '3.15'}
- run: chmod +x ./bash/refresh_peaks.sh && ./bash/refresh_peaks.sh
env:
PEAKS_ALL: ${{ inputs.backfill }}
JOB_BUDGET_MINUTES: ${{ inputs.backfill && '300' || '25' }}
+13 -8
View File
@@ -5,22 +5,23 @@ name: Refresh Store Data
# store page, every game once a week; name, header image, description,
# developer/publisher, release date, platforms, Metacritic and DRM notes from
# appdetails, every game once a month. Each daily run takes one slice of each
# rotation - roughly 630 store pages and 150 appdetails calls, ~40 minutes.
# rotation - about 1,400 store pages and 330 appdetails calls at 9,800 games,
# an hour. Past 2,000 pages or 500 appdetails a day a cycle doubles instead of
# the run growing, so it stays under an hour and a half.
# scripts/refresh_store_data.py explains the rotation and the rules (never
# blanks a value, never touches MANUAL_FIELDS).
#
# Then the anti-cheat sweep, committed separately: every game's anti-cheat as
# its developer declares it on Steam, about 160 GetItems requests plus a store
# its developer declares it on Steam, about 200 GetItems requests plus a store
# page for each disclosure Steam does not name there (~70), a few minutes.
# scripts/refresh_anticheat.py has its rules: Steam wins, an override wins
# over Steam, and no disclosure never clears a value. `only: anticheat` runs
# just the sweep.
#
# 16:00 UTC is the quiet part of this repository's day. Scheduled runs here
# start two to five hours late, the morning data jobs (reviews, dead links,
# purge) run for two to four hours, and the data-write group keeps only ONE
# pending run: a second one queueing behind a running job cancels the first.
# Nothing else long is due this late; the Daily Snapshot (23:00) comes after.
# 16:00 UTC is the quiet part of this repository's day: scheduled runs here
# start up to an hour late, and the morning jobs are done by then. A data job
# that is still running only delays this one - the data-write group queues
# runs (queue: max) instead of evicting the one waiting.
on:
schedule: [{cron: '0 16 * * *'}]
@@ -38,11 +39,12 @@ permissions: {contents: write}
concurrency:
group: data-write
cancel-in-progress: false
queue: max
jobs:
refresh:
runs-on: ubuntu-latest
timeout-minutes: 120
timeout-minutes: 150
steps:
# main, not the triggering commit: a run that waited its turn in
# data-write must work on the data as it is now, not as it was queued.
@@ -53,3 +55,6 @@ jobs:
- run: chmod +x ./bash/refresh_store.sh && ./bash/refresh_store.sh
env:
REFRESH_ONLY: ${{ inputs.only }}
# The store refresh's budget (core/budget.py); the anti-cheat sweep
# after it takes a few minutes and has none.
JOB_BUDGET_MINUTES: '100'
+6 -3
View File
@@ -1,7 +1,10 @@
#!/bin/bash
set -euo pipefail
echo "── Force re-fetch ALL games (v2.1 schema) ──"
# REFETCH_PART is the manual run's "part" box, K/N (scripts/refetch_all.py
# checks it): the whole catalogue no longer fits in one job.
part="${REFETCH_PART:-1/1}"
echo "── Force re-fetch, part ${part} ──"
pip install --quiet --require-hashes --only-binary :all: -r requirements.txt
python scripts/refetch_all.py
bash "$(dirname "$0")/commit_push.sh" "Force re-fetch all games → v2.1 schema [$(date +'%Y-%m-%d')]"
python scripts/refetch_all.py --part "$part"
bash "$(dirname "$0")/commit_push.sh" "Force re-fetch, part ${part} [$(date +'%Y-%m-%d')]"
+67 -20
View File
@@ -1,15 +1,33 @@
#!/usr/bin/env python3
"""
Force re-fetch ALL games. MANUAL ONLY.
Backup → migrate → clear fetchable → re-fetch → save.
Backup → migrate → re-fetch each game from scratch → save.
At about six seconds a game the whole catalogue takes far longer than a job
may run, so it is done in parts: --part K/N takes the games whose
crc32(appid) % N is K-1 (core/rotation.py), and the workflow's runs queue
one behind another. Each game is cleared and fetched on a copy that replaces
it only once appdetails has answered, so a failed request - or the job
budget running out (core/budget.py) - leaves every other game exactly as it
was. The oldest last_updated goes first: running a part that was cut short
again carries on where it stopped.
"""
import sys, os, shutil
import argparse
import copy
import os
import re
import shutil
import sys
from collections import Counter
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from core.budget import Deadline
from core.constants import DATA_DIR, STEAM_API_KEY
from core.data_store import load_main, save_main, migrate_record, now_iso
from core.data_store import extract_appid, load_main, save_main, migrate_record
from core.fetcher import fetch_full, process_batch
from core.overrides import OverrideGuard
from core.rotation import slice_of
from core.steam_client import get_client
CLEARABLE = {
@@ -49,13 +67,51 @@ def clear_fetchable(game):
game["metacritic"] = "N/A"
def main():
def parse_part(text: str) -> tuple[int, int]:
"""'K/N' -> (K, N) with 1 <= K <= N."""
m = re.fullmatch(r"\s*(\d+)\s*/\s*(\d+)\s*", text or "")
if not m or not 1 <= int(m.group(1)) <= int(m.group(2)):
raise argparse.ArgumentTypeError(f"expected K/N with 1 <= K <= N, got {text!r}")
return int(m.group(1)), int(m.group(2))
def in_part(game: dict, part: tuple[int, int]) -> bool:
appid = extract_appid(game.get("link", ""))
return bool(appid) and slice_of(appid, part[1]) == part[0] - 1
def refetch_one(game: dict, client, guard) -> bool:
"""Re-fetch one game from scratch, in place, or leave it exactly as it was
when appdetails does not answer. Returns whether it was re-fetched."""
appid = extract_appid(game.get("link", ""))
if not appid:
return False
status, details = client.fetch_app_details_full(appid)
if status != "ok" or not details:
return False
fresh = copy.deepcopy(game)
clear_fetchable(fresh)
fetch_full(fresh, client=client, prefetched_details=details, guard=guard)
game.clear()
game.update(fresh)
return True
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0].strip())
ap.add_argument("--part", type=parse_part, default=(1, 1),
help="K/N: re-fetch only the K-th of N parts of the catalogue (default 1/1)")
args = ap.parse_args(argv)
deadline = Deadline.from_env()
games = load_main()
if not games:
print("Empty.")
return
total = len(games)
print(f"Force re-fetch: {total} games\n")
targets = [g for g in games if in_part(g, args.part)]
# Oldest first: a part cut short carries on where it stopped next time.
targets.sort(key=lambda g: g.get("last_updated") or "")
print(f"Force re-fetch: part {args.part[0]}/{args.part[1]}, {len(targets)} of {len(games)} games\n")
backup_dir = DATA_DIR + ".bak"
if os.path.isdir(backup_dir):
@@ -63,30 +119,21 @@ def main():
shutil.copytree(DATA_DIR, backup_dir)
print(f"Backup -> {backup_dir}")
# Migrate + clear
for g in games:
migrate_record(g)
clear_fetchable(g)
print(f"Cleared fetchable fields. Re-fetching...\n")
if not STEAM_API_KEY:
print("⚠ No STEAM_API_KEY – players skipped")
client = get_client()
guard = OverrideGuard.load()
process_batch(games, lambda g: fetch_full(g, client=client, guard=guard), "Re-fetching")
stamp = now_iso()
for g in games:
g["last_updated"] = stamp
tally: Counter = Counter()
process_batch(targets, lambda g: tally.update([refetch_one(g, client, guard)]),
"Re-fetching", deadline=deadline)
save_main(games)
# Stats
has = lambda k: sum(1 for g in games if g.get(k) and g[k] not in ("", "N/A", [], False))
print(f"\n{'═'*50}")
print(f"Done {total} games. Backup: {backup_dir}")
print(f" name: {has('name')} | dev: {has('developer')} | pub: {has('publisher')}")
print(f" plat: {has('platforms')} | lang: {has('languages')} | tags: {has('tags')}")
print(f"Re-fetched {tally[True]} of {len(targets)} ({tally[False]} left as they were: "
f"appdetails did not answer). Backup: {backup_dir}")
if __name__ == "__main__":
+25 -9
View File
@@ -1,7 +1,8 @@
#!/usr/bin/env python3
"""
Refresh all_time_peak - the most players each game has had at once - from
SteamCharts, a thirtieth of the catalogue a day.
SteamCharts, a thirtieth of the catalogue a day (less, once that passes
PEAK_DAILY games).
core/peaks.py explains the field; core/steamcharts.py the source and why it is
read so carefully. Each run does two things:
@@ -12,7 +13,9 @@ read so carefully. Each run does two things:
field for the whole catalogue on the first run.
2. For the games whose slice is due (crc32(appid) % 30, see core/rotation.py:
every game once a month), read SteamCharts' recorded all-time peak and
raise the field to it.
raise the field to it. Past PEAK_DAILY games a day the cycle grows to
60, 90, ... days (rotation.cycle_for), never shorter: SteamCharts is still
asked about a game at most once a month.
The rules:
- all_time_peak only ever goes UP. A failed request, an untracked app or a
@@ -23,8 +26,9 @@ The rules:
never seen; both are normal and simply skipped until next month.
- A 403, a 429 or a Cloudflare challenge stops the SteamCharts part of the
run (everything already raised is kept and saved), and so do ten errors in
a row. The run still succeeds, with a warning: a refusal is SteamCharts'
call to make, not a pipeline failure.
a row, and so does the job's time budget (core/budget.py). The run still
succeeds, with a warning: a refusal is SteamCharts' call to make, not a
pipeline failure.
- Everything is written by save_main().
Usage (from the repository root):
@@ -41,12 +45,16 @@ from typing import Optional
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from core.budget import Deadline, expired
from core.data_store import extract_appid, load_main, migrate_record, save_main
from core.peaks import observed_peak, raise_peak
from core.rotation import day_number, due
from core.rotation import cycle_for, daily_order, day_number, due
from core.steamcharts import SteamChartsClient
PEAK_CYCLE_DAYS = 30
# The most SteamCharts pages a day before the cycle grows (rotation.cycle_for):
# about 17 minutes at 2-3 s a page.
PEAK_DAILY = 400
MAX_CONSECUTIVE_ERRORS = 10
@@ -55,13 +63,19 @@ def raise_from_samples(games: list[dict]) -> int:
return sum(1 for g in games if raise_peak(g, observed_peak(g)))
def raise_from_steamcharts(targets: list[dict], client, log=print) -> Counter:
def raise_from_steamcharts(targets: list[dict], client, log=print,
deadline: Optional[Deadline] = None) -> Counter:
"""Step 2. Returns a tally of the outcomes: "raised", "unchanged",
"untracked", "error", and "skipped" for games never asked because the run
stopped early."""
tally: Counter = Counter()
errors_in_a_row = 0
for i, game in enumerate(targets):
if expired(deadline):
tally["skipped"] += len(targets) - i
log(f"::warning::The job's time budget ran out with {len(targets) - i} SteamCharts "
f"pages left; those games are asked on their next turn.")
break
appid = extract_appid(game.get("link", ""))
if not appid:
continue
@@ -101,6 +115,7 @@ def main(argv: Optional[list[str]] = None) -> None:
ap.add_argument("--dry-run", action="store_true", help="fetch and report, write nothing")
args = ap.parse_args(argv)
deadline = Deadline.from_env()
games = load_main()
for g in games:
migrate_record(g)
@@ -109,13 +124,14 @@ def main(argv: Optional[list[str]] = None) -> None:
print(f"{len(games)} games; all_time_peak raised from our own samples for {from_samples}")
day = day_number() if args.day is None else args.day
targets = games if args.all else [g for g in games if due(g, PEAK_CYCLE_DAYS, day)]
cycle = cycle_for(len(games), PEAK_CYCLE_DAYS, PEAK_DAILY)
targets = games if args.all else [g for g in daily_order(games, day) if due(g, cycle, day)]
if args.limit is not None:
targets = targets[: args.limit]
scope = "every game" if args.all else f"slice {day % PEAK_CYCLE_DAYS}/{PEAK_CYCLE_DAYS}"
scope = "every game" if args.all else f"slice {day % cycle}/{cycle}"
print(f"SteamCharts: {len(targets)} games ({scope})")
tally = raise_from_steamcharts(targets, SteamChartsClient()) if targets else Counter()
tally = raise_from_steamcharts(targets, SteamChartsClient(), deadline=deadline) if targets else Counter()
summary = ", ".join(f"{k} {v}" for k, v in tally.most_common()) or "nothing asked"
print(f"\nSteamCharts: {summary}")
+52 -24
View File
@@ -18,8 +18,13 @@ Two groups, each on its own rotation, a slice of the catalogue per daily run:
developer, publisher, release_date, platforms,
metacritic, drm_notes every 30 days
That is ~630 store pages and ~150 appdetails calls a day - about 40 minutes
at the client's throttle - instead of 8,800 requests in one run.
That was ~1,400 store pages and ~330 appdetails calls a day at 9,800 games,
about an hour at the client's throttle. The cycles grow with the catalogue
(core/rotation.cycle_for): past PAGE_DAILY pages or DETAILS_DAILY appdetails
a day, a cycle doubles (triples, ...) instead of the run growing, and a game
keeps its weekday and its day of the month. A run that still meets its time
budget (core/budget.py) saves what it did; the slice is visited in an order
that changes daily, so the cut falls elsewhere next time.
A game's slice is crc32(appid) % cycle, NOT appid % cycle: Steam appids are
overwhelmingly multiples of 10, so `% 30` would leave 27 of the 30 monthly
@@ -37,7 +42,10 @@ The rules, because these fields were fill-if-empty until now:
gate, a delisted app or a changed page layout leaves the stored value alone.
- MANUAL_FIELDS are never written (genre, type_game, anti_cheat, safe, ...),
and neither is `notes`: health checks and "no longer free" belong to
check_dead_links.py, purge_unhealthy.py and mark_dead_games.py.
purge_unhealthy.py and mark_dead_games.py. A game appdetails no longer
calls free is only listed, in a ::warning:: - purge removes a game only
when Steam's store API flags it too, and this is where the opposite case
shows.
- has_paid_dlc is read only from a real store page - one with a language
table or tags - or, for a gated game, from the prices of its DLC. An
age-gate page has no DLC section and would read as False; when neither
@@ -60,26 +68,31 @@ Usage (from the repository root):
import argparse
import os
import sys
import zlib
from collections import Counter
from typing import Optional
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from core.budget import Deadline
from core.data_store import (
ELLIPSIS, extract_appid, is_empty, load_main, migrate_record, now_iso, save_main,
truncate_description,
)
from core.fetcher import process_batch, scrape_store_fields
from core.health_checker import free_per_appdetails
from core.normalize import (
clean_list, clean_text, decode_entities, normalize_records, normalize_release_date,
)
# The rotation (crc32 slices) is shared with refresh_peaks.py.
from core.rotation import day_number, due
from core.rotation import cycle_for, daily_order, day_number, due
from core.steam_client import get_client
PAGE_CYCLE_DAYS = 7
DETAILS_CYCLE_DAYS = 30
# The most a day's slice may hold before its cycle grows (rotation.cycle_for):
# with the repairs, about an hour and a half at the client's throttle.
PAGE_DAILY = 2000
DETAILS_DAILY = 500
# Games with empty store fields taken a day on top of the rotation. 25 on
# 2026-10-10; the cap is for the day something breaks for every game at once,
# so a two-hour job cannot turn into a whole-catalogue crawl.
@@ -205,8 +218,10 @@ def apply_app_details(game: dict, data: dict) -> list[str]:
return changed
def refresh_game(game: dict, client, page: bool, details: bool) -> list[str]:
"""Refresh one game in place. Returns the fields that changed."""
def refresh_game(game: dict, client, page: bool, details: bool,
not_free: Optional[list] = None) -> list[str]:
"""Refresh one game in place. Returns the fields that changed. A game
whose appdetails no longer call it free is added to `not_free`."""
appid = extract_appid(game.get("link", ""))
if not appid:
return []
@@ -216,6 +231,8 @@ def refresh_game(game: dict, client, page: bool, details: bool) -> list[str]:
data = client.fetch_app_details(appid)
if data:
changed += apply_app_details(game, data)
if not_free is not None and not free_per_appdetails(data):
not_free.append(f"{appid} {game.get('name', '')[:40]}")
if page:
# The appdetails just fetched spare the DLC check a second request.
scraped = scrape_store_fields(client, appid, details=data)
@@ -233,29 +250,31 @@ def missing_store_fields(game: dict) -> bool:
def build_plan(games: list[dict], day: int, want_page: bool = True,
want_details: bool = True, missing_only: bool = False) -> list[tuple]:
want_details: bool = True, missing_only: bool = False,
page_cycle: int = PAGE_CYCLE_DAYS,
details_cycle: int = DETAILS_CYCLE_DAYS) -> list[tuple]:
"""[(game, page, details)] for one day: the games with empty store fields
first (page group, every day), then the day's rotation slices.
The repairs are capped at MISSING_DAILY_CAP and ordered by a hash that
changes daily, so a residue Steam never fills cannot hold the same places
forever. missing_only keeps just the repairs, store page only.
Both parts come in an order that changes daily (rotation.daily_order):
the repairs are capped at MISSING_DAILY_CAP, so a residue Steam never
fills cannot hold the same places forever, and a run that meets its time
budget leaves a different part of the slice for the next one.
missing_only keeps just the repairs, store page only.
"""
repairs: list[dict] = []
if want_page:
repairs = [g for g in games if missing_store_fields(g)]
repairs.sort(key=lambda g: zlib.crc32(f"{day}:{extract_appid(g.get('link', ''))}".encode()))
repairs = repairs[:MISSING_DAILY_CAP]
repairs = daily_order([g for g in games if missing_store_fields(g)], day)[:MISSING_DAILY_CAP]
if missing_only:
return [(g, True, False) for g in repairs]
plan = [(g, True, want_details and due(g, DETAILS_CYCLE_DAYS, day)) for g in repairs]
plan = [(g, True, want_details and due(g, details_cycle, day)) for g in repairs]
taken = {id(g) for g in repairs}
for g in games:
for g in daily_order(games, day):
if id(g) in taken:
continue
page = want_page and due(g, PAGE_CYCLE_DAYS, day)
details = want_details and due(g, DETAILS_CYCLE_DAYS, day)
page = want_page and due(g, page_cycle, day)
details = want_details and due(g, details_cycle, day)
if page or details:
plan.append((g, page, details))
return plan
@@ -273,6 +292,7 @@ def main(argv: Optional[list[str]] = None) -> None:
ap.add_argument("--dry-run", action="store_true", help="fetch and report, write nothing")
args = ap.parse_args(argv)
deadline = Deadline.from_env()
day = day_number() if args.day is None else args.day
games = load_main()
for g in games:
@@ -284,7 +304,10 @@ def main(argv: Optional[list[str]] = None) -> None:
want_page = args.only in (None, "page")
want_details = args.only in (None, "details")
plan = build_plan(games, day, want_page, want_details, missing_only=args.missing_only)
page_cycle = cycle_for(len(games), PAGE_CYCLE_DAYS, PAGE_DAILY)
details_cycle = cycle_for(len(games), DETAILS_CYCLE_DAYS, DETAILS_DAILY)
plan = build_plan(games, day, want_page, want_details, missing_only=args.missing_only,
page_cycle=page_cycle, details_cycle=details_cycle)
if args.limit is not None:
plan = plan[: args.limit]
@@ -292,28 +315,33 @@ def main(argv: Optional[list[str]] = None) -> None:
n_details = sum(1 for _, _, d in plan if d)
n_missing = sum(1 for g, p, _ in plan if p and missing_store_fields(g))
print(f"{len(games)} games; day {day}: store page for {n_page} "
f"(slice {day % PAGE_CYCLE_DAYS}/{PAGE_CYCLE_DAYS}, {n_missing} with empty store fields), "
f"appdetails for {n_details} (slice {day % DETAILS_CYCLE_DAYS}/{DETAILS_CYCLE_DAYS})")
f"(slice {day % page_cycle}/{page_cycle}, {n_missing} with empty store fields), "
f"appdetails for {n_details} (slice {day % details_cycle}/{details_cycle})")
if not plan:
return
client = get_client()
flags = {id(g): (page, details) for g, page, details in plan}
fields: Counter = Counter()
not_free: list[str] = []
touched = 0
def refresh(game: dict) -> None:
nonlocal touched
page, details = flags[id(game)]
changed = refresh_game(game, client, page=page, details=details)
changed = refresh_game(game, client, page=page, details=details, not_free=not_free)
if changed:
touched += 1
fields.update(changed)
process_batch([g for g, _, _ in plan], refresh, "Refreshing")
done = process_batch([g for g, _, _ in plan], refresh, "Refreshing", deadline=deadline)
summary = ", ".join(f"{f} {n}" for f, n in fields.most_common()) or "nothing"
print(f"\n{touched} of {len(plan)} games changed: {summary}")
print(f"\n{touched} of {done} games changed: {summary}")
if not_free:
print(f"::warning::appdetails no longer calls {len(not_free)} game(s) free while Steam's "
f"store API still does, so purge keeps them: {'; '.join(not_free[:20])}"
f"{' …' if len(not_free) > 20 else ''}. Remove one in /admin if it is so.")
if args.dry_run:
print("Dry run: nothing written.")
return
+98 -1
View File
@@ -1,8 +1,15 @@
"""
refetch_all.py clears what Steam can give back and keeps what a person
decided. Run from the repository root: python -m pytest scripts/tests
decided, one game at a time: a game appdetails does not answer for, or the
ones a part cut short by its budget never reached, stay exactly as they were.
Run from the repository root: python -m pytest scripts/tests
"""
import os
import pytest
import refetch_all
from core import budget, data_store
def test_clear_fetchable_resets_a_default_to_the_skeleton_s_value():
@@ -19,3 +26,93 @@ def test_clear_fetchable_keeps_a_decision():
refetch_all.clear_fetchable(game)
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"], game["genre"]) == \
("BattlEye", "BattlEye [Kernel]", True, "FPS")
# ──────────── In parts, game by game ────────────
def _rec(appid: str, **fields) -> dict:
base = {"link": f"https://store.steampowered.com/app/{appid}/", "name": f"Game {appid}",
"description": "kept", "tags": ["FPS"], "reviews": "50% (Mixed)", "genre": "FPS",
"anti_cheat": "-", "anti_cheat_note": "", "is_kernel_ac": None, "type_game": "online",
"notes": "Not reviewed yet", "last_updated": "2026-01-01T00:00:00Z"}
base.update(fields)
return base
class _Client:
def __init__(self, failing=()):
self.failing = set(failing)
self.asked: list[str] = []
def fetch_app_details_full(self, appid):
self.asked.append(appid)
if appid in self.failing:
return ("network_error", None)
return ("ok", {"name": f"Fresh {appid}", "is_free": True})
@pytest.mark.parametrize("text, part", [("1/1", (1, 1)), (" 3 / 8 ", (3, 8)), ("8/8", (8, 8))])
def test_a_part_is_read(text, part):
assert refetch_all.parse_part(text) == part
@pytest.mark.parametrize("text", ["0/4", "5/4", "1/0", "x", "", "1/2/3"])
def test_a_bad_part_is_refused(text):
with pytest.raises(Exception):
refetch_all.parse_part(text)
def test_the_parts_split_the_catalogue():
games = [_rec(str(10 * i)) for i in range(1, 501)]
counts = [sum(refetch_all.in_part(g, (k, 8)) for g in games) for k in range(1, 9)]
assert sum(counts) == len(games) and min(counts) > 0
def test_a_game_appdetails_does_not_answer_for_is_left_as_it_was(monkeypatch):
calls: list[str] = []
monkeypatch.setattr(refetch_all, "fetch_full",
lambda g, client=None, prefetched_details=None, guard=None: calls.append(g["link"]))
game = _rec("730")
before = dict(game)
assert refetch_all.refetch_one(game, _Client(failing={"730"}), None) is False
assert game == before and calls == []
def test_a_game_is_cleared_and_fetched_on_a_copy(monkeypatch):
def fake_fetch(g, client=None, prefetched_details=None, guard=None):
assert g["description"] == "" and g["reviews"] == "N/A" # cleared before the fetch
g["name"] = prefetched_details["name"]
monkeypatch.setattr(refetch_all, "fetch_full", fake_fetch)
game = _rec("730", anti_cheat="BattlEye")
assert refetch_all.refetch_one(game, _Client(), None) is True
assert game["name"] == "Fresh 730" and game["anti_cheat"] == "BattlEye"
def test_a_cut_part_leaves_every_other_game_alone_and_resumes_oldest_first(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
os.makedirs("data")
games = [_rec(str(10 * i), last_updated=f"2026-01-{i:02d}T00:00:00Z") for i in range(1, 9)]
data_store.save_main(games, apply_human_overrides=False)
client = _Client()
monkeypatch.setattr(refetch_all, "get_client", lambda: client)
monkeypatch.setattr(refetch_all.OverrideGuard, "load", classmethod(lambda cls: None))
monkeypatch.setattr(refetch_all, "STEAM_API_KEY", "")
clock = [0.0]
monkeypatch.setattr(budget.time, "monotonic", lambda: clock[0])
def fake_fetch(g, client=None, prefetched_details=None, guard=None):
clock[0] += 40
g["name"] = prefetched_details["name"]
g["last_updated"] = "2026-10-23T00:00:00Z"
monkeypatch.setattr(refetch_all, "fetch_full", fake_fetch)
monkeypatch.setattr("core.fetcher.time.sleep", lambda s: None)
monkeypatch.setenv(budget.ENV, "1")
refetch_all.main(["--part", "1/1"])
after = {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
assert client.asked == ["10", "20"] # the two oldest, then the budget ran out
assert after["10"]["name"] == "Fresh 10" and after["30"]["name"] == "Game 30"
assert after["30"]["description"] == "kept" # never cleared
+47
View File
@@ -239,3 +239,50 @@ def test_refetch_all_never_clears_it():
# The other player fields are cleared and fetched again; this one cannot be.
assert "all_time_peak" not in refetch_all.CLEARABLE
assert "all_time_peak" not in refetch_all.CONDITIONAL
def test_the_budget_stops_the_steamcharts_part_and_keeps_what_was_raised(monkeypatch):
from core import budget
clock = [0.0]
monkeypatch.setattr(budget.time, "monotonic", lambda: clock[0])
deadline = budget.Deadline(1)
games = [_game(str(i)) for i in range(1, 6)]
class Slow(_Client):
def fetch(self, appid):
clock[0] += 25
return super().fetch(appid)
client = Slow({str(i): Result("ok", 100 * i) for i in range(1, 6)})
lines: list[str] = []
tally = rp.raise_from_steamcharts(games, client, log=lines.append, deadline=deadline)
assert client.asked == ["1", "2", "3"]
assert tally["raised"] == 3 and tally["skipped"] == 2
assert any(line.startswith("::warning::") for line in lines)
def test_a_larger_catalogue_is_asked_less_often_never_more():
assert rp.cycle_for(9847, rp.PEAK_CYCLE_DAYS, rp.PEAK_DAILY) == 30
assert rp.cycle_for(20000, rp.PEAK_CYCLE_DAYS, rp.PEAK_DAILY) == 60
def test_a_run_asks_today_s_slice_and_saves(tmp_path, monkeypatch, capsys):
import os
from core import budget, data_store, rotation
monkeypatch.chdir(tmp_path)
os.makedirs("data")
monkeypatch.delenv(budget.ENV, raising=False)
games = [_game(str(10 * i), current_players="0", peak_today="0") for i in range(1, 41)]
data_store.save_main(games, apply_human_overrides=False)
day = rotation.slice_of("10", rp.PEAK_CYCLE_DAYS) + 30 * 700
client = _Client({str(10 * i): Result("ok", 1000 + i) for i in range(1, 41)})
monkeypatch.setattr(rp, "SteamChartsClient", lambda: client)
rp.main(["--day", str(day)])
assert "10" in client.asked
assert all(rotation.slice_of(a, 30) == day % 30 for a in client.asked)
after = {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
assert after["10"]["all_time_peak"] == "1,001"
assert f"slice {day % 30}/30" in capsys.readouterr().out
+59
View File
@@ -391,3 +391,62 @@ def test_a_blurb_that_now_fits_further_is_written_but_is_not_news():
def test_a_rewritten_blurb_is_a_change():
game = _game(description="Old words, cut short…")
assert "description" in rsd.apply_app_details(game, {"short_description": "New words entirely."})
# ──────────── A growing catalogue ────────────
def test_the_page_cycle_grows_with_the_catalogue_and_keeps_its_weekday():
assert rsd.cycle_for(9847, rsd.PAGE_CYCLE_DAYS, rsd.PAGE_DAILY) == 7
games = [_game(str(10 * i)) for i in range(1, 12001)]
assert rsd.cycle_for(len(games), rsd.PAGE_CYCLE_DAYS, 1000) == 14
for day in range(20000, 20014):
plan = rsd.build_plan(games, day, want_details=False, page_cycle=14)
assert len(plan) < 1000
assert all(slice_of(rsd.extract_appid(g["link"]), 7) == day % 7 for g, _, _ in plan)
def test_the_slice_comes_in_an_order_that_changes_daily():
games = [_game(str(10 * i)) for i in range(1, 701)]
first = [g["link"] for g, _, _ in rsd.build_plan(games, 20006, want_details=False)]
again = [g["link"] for g, _, _ in rsd.build_plan(list(reversed(games)), 20006, want_details=False)]
week_later = [g["link"] for g, _, _ in rsd.build_plan(games, 20013, want_details=False)]
assert first == again
assert sorted(first) == sorted(week_later) and first != week_later
def test_a_game_appdetails_no_longer_call_free_is_listed_not_written():
game = _game()
before = dict(game)
listed: list[str] = []
details = {"name": "Counter-Strike 2", "is_free": False,
"price_overview": {"initial": 1499, "final": 1499}}
rsd.refresh_game(game, FakeClient(details=details), page=False, details=True, not_free=listed)
assert listed == ["730 Counter-Strike 2"]
assert game["notes"] == before["notes"]
listed.clear()
rsd.refresh_game(_game(), FakeClient(details={"name": "x", "is_free": True}), page=False, details=True,
not_free=listed)
assert listed == []
def test_a_run_refreshes_today_s_slice_and_lists_what_appdetails_calls_paid(tmp_path, monkeypatch, capsys):
import os
from core import budget, data_store
monkeypatch.chdir(tmp_path)
os.makedirs("data")
monkeypatch.delenv(budget.ENV, raising=False)
monkeypatch.setattr("core.fetcher.time.sleep", lambda s: None)
games = [_game(str(10 * i), name=f"Game {i}") for i in range(1, 61)]
data_store.save_main(games, apply_human_overrides=False)
client = FakeClient(details={"name": "Renamed", "is_free": False, "price_overview": {"initial": 999}})
monkeypatch.setattr(rsd, "get_client", lambda: client)
rsd.main(["--only", "details", "--day", "20030"])
out = capsys.readouterr().out
asked = [c.split(":")[1] for c in client.calls]
assert asked and all(rsd.due(_game(a), 30, 20030) for a in asked)
assert "slice 20/30" in out
assert "::warning::appdetails no longer calls" in out
after = {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
assert all(after[a]["name"] == "Renamed" for a in asked)