feat(pipeline): keep the store, peaks and re-fetch runs bounded
- Refresh Store Data: the page and appdetails cycles grow past 2,000 pages / 500 appdetails a day (7 -> 14, 30 -> 60) instead of the run, the slice comes in a daily order, and the store step has a budget. A game appdetails no longer calls free is listed in a ::warning::, the case the purge's two-signal rule cannot see; nothing is written. - Refresh All-Time Peaks: 30 -> 60 days past 400 pages a day, never shorter, so SteamCharts is still asked at most once a month a game; the budget stops it with a warning like a refusal does. - Force Re-fetch: --part K/N, oldest last_updated first, each game cleared on a copy that replaces it only once appdetails has answered; no final loop stamping every game. Parts queue one after another. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
74e1845fc4
commit
1da4d5940e
10 files changed
+406
-81
No files matched your search
@@ -1,16 +1,25 @@
|
||||
name : Force Re-fetch All (v2.1 Schema)
|
||||
|
||||
# ⚠ MANUAL ONLY – this re-fetches ALL games from Steam.
|
||||
# Takes a long time for large datasets (300+ games ≈ 15-30 min).
|
||||
# Creates a backup at data.jsonl.bak before starting.
|
||||
# ⚠ MANUAL ONLY – this re-fetches games from Steam from scratch, about six
|
||||
# seconds a game: far longer than one job may run once the catalogue is in
|
||||
# the thousands. Run it in PARTS: part "1/8" takes an eighth of the games
|
||||
# (crc32 slices, scripts/refetch_all.py), and the runs for 1/8 ... 8/8 can be
|
||||
# started together - the data-write group queues them (queue: max). A part cut
|
||||
# short by its budget keeps what it did; starting it again carries on with the
|
||||
# games it did not reach. Creates a backup at data.bak before starting.
|
||||
|
||||
on :
|
||||
workflow_dispatch :
|
||||
inputs :
|
||||
confirm :
|
||||
description : 'Type "yes" to confirm force re-fetch of ALL games'
|
||||
description : 'Type "yes" to confirm force re-fetch'
|
||||
required : true
|
||||
type : string
|
||||
part :
|
||||
description : 'Which part, K/N (1/8 ... 8/8 for the whole catalogue)'
|
||||
required : true
|
||||
default : '1/8'
|
||||
type : string
|
||||
|
||||
permissions :
|
||||
contents : write
|
||||
@@ -18,16 +27,21 @@ permissions :
|
||||
concurrency :
|
||||
group : data-write
|
||||
cancel-in-progress : false
|
||||
queue : max
|
||||
|
||||
jobs :
|
||||
refetch :
|
||||
if : ${{ github.event.inputs.confirm == 'yes' }}
|
||||
runs-on : ubuntu-latest
|
||||
timeout-minutes : 120
|
||||
timeout-minutes : 350
|
||||
steps :
|
||||
-
|
||||
name : Checkout repo
|
||||
uses : actions/checkout@v6
|
||||
# main as it is when the job STARTS: a part that waited its turn must
|
||||
# work on what the part before it committed.
|
||||
with :
|
||||
ref : main
|
||||
|
||||
-
|
||||
name : Setup Python
|
||||
@@ -36,9 +50,11 @@ jobs :
|
||||
python-version : '3.15'
|
||||
|
||||
-
|
||||
name : Force re-fetch all games
|
||||
name : Force re-fetch one part
|
||||
env :
|
||||
STEAM_API_KEY : ${{ secrets.STEAM_API_KEY }}
|
||||
REFETCH_PART : ${{ inputs.part }}
|
||||
JOB_BUDGET_MINUTES : '320'
|
||||
run : |
|
||||
chmod +x ./bash/refetch_all.sh
|
||||
./bash/refetch_all.sh
|
||||
@@ -1,16 +1,18 @@
|
||||
name: Refresh All-Time Peaks
|
||||
|
||||
# all_time_peak - the most players a game has had at once - from SteamCharts'
|
||||
# recorded peak, a thirtieth of the catalogue a day (~186 pages, 2-3 s apart:
|
||||
# about ten minutes), so every game is asked once a month. Every run also
|
||||
# raises the field from our own Steam API samples, which needs no requests.
|
||||
# scripts/refresh_peaks.py has the rules: the field only ever goes up, and a
|
||||
# refusal from SteamCharts ends the run early with a warning, not a failure.
|
||||
# recorded peak, a thirtieth of the catalogue a day (~330 pages, 2-3 s apart:
|
||||
# about a quarter of an hour), so every game is asked once a month. Past 400
|
||||
# pages a day the cycle grows to 60 days and more instead: never more often
|
||||
# than once a month a game. Every run also raises the field from our own
|
||||
# Steam API samples, which needs no requests. scripts/refresh_peaks.py has
|
||||
# the rules: the field only ever goes up, and a refusal from SteamCharts - or
|
||||
# the time budget running out - ends the run early with a warning, not a
|
||||
# failure.
|
||||
#
|
||||
# 19:30 UTC sits between Refresh Store Data (16:00, ~40 minutes) and the Daily
|
||||
# Snapshot (23:00), where only the short Ingest runs are due. Scheduled runs
|
||||
# here start up to a few hours late, and the data-write group keeps only ONE
|
||||
# pending run, so this stays clear of the long morning jobs.
|
||||
# 19:30 UTC sits between Refresh Store Data (16:00, about an hour) and the
|
||||
# Daily Snapshot (23:00). A data job still running only delays this one: the
|
||||
# data-write group queues runs (queue: max) instead of evicting them.
|
||||
#
|
||||
# The one-time backfill is a manual run with the box ticked: every game, about
|
||||
# four hours. Start it in the evening UTC, after Refresh Store Data.
|
||||
@@ -30,15 +32,20 @@ permissions: {contents: write}
|
||||
concurrency:
|
||||
group: data-write
|
||||
cancel-in-progress: false
|
||||
queue: max
|
||||
|
||||
jobs:
|
||||
refresh:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: ${{ inputs.backfill && 330 || 30 }}
|
||||
timeout-minutes: ${{ inputs.backfill && 340 || 45 }}
|
||||
steps:
|
||||
# main as it is when the job STARTS: a run that waited its turn must
|
||||
# work on what the job before it committed.
|
||||
- uses: actions/checkout@v6
|
||||
with: {ref: main}
|
||||
- uses: actions/setup-python@v6
|
||||
with: {python-version: '3.15'}
|
||||
- run: chmod +x ./bash/refresh_peaks.sh && ./bash/refresh_peaks.sh
|
||||
env:
|
||||
PEAKS_ALL: ${{ inputs.backfill }}
|
||||
JOB_BUDGET_MINUTES: ${{ inputs.backfill && '300' || '25' }}
|
||||
@@ -5,22 +5,23 @@ name: Refresh Store Data
|
||||
# store page, every game once a week; name, header image, description,
|
||||
# developer/publisher, release date, platforms, Metacritic and DRM notes from
|
||||
# appdetails, every game once a month. Each daily run takes one slice of each
|
||||
# rotation - roughly 630 store pages and 150 appdetails calls, ~40 minutes.
|
||||
# rotation - about 1,400 store pages and 330 appdetails calls at 9,800 games,
|
||||
# an hour. Past 2,000 pages or 500 appdetails a day a cycle doubles instead of
|
||||
# the run growing, so it stays under an hour and a half.
|
||||
# scripts/refresh_store_data.py explains the rotation and the rules (never
|
||||
# blanks a value, never touches MANUAL_FIELDS).
|
||||
#
|
||||
# Then the anti-cheat sweep, committed separately: every game's anti-cheat as
|
||||
# its developer declares it on Steam, about 160 GetItems requests plus a store
|
||||
# its developer declares it on Steam, about 200 GetItems requests plus a store
|
||||
# page for each disclosure Steam does not name there (~70), a few minutes.
|
||||
# scripts/refresh_anticheat.py has its rules: Steam wins, an override wins
|
||||
# over Steam, and no disclosure never clears a value. `only: anticheat` runs
|
||||
# just the sweep.
|
||||
#
|
||||
# 16:00 UTC is the quiet part of this repository's day. Scheduled runs here
|
||||
# start two to five hours late, the morning data jobs (reviews, dead links,
|
||||
# purge) run for two to four hours, and the data-write group keeps only ONE
|
||||
# pending run: a second one queueing behind a running job cancels the first.
|
||||
# Nothing else long is due this late; the Daily Snapshot (23:00) comes after.
|
||||
# 16:00 UTC is the quiet part of this repository's day: scheduled runs here
|
||||
# start up to an hour late, and the morning jobs are done by then. A data job
|
||||
# that is still running only delays this one - the data-write group queues
|
||||
# runs (queue: max) instead of evicting the one waiting.
|
||||
|
||||
on:
|
||||
schedule: [{cron: '0 16 * * *'}]
|
||||
@@ -38,11 +39,12 @@ permissions: {contents: write}
|
||||
concurrency:
|
||||
group: data-write
|
||||
cancel-in-progress: false
|
||||
queue: max
|
||||
|
||||
jobs:
|
||||
refresh:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 120
|
||||
timeout-minutes: 150
|
||||
steps:
|
||||
# main, not the triggering commit: a run that waited its turn in
|
||||
# data-write must work on the data as it is now, not as it was queued.
|
||||
@@ -53,3 +55,6 @@ jobs:
|
||||
- run: chmod +x ./bash/refresh_store.sh && ./bash/refresh_store.sh
|
||||
env:
|
||||
REFRESH_ONLY: ${{ inputs.only }}
|
||||
# The store refresh's budget (core/budget.py); the anti-cheat sweep
|
||||
# after it takes a few minutes and has none.
|
||||
JOB_BUDGET_MINUTES: '100'
|
||||
+6
-3
@@ -1,7 +1,10 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
echo "── Force re-fetch ALL games (v2.1 schema) ──"
|
||||
# REFETCH_PART is the manual run's "part" box, K/N (scripts/refetch_all.py
|
||||
# checks it): the whole catalogue no longer fits in one job.
|
||||
part="${REFETCH_PART:-1/1}"
|
||||
echo "── Force re-fetch, part ${part} ──"
|
||||
|
||||
pip install --quiet --require-hashes --only-binary :all: -r requirements.txt
|
||||
python scripts/refetch_all.py
|
||||
bash "$(dirname "$0")/commit_push.sh" "Force re-fetch all games → v2.1 schema [$(date +'%Y-%m-%d')]"
|
||||
python scripts/refetch_all.py --part "$part"
|
||||
bash "$(dirname "$0")/commit_push.sh" "Force re-fetch, part ${part} [$(date +'%Y-%m-%d')]"
|
||||
+67
-20
@@ -1,15 +1,33 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Force re-fetch ALL games. MANUAL ONLY.
|
||||
Backup → migrate → clear fetchable → re-fetch → save.
|
||||
Backup → migrate → re-fetch each game from scratch → save.
|
||||
|
||||
At about six seconds a game the whole catalogue takes far longer than a job
|
||||
may run, so it is done in parts: --part K/N takes the games whose
|
||||
crc32(appid) % N is K-1 (core/rotation.py), and the workflow's runs queue
|
||||
one behind another. Each game is cleared and fetched on a copy that replaces
|
||||
it only once appdetails has answered, so a failed request - or the job
|
||||
budget running out (core/budget.py) - leaves every other game exactly as it
|
||||
was. The oldest last_updated goes first: running a part that was cut short
|
||||
again carries on where it stopped.
|
||||
"""
|
||||
import sys, os, shutil
|
||||
import argparse
|
||||
import copy
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from core.budget import Deadline
|
||||
from core.constants import DATA_DIR, STEAM_API_KEY
|
||||
from core.data_store import load_main, save_main, migrate_record, now_iso
|
||||
from core.data_store import extract_appid, load_main, save_main, migrate_record
|
||||
from core.fetcher import fetch_full, process_batch
|
||||
from core.overrides import OverrideGuard
|
||||
from core.rotation import slice_of
|
||||
from core.steam_client import get_client
|
||||
|
||||
CLEARABLE = {
|
||||
@@ -49,13 +67,51 @@ def clear_fetchable(game):
|
||||
game["metacritic"] = "N/A"
|
||||
|
||||
|
||||
def main():
|
||||
def parse_part(text: str) -> tuple[int, int]:
|
||||
"""'K/N' -> (K, N) with 1 <= K <= N."""
|
||||
m = re.fullmatch(r"\s*(\d+)\s*/\s*(\d+)\s*", text or "")
|
||||
if not m or not 1 <= int(m.group(1)) <= int(m.group(2)):
|
||||
raise argparse.ArgumentTypeError(f"expected K/N with 1 <= K <= N, got {text!r}")
|
||||
return int(m.group(1)), int(m.group(2))
|
||||
|
||||
|
||||
def in_part(game: dict, part: tuple[int, int]) -> bool:
|
||||
appid = extract_appid(game.get("link", ""))
|
||||
return bool(appid) and slice_of(appid, part[1]) == part[0] - 1
|
||||
|
||||
|
||||
def refetch_one(game: dict, client, guard) -> bool:
|
||||
"""Re-fetch one game from scratch, in place, or leave it exactly as it was
|
||||
when appdetails does not answer. Returns whether it was re-fetched."""
|
||||
appid = extract_appid(game.get("link", ""))
|
||||
if not appid:
|
||||
return False
|
||||
status, details = client.fetch_app_details_full(appid)
|
||||
if status != "ok" or not details:
|
||||
return False
|
||||
fresh = copy.deepcopy(game)
|
||||
clear_fetchable(fresh)
|
||||
fetch_full(fresh, client=client, prefetched_details=details, guard=guard)
|
||||
game.clear()
|
||||
game.update(fresh)
|
||||
return True
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0].strip())
|
||||
ap.add_argument("--part", type=parse_part, default=(1, 1),
|
||||
help="K/N: re-fetch only the K-th of N parts of the catalogue (default 1/1)")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
deadline = Deadline.from_env()
|
||||
games = load_main()
|
||||
if not games:
|
||||
print("Empty.")
|
||||
return
|
||||
total = len(games)
|
||||
print(f"Force re-fetch: {total} games\n")
|
||||
targets = [g for g in games if in_part(g, args.part)]
|
||||
# Oldest first: a part cut short carries on where it stopped next time.
|
||||
targets.sort(key=lambda g: g.get("last_updated") or "")
|
||||
print(f"Force re-fetch: part {args.part[0]}/{args.part[1]}, {len(targets)} of {len(games)} games\n")
|
||||
|
||||
backup_dir = DATA_DIR + ".bak"
|
||||
if os.path.isdir(backup_dir):
|
||||
@@ -63,30 +119,21 @@ def main():
|
||||
shutil.copytree(DATA_DIR, backup_dir)
|
||||
print(f"Backup -> {backup_dir}")
|
||||
|
||||
# Migrate + clear
|
||||
for g in games:
|
||||
migrate_record(g)
|
||||
clear_fetchable(g)
|
||||
|
||||
print(f"Cleared fetchable fields. Re-fetching...\n")
|
||||
if not STEAM_API_KEY:
|
||||
print("⚠ No STEAM_API_KEY – players skipped")
|
||||
|
||||
client = get_client()
|
||||
guard = OverrideGuard.load()
|
||||
process_batch(games, lambda g: fetch_full(g, client=client, guard=guard), "Re-fetching")
|
||||
|
||||
stamp = now_iso()
|
||||
for g in games:
|
||||
g["last_updated"] = stamp
|
||||
tally: Counter = Counter()
|
||||
process_batch(targets, lambda g: tally.update([refetch_one(g, client, guard)]),
|
||||
"Re-fetching", deadline=deadline)
|
||||
save_main(games)
|
||||
|
||||
# Stats
|
||||
has = lambda k: sum(1 for g in games if g.get(k) and g[k] not in ("", "N/A", [], False))
|
||||
print(f"\n{'═'*50}")
|
||||
print(f"Done {total} games. Backup: {backup_dir}")
|
||||
print(f" name: {has('name')} | dev: {has('developer')} | pub: {has('publisher')}")
|
||||
print(f" plat: {has('platforms')} | lang: {has('languages')} | tags: {has('tags')}")
|
||||
print(f"Re-fetched {tally[True]} of {len(targets)} ({tally[False]} left as they were: "
|
||||
f"appdetails did not answer). Backup: {backup_dir}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Refresh all_time_peak - the most players each game has had at once - from
|
||||
SteamCharts, a thirtieth of the catalogue a day.
|
||||
SteamCharts, a thirtieth of the catalogue a day (less, once that passes
|
||||
PEAK_DAILY games).
|
||||
|
||||
core/peaks.py explains the field; core/steamcharts.py the source and why it is
|
||||
read so carefully. Each run does two things:
|
||||
@@ -12,7 +13,9 @@ read so carefully. Each run does two things:
|
||||
field for the whole catalogue on the first run.
|
||||
2. For the games whose slice is due (crc32(appid) % 30, see core/rotation.py:
|
||||
every game once a month), read SteamCharts' recorded all-time peak and
|
||||
raise the field to it.
|
||||
raise the field to it. Past PEAK_DAILY games a day the cycle grows to
|
||||
60, 90, ... days (rotation.cycle_for), never shorter: SteamCharts is still
|
||||
asked about a game at most once a month.
|
||||
|
||||
The rules:
|
||||
- all_time_peak only ever goes UP. A failed request, an untracked app or a
|
||||
@@ -23,8 +26,9 @@ The rules:
|
||||
never seen; both are normal and simply skipped until next month.
|
||||
- A 403, a 429 or a Cloudflare challenge stops the SteamCharts part of the
|
||||
run (everything already raised is kept and saved), and so do ten errors in
|
||||
a row. The run still succeeds, with a warning: a refusal is SteamCharts'
|
||||
call to make, not a pipeline failure.
|
||||
a row, and so does the job's time budget (core/budget.py). The run still
|
||||
succeeds, with a warning: a refusal is SteamCharts' call to make, not a
|
||||
pipeline failure.
|
||||
- Everything is written by save_main().
|
||||
|
||||
Usage (from the repository root):
|
||||
@@ -41,12 +45,16 @@ from typing import Optional
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from core.budget import Deadline, expired
|
||||
from core.data_store import extract_appid, load_main, migrate_record, save_main
|
||||
from core.peaks import observed_peak, raise_peak
|
||||
from core.rotation import day_number, due
|
||||
from core.rotation import cycle_for, daily_order, day_number, due
|
||||
from core.steamcharts import SteamChartsClient
|
||||
|
||||
PEAK_CYCLE_DAYS = 30
|
||||
# The most SteamCharts pages a day before the cycle grows (rotation.cycle_for):
|
||||
# about 17 minutes at 2-3 s a page.
|
||||
PEAK_DAILY = 400
|
||||
MAX_CONSECUTIVE_ERRORS = 10
|
||||
|
||||
|
||||
@@ -55,13 +63,19 @@ def raise_from_samples(games: list[dict]) -> int:
|
||||
return sum(1 for g in games if raise_peak(g, observed_peak(g)))
|
||||
|
||||
|
||||
def raise_from_steamcharts(targets: list[dict], client, log=print) -> Counter:
|
||||
def raise_from_steamcharts(targets: list[dict], client, log=print,
|
||||
deadline: Optional[Deadline] = None) -> Counter:
|
||||
"""Step 2. Returns a tally of the outcomes: "raised", "unchanged",
|
||||
"untracked", "error", and "skipped" for games never asked because the run
|
||||
stopped early."""
|
||||
tally: Counter = Counter()
|
||||
errors_in_a_row = 0
|
||||
for i, game in enumerate(targets):
|
||||
if expired(deadline):
|
||||
tally["skipped"] += len(targets) - i
|
||||
log(f"::warning::The job's time budget ran out with {len(targets) - i} SteamCharts "
|
||||
f"pages left; those games are asked on their next turn.")
|
||||
break
|
||||
appid = extract_appid(game.get("link", ""))
|
||||
if not appid:
|
||||
continue
|
||||
@@ -101,6 +115,7 @@ def main(argv: Optional[list[str]] = None) -> None:
|
||||
ap.add_argument("--dry-run", action="store_true", help="fetch and report, write nothing")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
deadline = Deadline.from_env()
|
||||
games = load_main()
|
||||
for g in games:
|
||||
migrate_record(g)
|
||||
@@ -109,13 +124,14 @@ def main(argv: Optional[list[str]] = None) -> None:
|
||||
print(f"{len(games)} games; all_time_peak raised from our own samples for {from_samples}")
|
||||
|
||||
day = day_number() if args.day is None else args.day
|
||||
targets = games if args.all else [g for g in games if due(g, PEAK_CYCLE_DAYS, day)]
|
||||
cycle = cycle_for(len(games), PEAK_CYCLE_DAYS, PEAK_DAILY)
|
||||
targets = games if args.all else [g for g in daily_order(games, day) if due(g, cycle, day)]
|
||||
if args.limit is not None:
|
||||
targets = targets[: args.limit]
|
||||
scope = "every game" if args.all else f"slice {day % PEAK_CYCLE_DAYS}/{PEAK_CYCLE_DAYS}"
|
||||
scope = "every game" if args.all else f"slice {day % cycle}/{cycle}"
|
||||
print(f"SteamCharts: {len(targets)} games ({scope})")
|
||||
|
||||
tally = raise_from_steamcharts(targets, SteamChartsClient()) if targets else Counter()
|
||||
tally = raise_from_steamcharts(targets, SteamChartsClient(), deadline=deadline) if targets else Counter()
|
||||
summary = ", ".join(f"{k} {v}" for k, v in tally.most_common()) or "nothing asked"
|
||||
print(f"\nSteamCharts: {summary}")
|
||||
|
||||
|
||||
@@ -18,8 +18,13 @@ Two groups, each on its own rotation, a slice of the catalogue per daily run:
|
||||
developer, publisher, release_date, platforms,
|
||||
metacritic, drm_notes every 30 days
|
||||
|
||||
That is ~630 store pages and ~150 appdetails calls a day - about 40 minutes
|
||||
at the client's throttle - instead of 8,800 requests in one run.
|
||||
That was ~1,400 store pages and ~330 appdetails calls a day at 9,800 games,
|
||||
about an hour at the client's throttle. The cycles grow with the catalogue
|
||||
(core/rotation.cycle_for): past PAGE_DAILY pages or DETAILS_DAILY appdetails
|
||||
a day, a cycle doubles (triples, ...) instead of the run growing, and a game
|
||||
keeps its weekday and its day of the month. A run that still meets its time
|
||||
budget (core/budget.py) saves what it did; the slice is visited in an order
|
||||
that changes daily, so the cut falls elsewhere next time.
|
||||
|
||||
A game's slice is crc32(appid) % cycle, NOT appid % cycle: Steam appids are
|
||||
overwhelmingly multiples of 10, so `% 30` would leave 27 of the 30 monthly
|
||||
@@ -37,7 +42,10 @@ The rules, because these fields were fill-if-empty until now:
|
||||
gate, a delisted app or a changed page layout leaves the stored value alone.
|
||||
- MANUAL_FIELDS are never written (genre, type_game, anti_cheat, safe, ...),
|
||||
and neither is `notes`: health checks and "no longer free" belong to
|
||||
check_dead_links.py, purge_unhealthy.py and mark_dead_games.py.
|
||||
purge_unhealthy.py and mark_dead_games.py. A game appdetails no longer
|
||||
calls free is only listed, in a ::warning:: - purge removes a game only
|
||||
when Steam's store API flags it too, and this is where the opposite case
|
||||
shows.
|
||||
- has_paid_dlc is read only from a real store page - one with a language
|
||||
table or tags - or, for a gated game, from the prices of its DLC. An
|
||||
age-gate page has no DLC section and would read as False; when neither
|
||||
@@ -60,26 +68,31 @@ Usage (from the repository root):
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import zlib
|
||||
from collections import Counter
|
||||
from typing import Optional
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from core.budget import Deadline
|
||||
from core.data_store import (
|
||||
ELLIPSIS, extract_appid, is_empty, load_main, migrate_record, now_iso, save_main,
|
||||
truncate_description,
|
||||
)
|
||||
from core.fetcher import process_batch, scrape_store_fields
|
||||
from core.health_checker import free_per_appdetails
|
||||
from core.normalize import (
|
||||
clean_list, clean_text, decode_entities, normalize_records, normalize_release_date,
|
||||
)
|
||||
# The rotation (crc32 slices) is shared with refresh_peaks.py.
|
||||
from core.rotation import day_number, due
|
||||
from core.rotation import cycle_for, daily_order, day_number, due
|
||||
from core.steam_client import get_client
|
||||
|
||||
PAGE_CYCLE_DAYS = 7
|
||||
DETAILS_CYCLE_DAYS = 30
|
||||
# The most a day's slice may hold before its cycle grows (rotation.cycle_for):
|
||||
# with the repairs, about an hour and a half at the client's throttle.
|
||||
PAGE_DAILY = 2000
|
||||
DETAILS_DAILY = 500
|
||||
# Games with empty store fields taken a day on top of the rotation. 25 on
|
||||
# 2026-10-10; the cap is for the day something breaks for every game at once,
|
||||
# so a two-hour job cannot turn into a whole-catalogue crawl.
|
||||
@@ -205,8 +218,10 @@ def apply_app_details(game: dict, data: dict) -> list[str]:
|
||||
return changed
|
||||
|
||||
|
||||
def refresh_game(game: dict, client, page: bool, details: bool) -> list[str]:
|
||||
"""Refresh one game in place. Returns the fields that changed."""
|
||||
def refresh_game(game: dict, client, page: bool, details: bool,
|
||||
not_free: Optional[list] = None) -> list[str]:
|
||||
"""Refresh one game in place. Returns the fields that changed. A game
|
||||
whose appdetails no longer call it free is added to `not_free`."""
|
||||
appid = extract_appid(game.get("link", ""))
|
||||
if not appid:
|
||||
return []
|
||||
@@ -216,6 +231,8 @@ def refresh_game(game: dict, client, page: bool, details: bool) -> list[str]:
|
||||
data = client.fetch_app_details(appid)
|
||||
if data:
|
||||
changed += apply_app_details(game, data)
|
||||
if not_free is not None and not free_per_appdetails(data):
|
||||
not_free.append(f"{appid} {game.get('name', '')[:40]}")
|
||||
if page:
|
||||
# The appdetails just fetched spare the DLC check a second request.
|
||||
scraped = scrape_store_fields(client, appid, details=data)
|
||||
@@ -233,29 +250,31 @@ def missing_store_fields(game: dict) -> bool:
|
||||
|
||||
|
||||
def build_plan(games: list[dict], day: int, want_page: bool = True,
|
||||
want_details: bool = True, missing_only: bool = False) -> list[tuple]:
|
||||
want_details: bool = True, missing_only: bool = False,
|
||||
page_cycle: int = PAGE_CYCLE_DAYS,
|
||||
details_cycle: int = DETAILS_CYCLE_DAYS) -> list[tuple]:
|
||||
"""[(game, page, details)] for one day: the games with empty store fields
|
||||
first (page group, every day), then the day's rotation slices.
|
||||
|
||||
The repairs are capped at MISSING_DAILY_CAP and ordered by a hash that
|
||||
changes daily, so a residue Steam never fills cannot hold the same places
|
||||
forever. missing_only keeps just the repairs, store page only.
|
||||
Both parts come in an order that changes daily (rotation.daily_order):
|
||||
the repairs are capped at MISSING_DAILY_CAP, so a residue Steam never
|
||||
fills cannot hold the same places forever, and a run that meets its time
|
||||
budget leaves a different part of the slice for the next one.
|
||||
missing_only keeps just the repairs, store page only.
|
||||
"""
|
||||
repairs: list[dict] = []
|
||||
if want_page:
|
||||
repairs = [g for g in games if missing_store_fields(g)]
|
||||
repairs.sort(key=lambda g: zlib.crc32(f"{day}:{extract_appid(g.get('link', ''))}".encode()))
|
||||
repairs = repairs[:MISSING_DAILY_CAP]
|
||||
repairs = daily_order([g for g in games if missing_store_fields(g)], day)[:MISSING_DAILY_CAP]
|
||||
if missing_only:
|
||||
return [(g, True, False) for g in repairs]
|
||||
|
||||
plan = [(g, True, want_details and due(g, DETAILS_CYCLE_DAYS, day)) for g in repairs]
|
||||
plan = [(g, True, want_details and due(g, details_cycle, day)) for g in repairs]
|
||||
taken = {id(g) for g in repairs}
|
||||
for g in games:
|
||||
for g in daily_order(games, day):
|
||||
if id(g) in taken:
|
||||
continue
|
||||
page = want_page and due(g, PAGE_CYCLE_DAYS, day)
|
||||
details = want_details and due(g, DETAILS_CYCLE_DAYS, day)
|
||||
page = want_page and due(g, page_cycle, day)
|
||||
details = want_details and due(g, details_cycle, day)
|
||||
if page or details:
|
||||
plan.append((g, page, details))
|
||||
return plan
|
||||
@@ -273,6 +292,7 @@ def main(argv: Optional[list[str]] = None) -> None:
|
||||
ap.add_argument("--dry-run", action="store_true", help="fetch and report, write nothing")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
deadline = Deadline.from_env()
|
||||
day = day_number() if args.day is None else args.day
|
||||
games = load_main()
|
||||
for g in games:
|
||||
@@ -284,7 +304,10 @@ def main(argv: Optional[list[str]] = None) -> None:
|
||||
|
||||
want_page = args.only in (None, "page")
|
||||
want_details = args.only in (None, "details")
|
||||
plan = build_plan(games, day, want_page, want_details, missing_only=args.missing_only)
|
||||
page_cycle = cycle_for(len(games), PAGE_CYCLE_DAYS, PAGE_DAILY)
|
||||
details_cycle = cycle_for(len(games), DETAILS_CYCLE_DAYS, DETAILS_DAILY)
|
||||
plan = build_plan(games, day, want_page, want_details, missing_only=args.missing_only,
|
||||
page_cycle=page_cycle, details_cycle=details_cycle)
|
||||
if args.limit is not None:
|
||||
plan = plan[: args.limit]
|
||||
|
||||
@@ -292,28 +315,33 @@ def main(argv: Optional[list[str]] = None) -> None:
|
||||
n_details = sum(1 for _, _, d in plan if d)
|
||||
n_missing = sum(1 for g, p, _ in plan if p and missing_store_fields(g))
|
||||
print(f"{len(games)} games; day {day}: store page for {n_page} "
|
||||
f"(slice {day % PAGE_CYCLE_DAYS}/{PAGE_CYCLE_DAYS}, {n_missing} with empty store fields), "
|
||||
f"appdetails for {n_details} (slice {day % DETAILS_CYCLE_DAYS}/{DETAILS_CYCLE_DAYS})")
|
||||
f"(slice {day % page_cycle}/{page_cycle}, {n_missing} with empty store fields), "
|
||||
f"appdetails for {n_details} (slice {day % details_cycle}/{details_cycle})")
|
||||
if not plan:
|
||||
return
|
||||
|
||||
client = get_client()
|
||||
flags = {id(g): (page, details) for g, page, details in plan}
|
||||
fields: Counter = Counter()
|
||||
not_free: list[str] = []
|
||||
touched = 0
|
||||
|
||||
def refresh(game: dict) -> None:
|
||||
nonlocal touched
|
||||
page, details = flags[id(game)]
|
||||
changed = refresh_game(game, client, page=page, details=details)
|
||||
changed = refresh_game(game, client, page=page, details=details, not_free=not_free)
|
||||
if changed:
|
||||
touched += 1
|
||||
fields.update(changed)
|
||||
|
||||
process_batch([g for g, _, _ in plan], refresh, "Refreshing")
|
||||
done = process_batch([g for g, _, _ in plan], refresh, "Refreshing", deadline=deadline)
|
||||
|
||||
summary = ", ".join(f"{f} {n}" for f, n in fields.most_common()) or "nothing"
|
||||
print(f"\n{touched} of {len(plan)} games changed: {summary}")
|
||||
print(f"\n{touched} of {done} games changed: {summary}")
|
||||
if not_free:
|
||||
print(f"::warning::appdetails no longer calls {len(not_free)} game(s) free while Steam's "
|
||||
f"store API still does, so purge keeps them: {'; '.join(not_free[:20])}"
|
||||
f"{' …' if len(not_free) > 20 else ''}. Remove one in /admin if it is so.")
|
||||
if args.dry_run:
|
||||
print("Dry run: nothing written.")
|
||||
return
|
||||
|
||||
@@ -1,8 +1,15 @@
|
||||
"""
|
||||
refetch_all.py clears what Steam can give back and keeps what a person
|
||||
decided. Run from the repository root: python -m pytest scripts/tests
|
||||
decided, one game at a time: a game appdetails does not answer for, or the
|
||||
ones a part cut short by its budget never reached, stay exactly as they were.
|
||||
Run from the repository root: python -m pytest scripts/tests
|
||||
"""
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
import refetch_all
|
||||
from core import budget, data_store
|
||||
|
||||
|
||||
def test_clear_fetchable_resets_a_default_to_the_skeleton_s_value():
|
||||
@@ -19,3 +26,93 @@ def test_clear_fetchable_keeps_a_decision():
|
||||
refetch_all.clear_fetchable(game)
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"], game["genre"]) == \
|
||||
("BattlEye", "BattlEye [Kernel]", True, "FPS")
|
||||
|
||||
|
||||
# ──────────── In parts, game by game ────────────
|
||||
|
||||
|
||||
def _rec(appid: str, **fields) -> dict:
|
||||
base = {"link": f"https://store.steampowered.com/app/{appid}/", "name": f"Game {appid}",
|
||||
"description": "kept", "tags": ["FPS"], "reviews": "50% (Mixed)", "genre": "FPS",
|
||||
"anti_cheat": "-", "anti_cheat_note": "", "is_kernel_ac": None, "type_game": "online",
|
||||
"notes": "Not reviewed yet", "last_updated": "2026-01-01T00:00:00Z"}
|
||||
base.update(fields)
|
||||
return base
|
||||
|
||||
|
||||
class _Client:
|
||||
def __init__(self, failing=()):
|
||||
self.failing = set(failing)
|
||||
self.asked: list[str] = []
|
||||
|
||||
def fetch_app_details_full(self, appid):
|
||||
self.asked.append(appid)
|
||||
if appid in self.failing:
|
||||
return ("network_error", None)
|
||||
return ("ok", {"name": f"Fresh {appid}", "is_free": True})
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text, part", [("1/1", (1, 1)), (" 3 / 8 ", (3, 8)), ("8/8", (8, 8))])
|
||||
def test_a_part_is_read(text, part):
|
||||
assert refetch_all.parse_part(text) == part
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text", ["0/4", "5/4", "1/0", "x", "", "1/2/3"])
|
||||
def test_a_bad_part_is_refused(text):
|
||||
with pytest.raises(Exception):
|
||||
refetch_all.parse_part(text)
|
||||
|
||||
|
||||
def test_the_parts_split_the_catalogue():
|
||||
games = [_rec(str(10 * i)) for i in range(1, 501)]
|
||||
counts = [sum(refetch_all.in_part(g, (k, 8)) for g in games) for k in range(1, 9)]
|
||||
assert sum(counts) == len(games) and min(counts) > 0
|
||||
|
||||
|
||||
def test_a_game_appdetails_does_not_answer_for_is_left_as_it_was(monkeypatch):
|
||||
calls: list[str] = []
|
||||
monkeypatch.setattr(refetch_all, "fetch_full",
|
||||
lambda g, client=None, prefetched_details=None, guard=None: calls.append(g["link"]))
|
||||
game = _rec("730")
|
||||
before = dict(game)
|
||||
assert refetch_all.refetch_one(game, _Client(failing={"730"}), None) is False
|
||||
assert game == before and calls == []
|
||||
|
||||
|
||||
def test_a_game_is_cleared_and_fetched_on_a_copy(monkeypatch):
|
||||
def fake_fetch(g, client=None, prefetched_details=None, guard=None):
|
||||
assert g["description"] == "" and g["reviews"] == "N/A" # cleared before the fetch
|
||||
g["name"] = prefetched_details["name"]
|
||||
|
||||
monkeypatch.setattr(refetch_all, "fetch_full", fake_fetch)
|
||||
game = _rec("730", anti_cheat="BattlEye")
|
||||
assert refetch_all.refetch_one(game, _Client(), None) is True
|
||||
assert game["name"] == "Fresh 730" and game["anti_cheat"] == "BattlEye"
|
||||
|
||||
|
||||
def test_a_cut_part_leaves_every_other_game_alone_and_resumes_oldest_first(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
os.makedirs("data")
|
||||
games = [_rec(str(10 * i), last_updated=f"2026-01-{i:02d}T00:00:00Z") for i in range(1, 9)]
|
||||
data_store.save_main(games, apply_human_overrides=False)
|
||||
client = _Client()
|
||||
monkeypatch.setattr(refetch_all, "get_client", lambda: client)
|
||||
monkeypatch.setattr(refetch_all.OverrideGuard, "load", classmethod(lambda cls: None))
|
||||
monkeypatch.setattr(refetch_all, "STEAM_API_KEY", "")
|
||||
|
||||
clock = [0.0]
|
||||
monkeypatch.setattr(budget.time, "monotonic", lambda: clock[0])
|
||||
|
||||
def fake_fetch(g, client=None, prefetched_details=None, guard=None):
|
||||
clock[0] += 40
|
||||
g["name"] = prefetched_details["name"]
|
||||
g["last_updated"] = "2026-10-23T00:00:00Z"
|
||||
|
||||
monkeypatch.setattr(refetch_all, "fetch_full", fake_fetch)
|
||||
monkeypatch.setattr("core.fetcher.time.sleep", lambda s: None)
|
||||
monkeypatch.setenv(budget.ENV, "1")
|
||||
refetch_all.main(["--part", "1/1"])
|
||||
after = {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
|
||||
assert client.asked == ["10", "20"] # the two oldest, then the budget ran out
|
||||
assert after["10"]["name"] == "Fresh 10" and after["30"]["name"] == "Game 30"
|
||||
assert after["30"]["description"] == "kept" # never cleared
|
||||
@@ -239,3 +239,50 @@ def test_refetch_all_never_clears_it():
|
||||
# The other player fields are cleared and fetched again; this one cannot be.
|
||||
assert "all_time_peak" not in refetch_all.CLEARABLE
|
||||
assert "all_time_peak" not in refetch_all.CONDITIONAL
|
||||
|
||||
|
||||
def test_the_budget_stops_the_steamcharts_part_and_keeps_what_was_raised(monkeypatch):
|
||||
from core import budget
|
||||
|
||||
clock = [0.0]
|
||||
monkeypatch.setattr(budget.time, "monotonic", lambda: clock[0])
|
||||
deadline = budget.Deadline(1)
|
||||
games = [_game(str(i)) for i in range(1, 6)]
|
||||
|
||||
class Slow(_Client):
|
||||
def fetch(self, appid):
|
||||
clock[0] += 25
|
||||
return super().fetch(appid)
|
||||
|
||||
client = Slow({str(i): Result("ok", 100 * i) for i in range(1, 6)})
|
||||
lines: list[str] = []
|
||||
tally = rp.raise_from_steamcharts(games, client, log=lines.append, deadline=deadline)
|
||||
assert client.asked == ["1", "2", "3"]
|
||||
assert tally["raised"] == 3 and tally["skipped"] == 2
|
||||
assert any(line.startswith("::warning::") for line in lines)
|
||||
|
||||
|
||||
def test_a_larger_catalogue_is_asked_less_often_never_more():
|
||||
assert rp.cycle_for(9847, rp.PEAK_CYCLE_DAYS, rp.PEAK_DAILY) == 30
|
||||
assert rp.cycle_for(20000, rp.PEAK_CYCLE_DAYS, rp.PEAK_DAILY) == 60
|
||||
|
||||
|
||||
def test_a_run_asks_today_s_slice_and_saves(tmp_path, monkeypatch, capsys):
|
||||
import os
|
||||
|
||||
from core import budget, data_store, rotation
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
os.makedirs("data")
|
||||
monkeypatch.delenv(budget.ENV, raising=False)
|
||||
games = [_game(str(10 * i), current_players="0", peak_today="0") for i in range(1, 41)]
|
||||
data_store.save_main(games, apply_human_overrides=False)
|
||||
day = rotation.slice_of("10", rp.PEAK_CYCLE_DAYS) + 30 * 700
|
||||
client = _Client({str(10 * i): Result("ok", 1000 + i) for i in range(1, 41)})
|
||||
monkeypatch.setattr(rp, "SteamChartsClient", lambda: client)
|
||||
rp.main(["--day", str(day)])
|
||||
assert "10" in client.asked
|
||||
assert all(rotation.slice_of(a, 30) == day % 30 for a in client.asked)
|
||||
after = {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
|
||||
assert after["10"]["all_time_peak"] == "1,001"
|
||||
assert f"slice {day % 30}/30" in capsys.readouterr().out
|
||||
@@ -391,3 +391,62 @@ def test_a_blurb_that_now_fits_further_is_written_but_is_not_news():
|
||||
def test_a_rewritten_blurb_is_a_change():
|
||||
game = _game(description="Old words, cut short…")
|
||||
assert "description" in rsd.apply_app_details(game, {"short_description": "New words entirely."})
|
||||
|
||||
|
||||
# ──────────── A growing catalogue ────────────
|
||||
|
||||
def test_the_page_cycle_grows_with_the_catalogue_and_keeps_its_weekday():
|
||||
assert rsd.cycle_for(9847, rsd.PAGE_CYCLE_DAYS, rsd.PAGE_DAILY) == 7
|
||||
games = [_game(str(10 * i)) for i in range(1, 12001)]
|
||||
assert rsd.cycle_for(len(games), rsd.PAGE_CYCLE_DAYS, 1000) == 14
|
||||
for day in range(20000, 20014):
|
||||
plan = rsd.build_plan(games, day, want_details=False, page_cycle=14)
|
||||
assert len(plan) < 1000
|
||||
assert all(slice_of(rsd.extract_appid(g["link"]), 7) == day % 7 for g, _, _ in plan)
|
||||
|
||||
|
||||
def test_the_slice_comes_in_an_order_that_changes_daily():
|
||||
games = [_game(str(10 * i)) for i in range(1, 701)]
|
||||
first = [g["link"] for g, _, _ in rsd.build_plan(games, 20006, want_details=False)]
|
||||
again = [g["link"] for g, _, _ in rsd.build_plan(list(reversed(games)), 20006, want_details=False)]
|
||||
week_later = [g["link"] for g, _, _ in rsd.build_plan(games, 20013, want_details=False)]
|
||||
assert first == again
|
||||
assert sorted(first) == sorted(week_later) and first != week_later
|
||||
|
||||
|
||||
def test_a_game_appdetails_no_longer_call_free_is_listed_not_written():
|
||||
game = _game()
|
||||
before = dict(game)
|
||||
listed: list[str] = []
|
||||
details = {"name": "Counter-Strike 2", "is_free": False,
|
||||
"price_overview": {"initial": 1499, "final": 1499}}
|
||||
rsd.refresh_game(game, FakeClient(details=details), page=False, details=True, not_free=listed)
|
||||
assert listed == ["730 Counter-Strike 2"]
|
||||
assert game["notes"] == before["notes"]
|
||||
listed.clear()
|
||||
rsd.refresh_game(_game(), FakeClient(details={"name": "x", "is_free": True}), page=False, details=True,
|
||||
not_free=listed)
|
||||
assert listed == []
|
||||
|
||||
|
||||
def test_a_run_refreshes_today_s_slice_and_lists_what_appdetails_calls_paid(tmp_path, monkeypatch, capsys):
|
||||
import os
|
||||
|
||||
from core import budget, data_store
|
||||
|
||||
monkeypatch.chdir(tmp_path)
|
||||
os.makedirs("data")
|
||||
monkeypatch.delenv(budget.ENV, raising=False)
|
||||
monkeypatch.setattr("core.fetcher.time.sleep", lambda s: None)
|
||||
games = [_game(str(10 * i), name=f"Game {i}") for i in range(1, 61)]
|
||||
data_store.save_main(games, apply_human_overrides=False)
|
||||
client = FakeClient(details={"name": "Renamed", "is_free": False, "price_overview": {"initial": 999}})
|
||||
monkeypatch.setattr(rsd, "get_client", lambda: client)
|
||||
rsd.main(["--only", "details", "--day", "20030"])
|
||||
out = capsys.readouterr().out
|
||||
asked = [c.split(":")[1] for c in client.calls]
|
||||
assert asked and all(rsd.due(_game(a), 30, 20030) for a in asked)
|
||||
assert "slice 20/30" in out
|
||||
assert "::warning::appdetails no longer calls" in out
|
||||
after = {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
|
||||
assert all(after[a]["name"] == "Renamed" for a in asked)
|
||||
Reference in new issue
Block a user