feat(pipeline): read reviews for every game from Steam's store API

Update Reviews asked appreviews about one game at a time, two seconds a
game. At 9,847 games it ran into GitHub's six-hour limit on 2026-10-23
and was stopped with nothing saved; five of its ten runs before that
were cancelled while waiting.

- SteamClient.fetch_items(): GetItems for a whole catalogue, a failed
  batch costing only its own ids (the anti-cheat sweep now uses it).
- update_reviews.py reads review_summary() for every game, ~2 minutes,
  daily. Steam floors its percentage; reviews_from_summary() floors
  appreviews in integers too, so an ingest and the refresh agree. A
  count of 0 is never written on GetItems' word (it said 0 for a game
  with thousands): a stored percentage or "N/A" is asked on appreviews,
  at most 300 a run. last_updated moves on a change, and on a 14-20 day
  heartbeat that keeps the site's 30-day "stale" badge quiet without
  rewriting every record every run.
- core/budget.py: JOB_BUDGET_MINUTES. Past it process_batch() starts no
  further game, so a job saves what it did instead of being killed;
  a ::warning:: when a loop used three quarters of its budget.
- rotation.cycle_for() / daily_order(): cycles that grow with the
  catalogue as multiples of their base, and an order that changes daily.
- The player-count loops skip the batch pauses meant for store pages
  (Top Online ~27 min instead of 48) and take a deadline, like
  update_all_full().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 committed 2026-10-23 22:21:59 +07:00
1 parent 01a631bb22
commit 49f0c56ed7
17 files changed
+1000 -55

No files matched your search

+22 -1
View File
@@ -1,19 +1,40 @@
name : Update Reviews
# Every game's review summary from IStoreBrowseService/GetItems, fifty games a
# request: about two minutes for the whole catalogue (scripts/update_reviews.py
# has the rules). It used to ask appreviews about one game at a time and, at
# 9,800 games, ran into GitHub's six-hour limit with nothing saved.
on :
schedule : [ { cron : '0 6 */2 * *' } ]
schedule : [ { cron : '0 6 * * *' } ]
workflow_dispatch :
inputs :
dry_run :
description : 'Dry run: read Steam, write nothing'
type : boolean
default : false
permissions : { contents : write }
# queue: max - a run that finds another data job running waits its turn
# (first in, first out) instead of evicting the run already waiting.
concurrency :
group : data-write
cancel-in-progress : false
queue : max
jobs :
reviews :
runs-on : ubuntu-latest
timeout-minutes : 60
steps :
-
uses : actions/checkout@v6
# main as it is when the job STARTS: a run that waited its turn must
# work on what the job before it committed.
with : { ref : main }
-
uses : actions/setup-python@v6
with : { python-version : '3.15' }
-
run : chmod +x ./bash/reviews.sh && ./bash/reviews.sh
env :
JOB_BUDGET_MINUTES : '40'
REVIEWS_DRY_RUN : ${{ inputs.dry_run }}
+5
View File
@@ -1,5 +1,10 @@
#!/bin/bash
set -euo pipefail
pip install --quiet --require-hashes --only-binary :all: -r requirements.txt
# REVIEWS_DRY_RUN is the manual run's "dry run" box: read Steam, write nothing.
if [[ "${REVIEWS_DRY_RUN:-}" == "true" ]]; then
python scripts/update_reviews.py --dry-run
exit 0
fi
python scripts/update_reviews.py
bash "$(dirname "$0")/commit_push.sh" "Update reviews [$(date +'%Y-%m-%d')]"
+81
View File
@@ -0,0 +1,81 @@
"""
How long a data job may keep starting work, so that a catalogue which outgrows
its schedule makes a run partial instead of making it fail.
A GitHub-hosted job is stopped at its timeout-minutes (six hours at most), and
a stopped job commits nothing: Update Reviews ran for exactly six hours on
2026-10-23 and threw away every review it had read. Each data workflow sets
JOB_BUDGET_MINUTES at least twenty minutes below its timeout. A loop that
reaches the budget starts no further game, the script saves what it did, and
the job commits it. The games it did not reach keep their values until the
next run, and since the loops visit their games in an order that changes daily
(core/rotation.daily_order) the next cut falls elsewhere.
A loop that ran out, or used more than three quarters of the budget, says so
with a ::warning:: annotation on the run: the sign that the catalogue is
outgrowing the job comes before any run is lost.
JOB_BUDGET_MINUTES unset, empty, zero or not a number means no deadline, so a
run on a laptop works as it always did.
"""
import math
import os
import time
from typing import Optional
ENV = "JOB_BUDGET_MINUTES"
WARN_SHARE = 0.75
class Deadline:
"""A budget of minutes counted from when it was made: from the start of
the script, so loading and saving the catalogue count too."""
__slots__ = ("start", "seconds")
def __init__(self, minutes: float, now: Optional[float] = None):
self.start = time.monotonic() if now is None else now
self.seconds = minutes * 60
@classmethod
def from_env(cls) -> Optional["Deadline"]:
try:
minutes = float(os.environ.get(ENV, "").strip())
except ValueError:
return None
if not math.isfinite(minutes) or minutes <= 0:
return None
return cls(minutes)
@property
def minutes(self) -> float:
return self.seconds / 60
def used(self) -> float:
"""The share of the budget spent so far, 0.0 to 1.0 and beyond."""
return (time.monotonic() - self.start) / self.seconds
def expired(self) -> bool:
return self.used() >= 1.0
def expired(deadline: Optional[Deadline]) -> bool:
return deadline is not None and deadline.expired()
def report(desc: str, done: int, total: int, deadline: Optional[Deadline], log=print) -> None:
"""One line on how a loop used the budget. A run that stopped short, or
came close, gets a ::warning:: - it is how a growing catalogue shows up in
the run's annotations before it costs one."""
if deadline is None:
return
spent = deadline.used() * deadline.minutes
if done < total:
log(f"::warning::{desc}: stopped after {done} of {total} when the job's "
f"{deadline.minutes:.0f}-minute budget ran out; the other {total - done} keep "
f"their values until the next run.")
elif deadline.used() > WARN_SHARE:
log(f"::warning::{desc}: {done} done in {spent:.0f} of the job's {deadline.minutes:.0f} "
f"budgeted minutes ({deadline.used():.0%}). The catalogue is outgrowing this "
f"schedule: give the job a longer cycle or a larger budget.")
else:
log(f"{desc}: {done} done in {spent:.0f} of {deadline.minutes:.0f} budgeted minutes.")
+57 -30
View File
@@ -12,6 +12,7 @@ import time
from typing import Optional
from .anticheat import apply_anti_cheat, from_categories
from .budget import Deadline, report
from .constants import (
STEAM_API_KEY, SKIP_GENRE_TAGS,
BATCH_SIZE, BATCH_PAUSE_MIN, BATCH_PAUSE_MAX,
@@ -21,6 +22,7 @@ from .data_store import extract_appid, now_iso, is_info_complete, is_empty
from .normalize import decode_entities, normalize_release_date
from .overrides import OverrideGuard
from .peaks import raise_peak
from .rotation import daily_order, day_number
from .scraper import scrape_store_page
from .store_items import paid_dlc_from_items, parse_store_item
@@ -139,15 +141,27 @@ def apply_details(game: dict, data: dict) -> dict:
return game
def reviews_text(total: int, pct: int, label: str) -> str:
"""The stored shape of a review summary: "85% (Very Positive)"."""
if total <= 0:
return "No reviews"
return f"{pct}% ({label})" if label else f"{pct}%"
def reviews_from_summary(summary: dict) -> str:
"""appreviews' query_summary as stored. The percentage is FLOORED, as
Steam floors its own (GetItems answered 85 for Counter-Strike 2 where
round() had given 86), so the daily refresh from GetItems
(update_reviews.py) and a game's first fetch here agree. In integers:
in floating point 29/100*100 is 28.999... and floors to 28."""
total = int(summary.get("total_reviews") or 0)
pos = int(summary.get("total_positive") or 0)
pct = pos * 100 // total if total > 0 else 0
return reviews_text(total, pct, summary.get("review_score_desc", ""))
def apply_reviews(game: dict, summary: dict) -> dict:
total = summary.get("total_reviews", 0)
pos = summary.get("total_positive", 0)
if total > 0:
pct = round((pos / total) * 100)
label = summary.get("review_score_desc", "")
game["reviews"] = f"{pct}% ({label})" if label else f"{pct}%"
else:
game["reviews"] = "No reviews"
game["reviews"] = reviews_from_summary(summary)
return game
@@ -307,26 +321,44 @@ def fetch_full(game: dict, client=None, fetch_players=True,
# ──────────── Batch processor ────────────
def process_batch(games: list[dict], fn, desc="Processing"):
"""Run fn on each game with batch pauses."""
def process_batch(games: list[dict], fn, desc="Processing", pause: bool = True,
deadline: Optional[Deadline] = None) -> int:
"""Run fn on each game, BATCH_SIZE at a time. Returns how many it ran on.
pause: the 15-30 s rest between batches that keeps the store-page and
appdetails loops polite. The keyed Web API (player counts) has no
need of it: it was half the time top-online took.
deadline: a core.budget.Deadline. Once it has passed no further game is
started - the one in hand finishes - and the caller saves what was
done (core/budget.py). Never pass one where an unvisited game
would be lost, as in ingest_new.py, which empties its whole queue.
"""
total = len(games)
done = 0
for bs in range(0, total, BATCH_SIZE):
if deadline is not None and deadline.expired():
break
be = min(bs + BATCH_SIZE, total)
batch_num = bs // BATCH_SIZE + 1
total_batches = (total + BATCH_SIZE - 1) // BATCH_SIZE
print(f"\n── Batch {batch_num}/{total_batches} ({be - bs} games) ──")
for i in range(bs, be):
if deadline is not None and deadline.expired():
break
name = games[i].get("name") or games[i].get("link", "?")
print(f" [{i+1}/{total}] {desc}: {name[:50]}...", end=" ")
fn(games[i])
done += 1
print("✓")
if be < total:
pause = random.uniform(BATCH_PAUSE_MIN, BATCH_PAUSE_MAX)
print(f" ⏳ {pause:.0f}s...")
time.sleep(pause)
if pause and done == be and be < total:
wait = random.uniform(BATCH_PAUSE_MIN, BATCH_PAUSE_MAX)
print(f" ⏳ {wait:.0f}s...")
time.sleep(wait)
report(desc, done, total, deadline)
return done
def update_all_full(games, force=False):
def update_all_full(games, force=False, deadline: Optional[Deadline] = None):
client = get_client()
to_fetch = games if force else [g for g in games if not is_info_complete(g)]
print(f"Full update: {len(to_fetch)} to fetch, {len(games)-len(to_fetch)} skipped")
@@ -334,22 +366,12 @@ def update_all_full(games, force=False):
print("All up to date!")
return
guard = OverrideGuard.load()
process_batch(to_fetch, lambda g: fetch_full(g, client=client, guard=guard), "Fetching")
process_batch(daily_order(to_fetch, day_number()),
lambda g: fetch_full(g, client=client, guard=guard), "Fetching", deadline=deadline)
def update_reviews_only(games):
client = get_client()
print(f"Reviews update: {len(games)} games")
def fn(g):
appid = extract_appid(g.get("link", ""))
if not appid: return
s = client.fetch_reviews(appid)
if s: apply_reviews(g, s)
g["last_updated"] = now_iso()
process_batch(games, fn, "Reviews")
def update_players_only(games, type_filter: str = "online"):
def update_players_only(games, type_filter: str = "online",
deadline: Optional[Deadline] = None, day: Optional[int] = None):
"""Refresh `current_players` + `peak_today` for the games matching
`type_filter`. Default 'online' preserves the historical behaviour
used by top_online.py.
@@ -360,6 +382,10 @@ def update_players_only(games, type_filter: str = "online"):
(single-player, story, etc). Empty type_game is
excluded — those are unclassified, not offline.
- "all" → every game with a non-empty type_game.
The keyed Web API is asked without process_batch's pauses, in an order
that changes daily, so a run that meets its deadline leaves a different
tail unsampled each time.
"""
if not STEAM_API_KEY:
print("No STEAM_API_KEY.")
@@ -380,4 +406,5 @@ def update_players_only(games, type_filter: str = "online"):
if not appid: return
count = client.fetch_player_count(appid, STEAM_API_KEY)
if count is not None: apply_players(g, count)
process_batch(target, fn, "Players")
target = daily_order(target, day_number() if day is None else day)
process_batch(target, fn, "Players", pause=False, deadline=deadline)
+7 -5
View File
@@ -44,6 +44,12 @@ class HealthResult:
def reason(self): return REASON_LABELS.get(self.status, self.status)
def free_per_appdetails(data: dict) -> bool:
"""Whether appdetails calls a game free: is_free, or a price that starts at 0."""
price = data.get("price_overview") or {}
return bool(data.get("is_free", False)) or (bool(price) and price.get("initial", 1) == 0)
def check_game_health(link: str, name: str = "",
client: Optional[SteamClient] = None) -> HealthResult:
"""
@@ -70,11 +76,7 @@ def check_game_health(link: str, name: str = "",
# Reject unreleased games – an F2P tracker only lists playable titles.
if data.get("release_date", {}).get("coming_soon", False):
return HealthResult(COMING_SOON, appid, resolved, link)
is_free = data.get("is_free", False)
price = data.get("price_overview", {})
if price:
is_free = is_free or price.get("initial", 1) == 0
if not is_free:
if not free_per_appdetails(data):
return HealthResult(NOT_FREE, appid, resolved, link)
# Healthy – return data so caller can reuse it
return HealthResult(OK, appid, resolved, link, data=data)
+26
View File
@@ -5,7 +5,13 @@ day of a cycle, so a daily run touches 1/cycle of the catalogue.
A game's slice is crc32(appid) % cycle, NOT appid % cycle: Steam appids are
overwhelmingly multiples of 10, so `% 30` would leave 27 of the 30 monthly
slices empty and pile the whole catalogue into the other three.
The cycle grows with the catalogue (cycle_for), so a daily run stays the same
size however many games there are; and a run visits its games in an order
that changes daily (daily_order), so one that stops at its time budget
(core/budget.py) leaves a different tail each day.
"""
import math
import zlib
from datetime import date, datetime, timezone
from typing import Optional
@@ -13,6 +19,26 @@ from typing import Optional
from .data_store import extract_appid
def cycle_for(count: int, base: int, daily: int) -> int:
"""The length of a rotation over `count` games: `base` days while a daily
slice stays within `daily` games, then the smallest MULTIPLE of `base`
that keeps it there.
A multiple, because crc32 % (k * base) leaves crc32 % base unchanged: a
game stays on the same day of the base cycle, so when the cycle grows its
visits only thin out - never two within `base` days. SteamCharts keeps its
one page per game a month, and the store page its weekday.
"""
if count <= 0 or daily <= 0:
return base
return base * max(1, math.ceil(count / (base * daily)))
def daily_order(games: list[dict], day: int) -> list[dict]:
"""The games in an order that is fixed within a day and changes the next."""
return sorted(games, key=lambda g: zlib.crc32(f"{day}:{extract_appid(g.get('link', ''))}".encode()))
def slice_of(appid: str, cycle: int) -> int:
"""The day of the cycle on which this game is refreshed. Stable forever:
crc32 is fixed, so a game keeps its weekday and its day of the month."""
+17 -6
View File
@@ -259,12 +259,16 @@ class SteamClient:
out[str(item["id"])] = item
return out
def fetch_anticheat_items(self, appids, give_up_after: int = 10) -> tuple[dict[str, dict], set[str]]:
"""GetItems asking only for extra_details - each game's anti-cheat
disclosure - for a whole catalogue: ({appid: item}, ids not answered).
def fetch_items(self, appids, data_request: dict,
give_up_after: int = 10) -> tuple[dict[str, dict], set[str]]:
"""GetItems for a whole catalogue, a batch at a time: ({appid: item},
ids not answered). This is how the daily sweeps read every game in a
hundred-odd requests instead of one request per game.
Unlike fetch_store_items() a failed batch costs only its own ids, which
the caller must treat as unknown, never as "no disclosure". After
the caller must treat as unknown - never as "no reviews", "no
disclosure" or "gone from the store". An id that IS answered for but is
missing from `items` is one Steam does not know (success 15). After
`give_up_after` failed batches in a row the rest is not asked at all.
"""
ids = [str(a) for a in appids if str(a).isdigit()]
@@ -276,7 +280,7 @@ class SteamClient:
if in_a_row >= give_up_after:
failed.update(chunk)
continue
items = self._store_items_batch(chunk, {"include_extra_details": True})
items = self._store_items_batch(chunk, data_request)
if items is None:
failed.update(chunk)
in_a_row += 1
@@ -286,6 +290,11 @@ class SteamClient:
out[str(item["id"])] = item
return out, failed
def fetch_anticheat_items(self, appids, give_up_after: int = 10) -> tuple[dict[str, dict], set[str]]:
"""fetch_items() asking only for extra_details - each game's anti-cheat
disclosure (core/anticheat.py)."""
return self.fetch_items(appids, {"include_extra_details": True}, give_up_after)
def fetch_tag_names(self) -> dict[int, str]:
"""{tagid: English name} from IStoreService/GetTagList, fetched once per
client. A failure is not cached, and returns {} (no tag is named)."""
@@ -343,7 +352,9 @@ class SteamClient:
return body if body.get("success") else None
def check_store_page(self, appid: str) -> int:
"""HEAD request only – lightweight dead link check."""
"""HEAD the store page: check_game_health()'s last resort when
appdetails answers without data. A delisted app redirects to the store
front and ends on 200, so this cannot tell one from a live game."""
self._throttle_store()
try:
resp = self._session.head(
+60
View File
@@ -16,6 +16,10 @@ Checked against stored HTML scrapes of 40 random games: 37 identical tag lists
(the other three had drifted since they were scraped), 40 identical language
lists, no unknown tag id. Tag ids are named by IStoreService/GetTagList.
The daily sweeps read the same items for the whole catalogue, a hundred-odd
requests where a request per game took hours: review_summary() for
update_reviews.py, screen_item() for purge_unhealthy.py.
Everything here is pure; steam_client.py makes the requests.
"""
from typing import Iterable, Optional
@@ -120,6 +124,62 @@ def parse_store_item(item: dict, tag_names: dict[int, str]) -> dict:
}
def review_summary(item: Optional[dict]) -> Optional[tuple[int, int, str]]:
"""(review_count, percent_positive, label) from an item asked for with
include_reviews - Steam's own summary, the one its store shows - or None
when the item says nothing usable: no item, no summary, a count without a
percentage. A count of 0 comes back as (0, 0, ""), and is NOT to be
trusted on its own: the summary has said 0 for a game with thousands of
reviews (845070, 2026-10-23), and lags behind a new game's first ones.
percent_positive is floored, as on the store, where round() would round
up (85 for Counter-Strike 2, which round() had made 86).
"""
summary = ((item or {}).get("reviews") or {}).get("summary_filtered")
if not isinstance(summary, dict):
return None
count = _int(summary.get("review_count"))
if count < 0:
return None
if count == 0:
return (0, 0, "")
pct = _int(summary.get("percent_positive"))
if not 0 <= pct <= 100:
return None
return (count, pct, str(summary.get("review_score_label") or "").strip())
# Why Steam's store says a game may no longer belong in the catalogue
# (screen_item), for purge_unhealthy.py to confirm with appdetails.
UNKNOWN = "unknown" # Steam does not know the app (success 15)
HIDDEN = "hidden" # visible: false
UNLISTED = "unlisted" # taken off the store
COMING_SOON = "coming_soon" # not released (any more, or yet)
NOT_FREE = "not_free" # no is_free flag
def screen_item(item: Optional[dict]) -> Optional[str]:
"""Why an item asked for with include_release flags its game, or None
for a game that is on the store, released and free. `item` None means
Steam answered the batch without it: an app it does not know.
Only a flag, never a verdict: purge_unhealthy.py removes a game only
when appdetails agrees. `type` is not read - a free mod is in the
catalogue on purpose (Black Reliquary); that check belongs to ingest.
"""
if item is None:
return UNKNOWN
if item.get("visible") is False:
return HIDDEN
if item.get("unlisted"):
return UNLISTED
if (item.get("release") or {}).get("is_coming_soon"):
return COMING_SOON
if not item.get("is_free"):
return NOT_FREE
return None
def paid_dlc_from_items(items: Iterable[dict]) -> Optional[bool]:
"""Whether any of these DLC items costs money. None when there is no item to
judge by (every DLC hidden, or the request failed): unknown, not "free".
+172
View File
@@ -0,0 +1,172 @@
"""
A job's time budget and the rotations that keep a daily run the same size
however large the catalogue grows. These pin that a budget is opt-in, that a
loop past its deadline starts no further game, that only the store loops
pause, and that a growing cycle never brings a game's visits closer together.
Run from the repository root: python -m pytest scripts/tests
"""
import zlib
import pytest
from core import budget, fetcher, rotation
from core.budget import Deadline
class Clock:
def __init__(self, now=1000.0):
self.now = now
def __call__(self):
return self.now
@pytest.fixture
def clock(monkeypatch):
c = Clock()
monkeypatch.setattr(budget.time, "monotonic", c)
return c
def _games(n: int) -> list[dict]:
return [{"link": f"https://store.steampowered.com/app/{(i + 1) * 10}/", "name": f"G{i}"} for i in range(n)]
# ──────────── Deadline ────────────
@pytest.mark.parametrize("raw", [None, "", "0", "-5", "soon", "nan", "inf"])
def test_no_budget_means_no_deadline(monkeypatch, raw):
if raw is None:
monkeypatch.delenv(budget.ENV, raising=False)
else:
monkeypatch.setenv(budget.ENV, raw)
assert Deadline.from_env() is None
assert budget.expired(None) is False
def test_a_budget_counts_from_when_it_was_made(monkeypatch, clock):
monkeypatch.setenv(budget.ENV, " 30 ")
deadline = Deadline.from_env()
assert deadline.minutes == 30
clock.now += 29 * 60
assert not budget.expired(deadline)
clock.now += 60
assert budget.expired(deadline) and deadline.used() == pytest.approx(1.0)
def test_the_report_warns_when_a_loop_ran_out_or_came_close(clock):
lines: list[str] = []
deadline = Deadline(10)
clock.now += 60
budget.report("Players", 5, 5, deadline, log=lines.append)
clock.now += 7 * 60
budget.report("Players", 5, 5, deadline, log=lines.append)
budget.report("Players", 3, 5, deadline, log=lines.append)
budget.report("Players", 5, 5, None, log=lines.append)
assert not lines[0].startswith("::warning::") and "1 of 10" in lines[0]
assert lines[1].startswith("::warning::") and "outgrowing" in lines[1]
assert lines[2].startswith("::warning::") and "stopped after 3 of 5" in lines[2]
assert len(lines) == 3
# ──────────── process_batch ────────────
def test_past_the_deadline_no_further_game_is_started(monkeypatch, clock):
monkeypatch.setattr(fetcher.time, "sleep", lambda s: None)
deadline = Deadline(1)
seen: list[str] = []
def fn(game):
seen.append(game["name"])
clock.now += 25 # the third game crosses the minute
assert fetcher.process_batch(_games(10), fn, "Test", deadline=deadline) == 3
assert seen == ["G0", "G1", "G2"]
def test_only_the_store_loops_pause_between_batches(monkeypatch):
monkeypatch.setattr(fetcher, "BATCH_SIZE", 2)
pauses: list[float] = []
monkeypatch.setattr(fetcher.time, "sleep", pauses.append)
assert fetcher.process_batch(_games(5), lambda g: None, "Store") == 5
assert len(pauses) == 2 # between the three batches, not after the last
pauses.clear()
assert fetcher.process_batch(_games(5), lambda g: None, "Players", pause=False) == 5
assert pauses == []
def test_players_are_asked_without_pauses_in_a_daily_order(monkeypatch):
monkeypatch.setattr(fetcher, "STEAM_API_KEY", "key")
monkeypatch.setattr(fetcher.time, "sleep", lambda s: pytest.fail("the Web API loop slept"))
asked: list[str] = []
class Client:
def fetch_player_count(self, appid, key):
asked.append(appid)
return 7
monkeypatch.setattr(fetcher, "get_client", lambda: Client())
games = [dict(g, type_game="online") for g in _games(120)]
fetcher.update_players_only(games, day=100)
first = list(asked)
asked.clear()
fetcher.update_players_only(games, day=101)
assert sorted(first) == sorted(asked) and first != asked
assert all(g["current_players"] == "7" for g in games)
def test_the_full_update_stops_at_its_deadline(monkeypatch, clock):
monkeypatch.setattr(fetcher.time, "sleep", lambda s: None)
monkeypatch.setattr(fetcher, "get_client", lambda: object())
monkeypatch.setattr(fetcher.OverrideGuard, "load", classmethod(lambda cls: None))
fetched: list[str] = []
def fake_fetch(game, client=None, guard=None):
fetched.append(game["name"])
clock.now += 61
monkeypatch.setattr(fetcher, "fetch_full", fake_fetch)
fetcher.update_all_full(_games(4), deadline=Deadline(2))
assert len(fetched) == 2
# ──────────── Rotation ────────────
@pytest.mark.parametrize("base, daily", [(7, 2000), (30, 500), (30, 400)])
def test_a_cycle_is_a_multiple_of_its_base_that_never_shrinks(base, daily):
last = base
for count in range(0, 100_001, 997):
cycle = rotation.cycle_for(count, base, daily)
assert cycle % base == 0 and cycle >= last
assert count / cycle <= daily or cycle == base and count <= base * daily
last = cycle
def test_the_cycle_stays_at_its_base_until_a_day_would_overflow():
assert rotation.cycle_for(9847, 7, 2000) == 7
assert rotation.cycle_for(14000, 7, 2000) == 7
assert rotation.cycle_for(14001, 7, 2000) == 14
assert rotation.cycle_for(12000, 30, 400) == 30
assert rotation.cycle_for(12001, 30, 400) == 60
assert rotation.cycle_for(5, 7, 0) == 7
def test_a_growing_cycle_keeps_every_visit_at_least_a_base_apart():
base = 30
for appid in ("730", "570", "2507950", "4219930"):
crc = zlib.crc32(appid.encode())
visits = []
for day in range(20000, 20400):
cycle = 30 if day < 20150 else 60 if day < 20300 else 90
if rotation.due({"link": f"https://store.steampowered.com/app/{appid}/"}, cycle, day):
visits.append(day)
assert all(day % base == crc % base for day in visits)
assert all(b - a >= base for a, b in zip(visits, visits[1:]))
def test_the_daily_order_holds_within_a_day_and_changes_the_next():
games = _games(50)
assert rotation.daily_order(games, 7) == rotation.daily_order(list(reversed(games)), 7)
assert rotation.daily_order(games, 7) != rotation.daily_order(games, 8)
assert sorted(g["name"] for g in rotation.daily_order(games, 8)) == sorted(g["name"] for g in games)
+58
View File
@@ -0,0 +1,58 @@
"""
check_game_health() is the confirmation every removal goes through: purge
asks it about the games Steam's store API flags, and ingest about every queued
link. These pin how appdetails' answers map to a verdict.
Steam is never called: the client is a fake. Run from the repository root:
python -m pytest scripts/tests
"""
import pytest
from core import health_checker as hc
LINK = "https://store.steampowered.com/app/730/"
class FakeClient:
def __init__(self, status="ok", data=None, head=200):
self.status, self.data, self.head = status, data, head
def fetch_app_details_full(self, appid):
return (self.status, self.data)
def check_store_page(self, appid):
return self.head
@pytest.mark.parametrize("data, expected", [
({"is_free": True}, True),
({"is_free": False, "price_overview": {"initial": 0}}, True),
({"is_free": False, "price_overview": {"initial": 1499}}, False),
({"is_free": False, "price_overview": None}, False),
({}, False),
])
def test_what_appdetails_calls_free(data, expected):
assert hc.free_per_appdetails(data) is expected
@pytest.mark.parametrize("client, status", [
(FakeClient(data={"name": "CS2", "is_free": True}), hc.OK),
(FakeClient(data={"name": "CS2", "is_free": False, "price_overview": {"initial": 999}}), hc.NOT_FREE),
(FakeClient(data={"name": "CS2", "is_free": True, "release_date": {"coming_soon": True}}), hc.COMING_SOON),
(FakeClient(status="unavailable"), hc.UNAVAILABLE),
(FakeClient(status="not_found"), hc.NOT_FOUND_404),
(FakeClient(status="network_error"), hc.NETWORK_ERROR),
(FakeClient(data={}, head=404), hc.NOT_FOUND_404),
(FakeClient(data={}, head=410), hc.NOT_FOUND_410),
(FakeClient(data={}, head=200), hc.OK),
(FakeClient(data={}, head=-1), hc.NETWORK_ERROR),
])
def test_appdetails_answers_map_to_a_verdict(client, status):
result = hc.check_game_health(LINK, "Counter-Strike 2", client)
assert result.status == status and result.appid == "730"
assert result.should_remove is (status in hc.REMOVABLE)
def test_a_link_without_an_appid_is_invalid_without_a_request():
result = hc.check_game_health("https://example.com/", "x", object())
assert result.status == hc.INVALID_FORMAT and result.should_remove
+76
View File
@@ -0,0 +1,76 @@
"""
The leaderboards sample every online (top_online.py) or offline
(top_offline.py) game's player count through the keyed Web API, and
update_data.py refills incomplete records. These run each script end to end
against a fake client: they write their page and save through save_main(),
and the Web API loop never pauses.
Run from the repository root: python -m pytest scripts/tests
"""
import os
import pytest
import top_offline
import top_online
import update_data
from core import budget, data_store, fetcher
def _rec(appid: str, type_game: str, players: str = "10") -> dict:
return {"link": f"https://store.steampowered.com/app/{appid}/", "name": f"Game {appid}",
"type_game": type_game, "status": "active", "current_players": players,
"peak_today": players, "reviews": "90% (Very Positive)", "genre": "Action",
"anti_cheat": "-", "notes": "Not reviewed yet", "last_updated": "2026-01-01T00:00:00Z"}
class Players:
def __init__(self, counts):
self.counts = counts
def fetch_player_count(self, appid, key):
return self.counts.get(appid)
@pytest.fixture
def repo(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
os.makedirs("data")
monkeypatch.delenv(budget.ENV, raising=False)
monkeypatch.setattr(fetcher, "STEAM_API_KEY", "key")
monkeypatch.setattr(fetcher.time, "sleep", lambda s: pytest.fail("the Web API loop slept"))
data_store.save_main([_rec("10", "online"), _rec("20", "online"), _rec("30", "offline")],
apply_human_overrides=False)
return monkeypatch
def _after() -> dict:
return {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
def test_top_online_samples_the_online_games_and_writes_its_page(repo):
repo.setattr(top_online, "STEAM_API_KEY", "key")
repo.setattr(fetcher, "get_client", lambda: Players({"10": 50000, "20": 3}))
top_online.main()
page = open(os.path.join("games", "top-online.md"), encoding="utf-8").read()
assert page.index("Game 10") < page.index("Game 20") and "Game 30" not in page
after = _after()
assert (after["10"]["current_players"], after["30"]["current_players"]) == ("50,000", "10")
def test_top_offline_samples_the_offline_games_and_writes_its_page(repo):
repo.setattr(top_offline, "STEAM_API_KEY", "key")
repo.setattr(fetcher, "get_client", lambda: Players({"30": 1200}))
top_offline.main()
page = open(os.path.join("games", "top-offline.md"), encoding="utf-8").read()
assert "Game 30" in page and "Game 10" not in page
assert _after()["30"]["current_players"] == "1,200"
def test_update_data_refills_only_incomplete_records(repo):
refilled: list[str] = []
repo.setattr(fetcher, "get_client", lambda: object())
repo.setattr(fetcher.OverrideGuard, "load", classmethod(lambda cls: None))
repo.setattr(fetcher, "fetch_full", lambda g, client=None, guard=None: refilled.append(g["link"]))
update_data.main()
assert len(refilled) == 3 # the fixture's records lack most store fields
+17
View File
@@ -132,6 +132,23 @@ def test_anticheat_items_survive_a_failed_batch(monkeypatch):
assert sent["data_request"] == {"include_extra_details": True}
def test_items_pass_the_data_request_through_and_tell_unknown_from_unanswered(monkeypatch):
monkeypatch.setattr(steam_client, "STORE_ITEMS_BATCH", 2)
answered = {"response": {"store_items": [
{"id": 730, "success": 1, "reviews": {}},
{"id": 1, "appid": 0, "success": 15}, # unknown to Steam
]}}
client, session = _client(_Resp(steam_client.STORE_ITEMS_URL, body=answered),
_Resp(steam_client.STORE_ITEMS_URL, status_code=500))
monkeypatch.setattr(steam_client, "MAX_RETRIES", 1)
items, failed = client.fetch_items(["730", "1", "570"], {"include_reviews": True})
assert list(items) == ["730"]
assert failed == {"570"} # "1" was answered for: Steam does not know it
sent = json.loads(session.requests[0]["params"]["input_json"])
assert sent["data_request"] == {"include_reviews": True}
assert sent["context"]["country_code"] == "US"
def test_anticheat_items_give_up_after_failures_in_a_row(monkeypatch):
monkeypatch.setattr(steam_client, "STORE_ITEMS_BATCH", 1)
bad = [_Resp(steam_client.STORE_ITEMS_URL, status_code=403) for _ in range(2)]
+218
View File
@@ -0,0 +1,218 @@
"""
update_reviews.py reads every game's review summary from GetItems in a
hundred-odd requests. These pin what makes that safe to run daily: Steam's
floored percentage is written in the stored shape, a count of 0 is never
trusted on its own, an unanswered game keeps its value, last_updated moves
only on a change or on the heartbeat that keeps the site's "stale" badge
quiet, and a second run writes the same bytes.
Steam is never called: the client is a fake. Run from the repository root:
python -m pytest scripts/tests
"""
import os
import zlib
from datetime import datetime, timedelta, timezone
import pytest
import update_reviews as ur
from core import budget, data_store
from core.fetcher import reviews_from_summary
STAMP = "2026-10-23T06:30:00Z"
OLD = "2026-01-01T00:00:00Z"
NOW = datetime(2026, 10, 23, 6, 30, tzinfo=timezone.utc)
def _recent(days: int = 1) -> str:
return (datetime.now(timezone.utc) - timedelta(days=days)).strftime("%Y-%m-%dT%H:%M:%SZ")
def _rec(appid: str, reviews: str, last_updated: str = OLD) -> dict:
return {"link": f"https://store.steampowered.com/app/{appid}/", "name": f"Game {appid}",
"reviews": reviews, "type_game": "online", "notes": "Not reviewed yet",
"last_updated": last_updated}
def _item(appid: str, count=None, pct=None, label=None) -> dict:
item = {"id": int(appid), "success": 1}
if count is not None:
summary = {"review_count": count, "review_score_label": label}
if pct is not None:
summary["percent_positive"] = pct
item["reviews"] = {"summary_filtered": summary}
return item
class FakeClient:
"""GetItems items by appid (absent: Steam does not know it), the appids
whose batch failed, and appreviews' query_summary by appid (None: failed)."""
def __init__(self, items=None, failed=(), appreviews=None):
self.items, self.failed = items or {}, set(failed)
self.appreviews = appreviews or {}
self.asked: list[str] = []
def fetch_items(self, appids, data_request, give_up_after=10):
assert data_request == {"include_reviews": True}
return ({a: self.items[a] for a in appids if a in self.items and a not in self.failed},
{a for a in appids if a in self.failed})
def fetch_reviews(self, appid):
self.asked.append(appid)
return self.appreviews.get(appid)
@pytest.fixture
def repo(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
os.makedirs("data")
monkeypatch.setattr(ur, "now_iso", lambda: STAMP)
monkeypatch.delenv(budget.ENV, raising=False)
monkeypatch.setattr("core.fetcher.time.sleep", lambda s: None)
def run(records=None, client=None, argv=()):
if records is not None:
data_store.save_main(records, apply_human_overrides=False)
monkeypatch.setattr(ur, "get_client", lambda: client)
ur.main(list(argv))
return {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
return run
def _bytes() -> dict:
return {n: open(os.path.join("data", n), "rb").read()
for n in sorted(os.listdir("data")) if n.endswith((".jsonl", ".json"))}
# ──────────── The stored shape ────────────
@pytest.mark.parametrize("pos, total, expected", [
(29, 100, "29% (Mixed)"), # floor(29/100*100) in floating point is 28
(2, 3, "66% (Mixed)"),
(999, 1000, "99% (Mixed)"),
(0, 0, "No reviews"),
])
def test_appreviews_is_floored_in_integers(pos, total, expected):
summary = {"total_positive": pos, "total_reviews": total, "review_score_desc": "Mixed"}
assert reviews_from_summary(summary) == expected
@pytest.mark.parametrize("stored, item, expected", [
("86% (Very Positive)", _item("1", 9943066, 85, "Very Positive"), (ur.SET, "85% (Very Positive)")),
("N/A", _item("1", 3, 100, "3 user reviews"), (ur.SET, "100% (3 user reviews)")),
("N/A", _item("1", 12, 50, ""), (ur.SET, "50%")),
("No reviews", _item("1", 0, 0, "No user reviews"), (ur.KEEP, None)),
("93% (Very Positive)", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
("N/A", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
("", _item("1", 0), (ur.ASK, None)),
("50% (Mixed)", None, (ur.UNKNOWN, None)),
("50% (Mixed)", _item("1"), (ur.UNKNOWN, None)), # no summary
("50% (Mixed)", _item("1", 40, None, "Mixed"), (ur.UNKNOWN, None)), # count without a share
("50% (Mixed)", _item("1", 40, 101, "Mixed"), (ur.UNKNOWN, None)),
("50% (Mixed)", {"reviews": {"summary_filtered": {"percent_positive": 50}}}, (ur.UNKNOWN, None)),
])
def test_what_an_item_says_to_do(stored, item, expected):
assert ur.decide(stored, item) == expected
def test_the_heartbeat_spreads_over_a_week_and_reads_a_bad_stamp_as_due():
game = _rec("730", "85% (Very Positive)")
days = ur.HEARTBEAT_DAYS + zlib.crc32(b"730") % ur.HEARTBEAT_JITTER_DAYS
assert 14 <= days <= 20
game["last_updated"] = (NOW - timedelta(days=days - 1)).strftime("%Y-%m-%dT%H:%M:%SZ")
assert not ur.heartbeat_due(game, NOW)
game["last_updated"] = (NOW - timedelta(days=days)).strftime("%Y-%m-%dT%H:%M:%SZ")
assert ur.heartbeat_due(game, NOW)
game["last_updated"] = ""
assert ur.heartbeat_due(game, NOW)
# ──────────── A run ────────────
def test_a_changed_summary_is_written_and_stamped(repo):
client = FakeClient(items={"730": _item("730", 100, 85, "Very Positive")})
out = repo([_rec("730", "86% (Very Positive)", _recent())], client)
assert out["730"]["reviews"] == "85% (Very Positive)"
assert out["730"]["last_updated"] == STAMP
assert client.asked == []
def test_an_unchanged_game_keeps_its_stamp_until_its_heartbeat(repo):
recent, old = _recent(), _recent(days=40)
client = FakeClient(items={"730": _item("730", 100, 85, "Very Positive"),
"570": _item("570", 100, 80, "Very Positive")})
out = repo([_rec("730", "85% (Very Positive)", recent), _rec("570", "80% (Very Positive)", old)], client)
assert out["730"]["last_updated"] == recent
assert out["570"]["last_updated"] == STAMP # read, and due its heartbeat
def test_a_second_run_writes_the_same_bytes(repo):
client = FakeClient(items={"730": _item("730", 100, 85, "Very Positive"),
"570": _item("570", 0, 0, "No user reviews")},
appreviews={"570": {"total_reviews": 4, "total_positive": 3,
"review_score_desc": "4 user reviews"}})
repo([_rec("730", "86% (Very Positive)"), _rec("570", "N/A")], client)
first = _bytes()
out = repo(None, client)
assert _bytes() == first
assert out["570"]["reviews"] == "75% (4 user reviews)"
def test_a_zero_is_checked_on_appreviews_before_a_percentage_goes(repo):
zero = _item("845070", 0, 0, "No user reviews")
failing = FakeClient(items={"845070": zero}, appreviews={"845070": None})
out = repo([_rec("845070", "93% (Very Positive)")], failing)
assert out["845070"]["reviews"] == "93% (Very Positive)"
assert out["845070"]["last_updated"] == OLD # not read: no heartbeat either
assert failing.asked == ["845070"]
answered = FakeClient(items={"845070": zero}, appreviews={"845070": {
"total_reviews": 5843, "total_positive": 5318, "review_score_desc": "Very Positive"}})
assert repo(None, answered)["845070"]["reviews"] == "91% (Very Positive)"
gone = FakeClient(items={"845070": zero}, appreviews={"845070": {"total_reviews": 0}})
assert repo(None, gone)["845070"]["reviews"] == "No reviews"
def test_no_reviews_stays_without_asking(repo):
client = FakeClient(items={"4263530": _item("4263530", 0, 0, "No user reviews")})
out = repo([_rec("4263530", "No reviews", _recent())], client)
assert out["4263530"]["reviews"] == "No reviews" and client.asked == []
def test_an_unanswered_game_keeps_everything(repo):
client = FakeClient(items={"730": _item("730", 100, 85, "Very Positive"),
"570": _item("570", 100, 70, "Mostly Positive")}, failed={"570"})
out = repo([_rec("730", "86% (Very Positive)"), _rec("570", "80% (Very Positive)")], client)
assert out["570"]["reviews"] == "80% (Very Positive)" and out["570"]["last_updated"] == OLD
assert out["730"]["reviews"] == "85% (Very Positive)"
def test_no_answer_at_all_writes_nothing(repo):
repo([_rec("730", "86% (Very Positive)"), _rec("570", "80% (Very Positive)")], FakeClient())
before = _bytes()
repo(None, FakeClient(failed={"730", "570"}))
assert _bytes() == before
def test_appreviews_is_capped_and_stops_at_the_deadline(repo, monkeypatch):
zero = {a: _item(a, 0, 0, "No user reviews") for a in ("10", "20", "30")}
summary = {"total_reviews": 1, "total_positive": 1, "review_score_desc": "1 user reviews"}
client = FakeClient(items=zero, appreviews={a: summary for a in zero})
monkeypatch.setattr(ur, "ASK_CAP", 2)
out = repo([_rec(a, "N/A") for a in zero], client)
assert len(client.asked) == 2
assert sum(g["reviews"] == "100% (1 user reviews)" for g in out.values()) == 2
client = FakeClient(items=zero, appreviews={a: summary for a in zero})
monkeypatch.setenv(budget.ENV, "0.000001")
repo(None, client)
assert client.asked == []
def test_a_dry_run_writes_nothing(repo):
repo([_rec("730", "86% (Very Positive)")], FakeClient())
before = _bytes()
repo(None, FakeClient(items={"730": _item("730", 100, 85, "Very Positive")}), argv=["--dry-run"])
assert _bytes() == before
+3 -1
View File
@@ -12,6 +12,7 @@ Output: games/top-offline.md (10 columns, same shape as top-online.md).
import sys, os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from datetime import datetime, timezone
from core.budget import Deadline
from core.constants import STEAM_API_KEY, GAMES_DIR, TOP_OFFLINE_LIMIT
from core.data_store import load_main, save_main, extract_appid
from core.fetcher import update_players_only
@@ -58,6 +59,7 @@ def main():
if not STEAM_API_KEY:
print("No Steam API key (set STEAM_API_KEY).")
sys.exit(1)
deadline = Deadline.from_env()
games = load_main()
offline = [g for g in games if _is_offline(g) and g.get("status", "active") == "active"]
if not offline:
@@ -68,7 +70,7 @@ def main():
for g in offline:
aid = extract_appid(g.get("link", ""))
if aid: prev[aid] = _ppc(g.get("current_players", "0"))
update_players_only(games, type_filter="offline")
update_players_only(games, type_filter="offline", deadline=deadline)
# Re-filter after update (status may have changed).
offline = [g for g in games if _is_offline(g) and g.get("status", "active") == "active"]
offline.sort(key=lambda g: _ppc(g.get("current_players", "0")), reverse=True)
+3 -1
View File
@@ -3,6 +3,7 @@
import sys, os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from datetime import datetime, timezone
from core.budget import Deadline
from core.constants import STEAM_API_KEY, GAMES_DIR, TOP_ONLINE_LIMIT
from core.data_store import load_main, save_main, extract_appid
from core.fetcher import update_players_only
@@ -39,6 +40,7 @@ def _stars(r):
def main():
if not STEAM_API_KEY: print("No key."); sys.exit(1)
deadline = Deadline.from_env()
games = load_main()
online = [g for g in games if g.get("type_game","").lower()=="online"
and g.get("status","active")=="active"]
@@ -48,7 +50,7 @@ def main():
for g in online:
aid = extract_appid(g.get("link",""))
if aid: prev[aid] = _ppc(g.get("current_players","0"))
update_players_only(games)
update_players_only(games, deadline=deadline)
# Re-filter after update (status may have changed)
online = [g for g in games if g.get("type_game","").lower()=="online"
and g.get("status","active")=="active"]
+3 -1
View File
@@ -2,14 +2,16 @@
"""Full data update – enriches incomplete records."""
import sys, os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from core.budget import Deadline
from core.data_store import load_main, save_main, migrate_record
from core.fetcher import update_all_full
def main():
deadline = Deadline.from_env()
games = load_main()
print(f"Loaded {len(games)} games")
for g in games: migrate_record(g)
update_all_full(games)
update_all_full(games, deadline=deadline)
save_main(games)
print(f"\n✓ Saved {len(games)} games")
+175 -10
View File
@@ -1,15 +1,180 @@
#!/usr/bin/env python3
"""Update reviews only. Every 2 days."""
import sys, os
"""
Refresh every game's review summary from Steam's own, a hundred-odd requests a day.
This used to ask store.steampowered.com/appreviews about one game at a time:
two seconds a game, five and a half hours at 9,800 games, and on 2026-10-23 it
ran into GitHub's six-hour limit and was stopped with nothing saved.
IStoreBrowseService/GetItems answers for fifty games a request with the summary
the store shows (core/store_items.review_summary): the whole catalogue in about
two minutes.
The rules:
- A summary with reviews is written "85% (Very Positive)": Steam's floored
percentage and its label, the vocabulary appreviews has ("3 user
reviews", "Mixed").
- A count of 0 is never written on GetItems' word alone: its summary has
said 0 for a game with thousands of reviews. A game stored as "No
reviews" stays so; one stored with a percentage, "N/A" or nothing is asked
on appreviews instead (at most ASK_CAP a run, in an order that changes
daily), and that answer is written, "No reviews" included.
- No item, no usable summary, a failed batch or a failed appreviews request
leaves the stored value alone. When GetItems answers for no game at all,
nothing is written.
- last_updated moves when the value changes, and on a heartbeat: a game
whose summary was read and whose stamp is 14 to 20 days old (crc32
jitter, so the stamps spread out) is stamped again. The site calls a
record older than 30 days stale (web/src/lib/validation.ts, /health),
and this job used to stamp every game on every run; the heartbeat keeps
that meaning - read by the pipeline lately - without rewriting every
record every day.
- Everything is written by save_main().
Usage (from the repository root):
python scripts/update_reviews.py
python scripts/update_reviews.py --limit 200 --dry-run
"""
import argparse
import os
import sys
import zlib
from collections import Counter
from datetime import datetime, timedelta, timezone
from typing import Optional
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from core.data_store import load_main, save_main
from core.fetcher import update_reviews_only
def main():
from core.budget import Deadline
from core.data_store import extract_appid, load_main, migrate_record, now_iso, save_main
from core.fetcher import process_batch, reviews_from_summary, reviews_text
from core.normalize import normalize_records
from core.rotation import daily_order, day_number
from core.steam_client import get_client
from core.store_items import review_summary
ASK_CAP = 300
HEARTBEAT_DAYS = 14
HEARTBEAT_JITTER_DAYS = 7
NO_REVIEWS = "No reviews"
# What decide() says to do with one game.
SET = "set" # write this value
ASK = "ask" # ask appreviews: GetItems said 0
KEEP = "keep" # read, and nothing to write
UNKNOWN = "unknown" # not read: no item or no usable summary
def decide(stored: str, item: Optional[dict]) -> tuple[str, Optional[str]]:
"""What GetItems' item says to do with a game whose reviews are `stored`."""
found = review_summary(item)
if found is None:
return UNKNOWN, None
count, pct, label = found
if count > 0:
return SET, reviews_text(count, pct, label)
return (KEEP, None) if stored == NO_REVIEWS else (ASK, None)
def heartbeat_due(game: dict, now: datetime) -> bool:
"""Whether a game read today is due a new stamp although nothing changed:
its last_updated is 14 to 20 days old (by its appid), or unreadable."""
appid = extract_appid(game.get("link", "")) or ""
days = HEARTBEAT_DAYS + zlib.crc32(appid.encode()) % HEARTBEAT_JITTER_DAYS
try:
stamp = datetime.strptime(game.get("last_updated") or "", "%Y-%m-%dT%H:%M:%SZ")
except ValueError:
return True
return now - stamp.replace(tzinfo=timezone.utc) >= timedelta(days=days)
def record(game: dict, value: Optional[str], now: datetime, tally: Counter) -> None:
"""A game whose summary was read: write `value` (None: nothing new) and
stamp it as the rules say."""
if value is not None and game.get("reviews") != value:
game["reviews"] = value
game["last_updated"] = now_iso()
tally["changed"] += 1
elif heartbeat_due(game, now):
game["last_updated"] = now_iso()
tally["heartbeat"] += 1
def refresh(games: list[dict], client, deadline: Optional[Deadline] = None,
day: Optional[int] = None, now: Optional[datetime] = None) -> Counter:
"""Refresh the reviews of `games` in place. Returns a tally: changed,
heartbeat, unknown, unanswered (batches GetItems failed), asked,
ask_failed, ask_skipped (over the cap or past the deadline), and
no_answer when GetItems answered for nobody."""
now = now or datetime.now(timezone.utc)
by_id = {}
for game in games:
appid = extract_appid(game.get("link", ""))
if appid:
by_id[appid] = game
items, failed = client.fetch_items(list(by_id), {"include_reviews": True})
tally: Counter = Counter(unanswered=len(failed))
if by_id and len(failed) == len(by_id):
tally["no_answer"] = 1
return tally
asks: list[dict] = []
for appid, game in by_id.items():
if appid in failed:
continue
kind, value = decide(game.get("reviews", ""), items.get(appid))
if kind == UNKNOWN:
tally["unknown"] += 1
elif kind == ASK:
asks.append(game)
else:
record(game, value, now, tally)
def ask(game: dict) -> None:
summary = client.fetch_reviews(extract_appid(game["link"]))
if summary is None:
tally["ask_failed"] += 1
else:
record(game, reviews_from_summary(summary), now, tally)
asks = daily_order(asks, day_number() if day is None else day)
done = process_batch(asks[:ASK_CAP], ask, "Reviews from appreviews", deadline=deadline) if asks else 0
tally["asked"] = done
tally["ask_skipped"] = len(asks) - done
return tally
def main(argv: Optional[list[str]] = None) -> None:
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0].strip())
ap.add_argument("--limit", type=int, help="at most this many games (for trying it out)")
ap.add_argument("--dry-run", action="store_true", help="ask Steam and report, write nothing")
args = ap.parse_args(argv)
deadline = Deadline.from_env()
games = load_main()
print(f"{len(games)} games – updating reviews...")
update_reviews_only(games)
save_main(games)
print(f"✓ Done.")
for g in games:
migrate_record(g)
# Compared as stored, like the other refresh jobs.
normalize_records(games)
targets = games if args.limit is None else games[: args.limit]
print(f"{len(games)} games; reading the review summaries of {len(targets)} from Steam")
if __name__ == "__main__": main()
tally = refresh(targets, get_client(), deadline=deadline)
if tally["no_answer"]:
print("::warning::GetItems answered for no game: nothing written.")
return
print(f"\nReviews: {tally['changed']} changed, {tally['heartbeat']} re-stamped (heartbeat), "
f"{tally['unknown']} without a usable summary; appreviews asked about {tally['asked']} "
f"({tally['ask_failed']} failed, {tally['ask_skipped']} left for another day)")
if tally["unanswered"]:
print(f"::warning::GetItems did not answer for {tally['unanswered']} games; "
f"they keep their reviews until the next run.")
if args.dry_run:
print("Dry run: nothing written.")
return
save_main(games)
print(f"✓ Saved {len(games)} games")
if __name__ == "__main__":
main()