feat(pipeline): read reviews for every game from Steam's store API
Update Reviews asked appreviews about one game at a time, two seconds a game. At 9,847 games it ran into GitHub's six-hour limit on 2026-10-23 and was stopped with nothing saved; five of its ten runs before that were cancelled while waiting. - SteamClient.fetch_items(): GetItems for a whole catalogue, a failed batch costing only its own ids (the anti-cheat sweep now uses it). - update_reviews.py reads review_summary() for every game, ~2 minutes, daily. Steam floors its percentage; reviews_from_summary() floors appreviews in integers too, so an ingest and the refresh agree. A count of 0 is never written on GetItems' word (it said 0 for a game with thousands): a stored percentage or "N/A" is asked on appreviews, at most 300 a run. last_updated moves on a change, and on a 14-20 day heartbeat that keeps the site's 30-day "stale" badge quiet without rewriting every record every run. - core/budget.py: JOB_BUDGET_MINUTES. Past it process_batch() starts no further game, so a job saves what it did instead of being killed; a ::warning:: when a loop used three quarters of its budget. - rotation.cycle_for() / daily_order(): cycles that grow with the catalogue as multiples of their base, and an order that changes daily. - The player-count loops skip the batch pauses meant for store pages (Top Online ~27 min instead of 48) and take a deadline, like update_all_full(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
01a631bb22
commit
49f0c56ed7
17 files changed
+1000
-55
No files matched your search
@@ -1,19 +1,40 @@
|
||||
name : Update Reviews
|
||||
|
||||
# Every game's review summary from IStoreBrowseService/GetItems, fifty games a
|
||||
# request: about two minutes for the whole catalogue (scripts/update_reviews.py
|
||||
# has the rules). It used to ask appreviews about one game at a time and, at
|
||||
# 9,800 games, ran into GitHub's six-hour limit with nothing saved.
|
||||
|
||||
on :
|
||||
schedule : [ { cron : '0 6 */2 * *' } ]
|
||||
schedule : [ { cron : '0 6 * * *' } ]
|
||||
workflow_dispatch :
|
||||
inputs :
|
||||
dry_run :
|
||||
description : 'Dry run: read Steam, write nothing'
|
||||
type : boolean
|
||||
default : false
|
||||
permissions : { contents : write }
|
||||
# queue: max - a run that finds another data job running waits its turn
|
||||
# (first in, first out) instead of evicting the run already waiting.
|
||||
concurrency :
|
||||
group : data-write
|
||||
cancel-in-progress : false
|
||||
queue : max
|
||||
jobs :
|
||||
reviews :
|
||||
runs-on : ubuntu-latest
|
||||
timeout-minutes : 60
|
||||
steps :
|
||||
-
|
||||
uses : actions/checkout@v6
|
||||
# main as it is when the job STARTS: a run that waited its turn must
|
||||
# work on what the job before it committed.
|
||||
with : { ref : main }
|
||||
-
|
||||
uses : actions/setup-python@v6
|
||||
with : { python-version : '3.15' }
|
||||
-
|
||||
run : chmod +x ./bash/reviews.sh && ./bash/reviews.sh
|
||||
env :
|
||||
JOB_BUDGET_MINUTES : '40'
|
||||
REVIEWS_DRY_RUN : ${{ inputs.dry_run }}
|
||||
@@ -1,5 +1,10 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
pip install --quiet --require-hashes --only-binary :all: -r requirements.txt
|
||||
# REVIEWS_DRY_RUN is the manual run's "dry run" box: read Steam, write nothing.
|
||||
if [[ "${REVIEWS_DRY_RUN:-}" == "true" ]]; then
|
||||
python scripts/update_reviews.py --dry-run
|
||||
exit 0
|
||||
fi
|
||||
python scripts/update_reviews.py
|
||||
bash "$(dirname "$0")/commit_push.sh" "Update reviews [$(date +'%Y-%m-%d')]"
|
||||
@@ -0,0 +1,81 @@
|
||||
"""
|
||||
How long a data job may keep starting work, so that a catalogue which outgrows
|
||||
its schedule makes a run partial instead of making it fail.
|
||||
|
||||
A GitHub-hosted job is stopped at its timeout-minutes (six hours at most), and
|
||||
a stopped job commits nothing: Update Reviews ran for exactly six hours on
|
||||
2026-10-23 and threw away every review it had read. Each data workflow sets
|
||||
JOB_BUDGET_MINUTES at least twenty minutes below its timeout. A loop that
|
||||
reaches the budget starts no further game, the script saves what it did, and
|
||||
the job commits it. The games it did not reach keep their values until the
|
||||
next run, and since the loops visit their games in an order that changes daily
|
||||
(core/rotation.daily_order) the next cut falls elsewhere.
|
||||
|
||||
A loop that ran out, or used more than three quarters of the budget, says so
|
||||
with a ::warning:: annotation on the run: the sign that the catalogue is
|
||||
outgrowing the job comes before any run is lost.
|
||||
|
||||
JOB_BUDGET_MINUTES unset, empty, zero or not a number means no deadline, so a
|
||||
run on a laptop works as it always did.
|
||||
"""
|
||||
import math
|
||||
import os
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
ENV = "JOB_BUDGET_MINUTES"
|
||||
WARN_SHARE = 0.75
|
||||
|
||||
|
||||
class Deadline:
|
||||
"""A budget of minutes counted from when it was made: from the start of
|
||||
the script, so loading and saving the catalogue count too."""
|
||||
__slots__ = ("start", "seconds")
|
||||
|
||||
def __init__(self, minutes: float, now: Optional[float] = None):
|
||||
self.start = time.monotonic() if now is None else now
|
||||
self.seconds = minutes * 60
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> Optional["Deadline"]:
|
||||
try:
|
||||
minutes = float(os.environ.get(ENV, "").strip())
|
||||
except ValueError:
|
||||
return None
|
||||
if not math.isfinite(minutes) or minutes <= 0:
|
||||
return None
|
||||
return cls(minutes)
|
||||
|
||||
@property
|
||||
def minutes(self) -> float:
|
||||
return self.seconds / 60
|
||||
|
||||
def used(self) -> float:
|
||||
"""The share of the budget spent so far, 0.0 to 1.0 and beyond."""
|
||||
return (time.monotonic() - self.start) / self.seconds
|
||||
|
||||
def expired(self) -> bool:
|
||||
return self.used() >= 1.0
|
||||
|
||||
|
||||
def expired(deadline: Optional[Deadline]) -> bool:
|
||||
return deadline is not None and deadline.expired()
|
||||
|
||||
|
||||
def report(desc: str, done: int, total: int, deadline: Optional[Deadline], log=print) -> None:
|
||||
"""One line on how a loop used the budget. A run that stopped short, or
|
||||
came close, gets a ::warning:: - it is how a growing catalogue shows up in
|
||||
the run's annotations before it costs one."""
|
||||
if deadline is None:
|
||||
return
|
||||
spent = deadline.used() * deadline.minutes
|
||||
if done < total:
|
||||
log(f"::warning::{desc}: stopped after {done} of {total} when the job's "
|
||||
f"{deadline.minutes:.0f}-minute budget ran out; the other {total - done} keep "
|
||||
f"their values until the next run.")
|
||||
elif deadline.used() > WARN_SHARE:
|
||||
log(f"::warning::{desc}: {done} done in {spent:.0f} of the job's {deadline.minutes:.0f} "
|
||||
f"budgeted minutes ({deadline.used():.0%}). The catalogue is outgrowing this "
|
||||
f"schedule: give the job a longer cycle or a larger budget.")
|
||||
else:
|
||||
log(f"{desc}: {done} done in {spent:.0f} of {deadline.minutes:.0f} budgeted minutes.")
|
||||
+57
-30
@@ -12,6 +12,7 @@ import time
|
||||
from typing import Optional
|
||||
|
||||
from .anticheat import apply_anti_cheat, from_categories
|
||||
from .budget import Deadline, report
|
||||
from .constants import (
|
||||
STEAM_API_KEY, SKIP_GENRE_TAGS,
|
||||
BATCH_SIZE, BATCH_PAUSE_MIN, BATCH_PAUSE_MAX,
|
||||
@@ -21,6 +22,7 @@ from .data_store import extract_appid, now_iso, is_info_complete, is_empty
|
||||
from .normalize import decode_entities, normalize_release_date
|
||||
from .overrides import OverrideGuard
|
||||
from .peaks import raise_peak
|
||||
from .rotation import daily_order, day_number
|
||||
from .scraper import scrape_store_page
|
||||
from .store_items import paid_dlc_from_items, parse_store_item
|
||||
|
||||
@@ -139,15 +141,27 @@ def apply_details(game: dict, data: dict) -> dict:
|
||||
return game
|
||||
|
||||
|
||||
def reviews_text(total: int, pct: int, label: str) -> str:
|
||||
"""The stored shape of a review summary: "85% (Very Positive)"."""
|
||||
if total <= 0:
|
||||
return "No reviews"
|
||||
return f"{pct}% ({label})" if label else f"{pct}%"
|
||||
|
||||
|
||||
def reviews_from_summary(summary: dict) -> str:
|
||||
"""appreviews' query_summary as stored. The percentage is FLOORED, as
|
||||
Steam floors its own (GetItems answered 85 for Counter-Strike 2 where
|
||||
round() had given 86), so the daily refresh from GetItems
|
||||
(update_reviews.py) and a game's first fetch here agree. In integers:
|
||||
in floating point 29/100*100 is 28.999... and floors to 28."""
|
||||
total = int(summary.get("total_reviews") or 0)
|
||||
pos = int(summary.get("total_positive") or 0)
|
||||
pct = pos * 100 // total if total > 0 else 0
|
||||
return reviews_text(total, pct, summary.get("review_score_desc", ""))
|
||||
|
||||
|
||||
def apply_reviews(game: dict, summary: dict) -> dict:
|
||||
total = summary.get("total_reviews", 0)
|
||||
pos = summary.get("total_positive", 0)
|
||||
if total > 0:
|
||||
pct = round((pos / total) * 100)
|
||||
label = summary.get("review_score_desc", "")
|
||||
game["reviews"] = f"{pct}% ({label})" if label else f"{pct}%"
|
||||
else:
|
||||
game["reviews"] = "No reviews"
|
||||
game["reviews"] = reviews_from_summary(summary)
|
||||
return game
|
||||
|
||||
|
||||
@@ -307,26 +321,44 @@ def fetch_full(game: dict, client=None, fetch_players=True,
|
||||
|
||||
# ──────────── Batch processor ────────────
|
||||
|
||||
def process_batch(games: list[dict], fn, desc="Processing"):
|
||||
"""Run fn on each game with batch pauses."""
|
||||
def process_batch(games: list[dict], fn, desc="Processing", pause: bool = True,
|
||||
deadline: Optional[Deadline] = None) -> int:
|
||||
"""Run fn on each game, BATCH_SIZE at a time. Returns how many it ran on.
|
||||
|
||||
pause: the 15-30 s rest between batches that keeps the store-page and
|
||||
appdetails loops polite. The keyed Web API (player counts) has no
|
||||
need of it: it was half the time top-online took.
|
||||
deadline: a core.budget.Deadline. Once it has passed no further game is
|
||||
started - the one in hand finishes - and the caller saves what was
|
||||
done (core/budget.py). Never pass one where an unvisited game
|
||||
would be lost, as in ingest_new.py, which empties its whole queue.
|
||||
"""
|
||||
total = len(games)
|
||||
done = 0
|
||||
for bs in range(0, total, BATCH_SIZE):
|
||||
if deadline is not None and deadline.expired():
|
||||
break
|
||||
be = min(bs + BATCH_SIZE, total)
|
||||
batch_num = bs // BATCH_SIZE + 1
|
||||
total_batches = (total + BATCH_SIZE - 1) // BATCH_SIZE
|
||||
print(f"\n── Batch {batch_num}/{total_batches} ({be - bs} games) ──")
|
||||
for i in range(bs, be):
|
||||
if deadline is not None and deadline.expired():
|
||||
break
|
||||
name = games[i].get("name") or games[i].get("link", "?")
|
||||
print(f" [{i+1}/{total}] {desc}: {name[:50]}...", end=" ")
|
||||
fn(games[i])
|
||||
done += 1
|
||||
print("✓")
|
||||
if be < total:
|
||||
pause = random.uniform(BATCH_PAUSE_MIN, BATCH_PAUSE_MAX)
|
||||
print(f" ⏳ {pause:.0f}s...")
|
||||
time.sleep(pause)
|
||||
if pause and done == be and be < total:
|
||||
wait = random.uniform(BATCH_PAUSE_MIN, BATCH_PAUSE_MAX)
|
||||
print(f" ⏳ {wait:.0f}s...")
|
||||
time.sleep(wait)
|
||||
report(desc, done, total, deadline)
|
||||
return done
|
||||
|
||||
|
||||
def update_all_full(games, force=False):
|
||||
def update_all_full(games, force=False, deadline: Optional[Deadline] = None):
|
||||
client = get_client()
|
||||
to_fetch = games if force else [g for g in games if not is_info_complete(g)]
|
||||
print(f"Full update: {len(to_fetch)} to fetch, {len(games)-len(to_fetch)} skipped")
|
||||
@@ -334,22 +366,12 @@ def update_all_full(games, force=False):
|
||||
print("All up to date!")
|
||||
return
|
||||
guard = OverrideGuard.load()
|
||||
process_batch(to_fetch, lambda g: fetch_full(g, client=client, guard=guard), "Fetching")
|
||||
process_batch(daily_order(to_fetch, day_number()),
|
||||
lambda g: fetch_full(g, client=client, guard=guard), "Fetching", deadline=deadline)
|
||||
|
||||
|
||||
def update_reviews_only(games):
|
||||
client = get_client()
|
||||
print(f"Reviews update: {len(games)} games")
|
||||
def fn(g):
|
||||
appid = extract_appid(g.get("link", ""))
|
||||
if not appid: return
|
||||
s = client.fetch_reviews(appid)
|
||||
if s: apply_reviews(g, s)
|
||||
g["last_updated"] = now_iso()
|
||||
process_batch(games, fn, "Reviews")
|
||||
|
||||
|
||||
def update_players_only(games, type_filter: str = "online"):
|
||||
def update_players_only(games, type_filter: str = "online",
|
||||
deadline: Optional[Deadline] = None, day: Optional[int] = None):
|
||||
"""Refresh `current_players` + `peak_today` for the games matching
|
||||
`type_filter`. Default 'online' preserves the historical behaviour
|
||||
used by top_online.py.
|
||||
@@ -360,6 +382,10 @@ def update_players_only(games, type_filter: str = "online"):
|
||||
(single-player, story, etc). Empty type_game is
|
||||
excluded — those are unclassified, not offline.
|
||||
- "all" → every game with a non-empty type_game.
|
||||
|
||||
The keyed Web API is asked without process_batch's pauses, in an order
|
||||
that changes daily, so a run that meets its deadline leaves a different
|
||||
tail unsampled each time.
|
||||
"""
|
||||
if not STEAM_API_KEY:
|
||||
print("No STEAM_API_KEY.")
|
||||
@@ -380,4 +406,5 @@ def update_players_only(games, type_filter: str = "online"):
|
||||
if not appid: return
|
||||
count = client.fetch_player_count(appid, STEAM_API_KEY)
|
||||
if count is not None: apply_players(g, count)
|
||||
process_batch(target, fn, "Players")
|
||||
target = daily_order(target, day_number() if day is None else day)
|
||||
process_batch(target, fn, "Players", pause=False, deadline=deadline)
|
||||
@@ -44,6 +44,12 @@ class HealthResult:
|
||||
def reason(self): return REASON_LABELS.get(self.status, self.status)
|
||||
|
||||
|
||||
def free_per_appdetails(data: dict) -> bool:
|
||||
"""Whether appdetails calls a game free: is_free, or a price that starts at 0."""
|
||||
price = data.get("price_overview") or {}
|
||||
return bool(data.get("is_free", False)) or (bool(price) and price.get("initial", 1) == 0)
|
||||
|
||||
|
||||
def check_game_health(link: str, name: str = "",
|
||||
client: Optional[SteamClient] = None) -> HealthResult:
|
||||
"""
|
||||
@@ -70,11 +76,7 @@ def check_game_health(link: str, name: str = "",
|
||||
# Reject unreleased games – an F2P tracker only lists playable titles.
|
||||
if data.get("release_date", {}).get("coming_soon", False):
|
||||
return HealthResult(COMING_SOON, appid, resolved, link)
|
||||
is_free = data.get("is_free", False)
|
||||
price = data.get("price_overview", {})
|
||||
if price:
|
||||
is_free = is_free or price.get("initial", 1) == 0
|
||||
if not is_free:
|
||||
if not free_per_appdetails(data):
|
||||
return HealthResult(NOT_FREE, appid, resolved, link)
|
||||
# Healthy – return data so caller can reuse it
|
||||
return HealthResult(OK, appid, resolved, link, data=data)
|
||||
|
||||
@@ -5,7 +5,13 @@ day of a cycle, so a daily run touches 1/cycle of the catalogue.
|
||||
A game's slice is crc32(appid) % cycle, NOT appid % cycle: Steam appids are
|
||||
overwhelmingly multiples of 10, so `% 30` would leave 27 of the 30 monthly
|
||||
slices empty and pile the whole catalogue into the other three.
|
||||
|
||||
The cycle grows with the catalogue (cycle_for), so a daily run stays the same
|
||||
size however many games there are; and a run visits its games in an order
|
||||
that changes daily (daily_order), so one that stops at its time budget
|
||||
(core/budget.py) leaves a different tail each day.
|
||||
"""
|
||||
import math
|
||||
import zlib
|
||||
from datetime import date, datetime, timezone
|
||||
from typing import Optional
|
||||
@@ -13,6 +19,26 @@ from typing import Optional
|
||||
from .data_store import extract_appid
|
||||
|
||||
|
||||
def cycle_for(count: int, base: int, daily: int) -> int:
|
||||
"""The length of a rotation over `count` games: `base` days while a daily
|
||||
slice stays within `daily` games, then the smallest MULTIPLE of `base`
|
||||
that keeps it there.
|
||||
|
||||
A multiple, because crc32 % (k * base) leaves crc32 % base unchanged: a
|
||||
game stays on the same day of the base cycle, so when the cycle grows its
|
||||
visits only thin out - never two within `base` days. SteamCharts keeps its
|
||||
one page per game a month, and the store page its weekday.
|
||||
"""
|
||||
if count <= 0 or daily <= 0:
|
||||
return base
|
||||
return base * max(1, math.ceil(count / (base * daily)))
|
||||
|
||||
|
||||
def daily_order(games: list[dict], day: int) -> list[dict]:
|
||||
"""The games in an order that is fixed within a day and changes the next."""
|
||||
return sorted(games, key=lambda g: zlib.crc32(f"{day}:{extract_appid(g.get('link', ''))}".encode()))
|
||||
|
||||
|
||||
def slice_of(appid: str, cycle: int) -> int:
|
||||
"""The day of the cycle on which this game is refreshed. Stable forever:
|
||||
crc32 is fixed, so a game keeps its weekday and its day of the month."""
|
||||
|
||||
@@ -259,12 +259,16 @@ class SteamClient:
|
||||
out[str(item["id"])] = item
|
||||
return out
|
||||
|
||||
def fetch_anticheat_items(self, appids, give_up_after: int = 10) -> tuple[dict[str, dict], set[str]]:
|
||||
"""GetItems asking only for extra_details - each game's anti-cheat
|
||||
disclosure - for a whole catalogue: ({appid: item}, ids not answered).
|
||||
def fetch_items(self, appids, data_request: dict,
|
||||
give_up_after: int = 10) -> tuple[dict[str, dict], set[str]]:
|
||||
"""GetItems for a whole catalogue, a batch at a time: ({appid: item},
|
||||
ids not answered). This is how the daily sweeps read every game in a
|
||||
hundred-odd requests instead of one request per game.
|
||||
|
||||
Unlike fetch_store_items() a failed batch costs only its own ids, which
|
||||
the caller must treat as unknown, never as "no disclosure". After
|
||||
the caller must treat as unknown - never as "no reviews", "no
|
||||
disclosure" or "gone from the store". An id that IS answered for but is
|
||||
missing from `items` is one Steam does not know (success 15). After
|
||||
`give_up_after` failed batches in a row the rest is not asked at all.
|
||||
"""
|
||||
ids = [str(a) for a in appids if str(a).isdigit()]
|
||||
@@ -276,7 +280,7 @@ class SteamClient:
|
||||
if in_a_row >= give_up_after:
|
||||
failed.update(chunk)
|
||||
continue
|
||||
items = self._store_items_batch(chunk, {"include_extra_details": True})
|
||||
items = self._store_items_batch(chunk, data_request)
|
||||
if items is None:
|
||||
failed.update(chunk)
|
||||
in_a_row += 1
|
||||
@@ -286,6 +290,11 @@ class SteamClient:
|
||||
out[str(item["id"])] = item
|
||||
return out, failed
|
||||
|
||||
def fetch_anticheat_items(self, appids, give_up_after: int = 10) -> tuple[dict[str, dict], set[str]]:
|
||||
"""fetch_items() asking only for extra_details - each game's anti-cheat
|
||||
disclosure (core/anticheat.py)."""
|
||||
return self.fetch_items(appids, {"include_extra_details": True}, give_up_after)
|
||||
|
||||
def fetch_tag_names(self) -> dict[int, str]:
|
||||
"""{tagid: English name} from IStoreService/GetTagList, fetched once per
|
||||
client. A failure is not cached, and returns {} (no tag is named)."""
|
||||
@@ -343,7 +352,9 @@ class SteamClient:
|
||||
return body if body.get("success") else None
|
||||
|
||||
def check_store_page(self, appid: str) -> int:
|
||||
"""HEAD request only – lightweight dead link check."""
|
||||
"""HEAD the store page: check_game_health()'s last resort when
|
||||
appdetails answers without data. A delisted app redirects to the store
|
||||
front and ends on 200, so this cannot tell one from a live game."""
|
||||
self._throttle_store()
|
||||
try:
|
||||
resp = self._session.head(
|
||||
|
||||
@@ -16,6 +16,10 @@ Checked against stored HTML scrapes of 40 random games: 37 identical tag lists
|
||||
(the other three had drifted since they were scraped), 40 identical language
|
||||
lists, no unknown tag id. Tag ids are named by IStoreService/GetTagList.
|
||||
|
||||
The daily sweeps read the same items for the whole catalogue, a hundred-odd
|
||||
requests where a request per game took hours: review_summary() for
|
||||
update_reviews.py, screen_item() for purge_unhealthy.py.
|
||||
|
||||
Everything here is pure; steam_client.py makes the requests.
|
||||
"""
|
||||
from typing import Iterable, Optional
|
||||
@@ -120,6 +124,62 @@ def parse_store_item(item: dict, tag_names: dict[int, str]) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def review_summary(item: Optional[dict]) -> Optional[tuple[int, int, str]]:
|
||||
"""(review_count, percent_positive, label) from an item asked for with
|
||||
include_reviews - Steam's own summary, the one its store shows - or None
|
||||
when the item says nothing usable: no item, no summary, a count without a
|
||||
percentage. A count of 0 comes back as (0, 0, ""), and is NOT to be
|
||||
trusted on its own: the summary has said 0 for a game with thousands of
|
||||
reviews (845070, 2026-10-23), and lags behind a new game's first ones.
|
||||
|
||||
percent_positive is floored, as on the store, where round() would round
|
||||
up (85 for Counter-Strike 2, which round() had made 86).
|
||||
"""
|
||||
summary = ((item or {}).get("reviews") or {}).get("summary_filtered")
|
||||
if not isinstance(summary, dict):
|
||||
return None
|
||||
count = _int(summary.get("review_count"))
|
||||
if count < 0:
|
||||
return None
|
||||
if count == 0:
|
||||
return (0, 0, "")
|
||||
pct = _int(summary.get("percent_positive"))
|
||||
if not 0 <= pct <= 100:
|
||||
return None
|
||||
return (count, pct, str(summary.get("review_score_label") or "").strip())
|
||||
|
||||
|
||||
# Why Steam's store says a game may no longer belong in the catalogue
|
||||
# (screen_item), for purge_unhealthy.py to confirm with appdetails.
|
||||
UNKNOWN = "unknown" # Steam does not know the app (success 15)
|
||||
HIDDEN = "hidden" # visible: false
|
||||
UNLISTED = "unlisted" # taken off the store
|
||||
COMING_SOON = "coming_soon" # not released (any more, or yet)
|
||||
NOT_FREE = "not_free" # no is_free flag
|
||||
|
||||
|
||||
def screen_item(item: Optional[dict]) -> Optional[str]:
|
||||
"""Why an item asked for with include_release flags its game, or None
|
||||
for a game that is on the store, released and free. `item` None means
|
||||
Steam answered the batch without it: an app it does not know.
|
||||
|
||||
Only a flag, never a verdict: purge_unhealthy.py removes a game only
|
||||
when appdetails agrees. `type` is not read - a free mod is in the
|
||||
catalogue on purpose (Black Reliquary); that check belongs to ingest.
|
||||
"""
|
||||
if item is None:
|
||||
return UNKNOWN
|
||||
if item.get("visible") is False:
|
||||
return HIDDEN
|
||||
if item.get("unlisted"):
|
||||
return UNLISTED
|
||||
if (item.get("release") or {}).get("is_coming_soon"):
|
||||
return COMING_SOON
|
||||
if not item.get("is_free"):
|
||||
return NOT_FREE
|
||||
return None
|
||||
|
||||
|
||||
def paid_dlc_from_items(items: Iterable[dict]) -> Optional[bool]:
|
||||
"""Whether any of these DLC items costs money. None when there is no item to
|
||||
judge by (every DLC hidden, or the request failed): unknown, not "free".
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
"""
|
||||
A job's time budget and the rotations that keep a daily run the same size
|
||||
however large the catalogue grows. These pin that a budget is opt-in, that a
|
||||
loop past its deadline starts no further game, that only the store loops
|
||||
pause, and that a growing cycle never brings a game's visits closer together.
|
||||
|
||||
Run from the repository root: python -m pytest scripts/tests
|
||||
"""
|
||||
import zlib
|
||||
|
||||
import pytest
|
||||
|
||||
from core import budget, fetcher, rotation
|
||||
from core.budget import Deadline
|
||||
|
||||
|
||||
class Clock:
|
||||
def __init__(self, now=1000.0):
|
||||
self.now = now
|
||||
|
||||
def __call__(self):
|
||||
return self.now
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def clock(monkeypatch):
|
||||
c = Clock()
|
||||
monkeypatch.setattr(budget.time, "monotonic", c)
|
||||
return c
|
||||
|
||||
|
||||
def _games(n: int) -> list[dict]:
|
||||
return [{"link": f"https://store.steampowered.com/app/{(i + 1) * 10}/", "name": f"G{i}"} for i in range(n)]
|
||||
|
||||
|
||||
# ──────────── Deadline ────────────
|
||||
|
||||
@pytest.mark.parametrize("raw", [None, "", "0", "-5", "soon", "nan", "inf"])
|
||||
def test_no_budget_means_no_deadline(monkeypatch, raw):
|
||||
if raw is None:
|
||||
monkeypatch.delenv(budget.ENV, raising=False)
|
||||
else:
|
||||
monkeypatch.setenv(budget.ENV, raw)
|
||||
assert Deadline.from_env() is None
|
||||
assert budget.expired(None) is False
|
||||
|
||||
|
||||
def test_a_budget_counts_from_when_it_was_made(monkeypatch, clock):
|
||||
monkeypatch.setenv(budget.ENV, " 30 ")
|
||||
deadline = Deadline.from_env()
|
||||
assert deadline.minutes == 30
|
||||
clock.now += 29 * 60
|
||||
assert not budget.expired(deadline)
|
||||
clock.now += 60
|
||||
assert budget.expired(deadline) and deadline.used() == pytest.approx(1.0)
|
||||
|
||||
|
||||
def test_the_report_warns_when_a_loop_ran_out_or_came_close(clock):
|
||||
lines: list[str] = []
|
||||
deadline = Deadline(10)
|
||||
clock.now += 60
|
||||
budget.report("Players", 5, 5, deadline, log=lines.append)
|
||||
clock.now += 7 * 60
|
||||
budget.report("Players", 5, 5, deadline, log=lines.append)
|
||||
budget.report("Players", 3, 5, deadline, log=lines.append)
|
||||
budget.report("Players", 5, 5, None, log=lines.append)
|
||||
assert not lines[0].startswith("::warning::") and "1 of 10" in lines[0]
|
||||
assert lines[1].startswith("::warning::") and "outgrowing" in lines[1]
|
||||
assert lines[2].startswith("::warning::") and "stopped after 3 of 5" in lines[2]
|
||||
assert len(lines) == 3
|
||||
|
||||
|
||||
# ──────────── process_batch ────────────
|
||||
|
||||
def test_past_the_deadline_no_further_game_is_started(monkeypatch, clock):
|
||||
monkeypatch.setattr(fetcher.time, "sleep", lambda s: None)
|
||||
deadline = Deadline(1)
|
||||
seen: list[str] = []
|
||||
|
||||
def fn(game):
|
||||
seen.append(game["name"])
|
||||
clock.now += 25 # the third game crosses the minute
|
||||
|
||||
assert fetcher.process_batch(_games(10), fn, "Test", deadline=deadline) == 3
|
||||
assert seen == ["G0", "G1", "G2"]
|
||||
|
||||
|
||||
def test_only_the_store_loops_pause_between_batches(monkeypatch):
|
||||
monkeypatch.setattr(fetcher, "BATCH_SIZE", 2)
|
||||
pauses: list[float] = []
|
||||
monkeypatch.setattr(fetcher.time, "sleep", pauses.append)
|
||||
assert fetcher.process_batch(_games(5), lambda g: None, "Store") == 5
|
||||
assert len(pauses) == 2 # between the three batches, not after the last
|
||||
pauses.clear()
|
||||
assert fetcher.process_batch(_games(5), lambda g: None, "Players", pause=False) == 5
|
||||
assert pauses == []
|
||||
|
||||
|
||||
def test_players_are_asked_without_pauses_in_a_daily_order(monkeypatch):
|
||||
monkeypatch.setattr(fetcher, "STEAM_API_KEY", "key")
|
||||
monkeypatch.setattr(fetcher.time, "sleep", lambda s: pytest.fail("the Web API loop slept"))
|
||||
asked: list[str] = []
|
||||
|
||||
class Client:
|
||||
def fetch_player_count(self, appid, key):
|
||||
asked.append(appid)
|
||||
return 7
|
||||
|
||||
monkeypatch.setattr(fetcher, "get_client", lambda: Client())
|
||||
games = [dict(g, type_game="online") for g in _games(120)]
|
||||
fetcher.update_players_only(games, day=100)
|
||||
first = list(asked)
|
||||
asked.clear()
|
||||
fetcher.update_players_only(games, day=101)
|
||||
assert sorted(first) == sorted(asked) and first != asked
|
||||
assert all(g["current_players"] == "7" for g in games)
|
||||
|
||||
|
||||
def test_the_full_update_stops_at_its_deadline(monkeypatch, clock):
|
||||
monkeypatch.setattr(fetcher.time, "sleep", lambda s: None)
|
||||
monkeypatch.setattr(fetcher, "get_client", lambda: object())
|
||||
monkeypatch.setattr(fetcher.OverrideGuard, "load", classmethod(lambda cls: None))
|
||||
fetched: list[str] = []
|
||||
|
||||
def fake_fetch(game, client=None, guard=None):
|
||||
fetched.append(game["name"])
|
||||
clock.now += 61
|
||||
|
||||
monkeypatch.setattr(fetcher, "fetch_full", fake_fetch)
|
||||
fetcher.update_all_full(_games(4), deadline=Deadline(2))
|
||||
assert len(fetched) == 2
|
||||
|
||||
|
||||
# ──────────── Rotation ────────────
|
||||
|
||||
@pytest.mark.parametrize("base, daily", [(7, 2000), (30, 500), (30, 400)])
|
||||
def test_a_cycle_is_a_multiple_of_its_base_that_never_shrinks(base, daily):
|
||||
last = base
|
||||
for count in range(0, 100_001, 997):
|
||||
cycle = rotation.cycle_for(count, base, daily)
|
||||
assert cycle % base == 0 and cycle >= last
|
||||
assert count / cycle <= daily or cycle == base and count <= base * daily
|
||||
last = cycle
|
||||
|
||||
|
||||
def test_the_cycle_stays_at_its_base_until_a_day_would_overflow():
|
||||
assert rotation.cycle_for(9847, 7, 2000) == 7
|
||||
assert rotation.cycle_for(14000, 7, 2000) == 7
|
||||
assert rotation.cycle_for(14001, 7, 2000) == 14
|
||||
assert rotation.cycle_for(12000, 30, 400) == 30
|
||||
assert rotation.cycle_for(12001, 30, 400) == 60
|
||||
assert rotation.cycle_for(5, 7, 0) == 7
|
||||
|
||||
|
||||
def test_a_growing_cycle_keeps_every_visit_at_least_a_base_apart():
|
||||
base = 30
|
||||
for appid in ("730", "570", "2507950", "4219930"):
|
||||
crc = zlib.crc32(appid.encode())
|
||||
visits = []
|
||||
for day in range(20000, 20400):
|
||||
cycle = 30 if day < 20150 else 60 if day < 20300 else 90
|
||||
if rotation.due({"link": f"https://store.steampowered.com/app/{appid}/"}, cycle, day):
|
||||
visits.append(day)
|
||||
assert all(day % base == crc % base for day in visits)
|
||||
assert all(b - a >= base for a, b in zip(visits, visits[1:]))
|
||||
|
||||
|
||||
def test_the_daily_order_holds_within_a_day_and_changes_the_next():
|
||||
games = _games(50)
|
||||
assert rotation.daily_order(games, 7) == rotation.daily_order(list(reversed(games)), 7)
|
||||
assert rotation.daily_order(games, 7) != rotation.daily_order(games, 8)
|
||||
assert sorted(g["name"] for g in rotation.daily_order(games, 8)) == sorted(g["name"] for g in games)
|
||||
@@ -0,0 +1,58 @@
|
||||
"""
|
||||
check_game_health() is the confirmation every removal goes through: purge
|
||||
asks it about the games Steam's store API flags, and ingest about every queued
|
||||
link. These pin how appdetails' answers map to a verdict.
|
||||
|
||||
Steam is never called: the client is a fake. Run from the repository root:
|
||||
python -m pytest scripts/tests
|
||||
"""
|
||||
import pytest
|
||||
|
||||
from core import health_checker as hc
|
||||
|
||||
LINK = "https://store.steampowered.com/app/730/"
|
||||
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, status="ok", data=None, head=200):
|
||||
self.status, self.data, self.head = status, data, head
|
||||
|
||||
def fetch_app_details_full(self, appid):
|
||||
return (self.status, self.data)
|
||||
|
||||
def check_store_page(self, appid):
|
||||
return self.head
|
||||
|
||||
|
||||
@pytest.mark.parametrize("data, expected", [
|
||||
({"is_free": True}, True),
|
||||
({"is_free": False, "price_overview": {"initial": 0}}, True),
|
||||
({"is_free": False, "price_overview": {"initial": 1499}}, False),
|
||||
({"is_free": False, "price_overview": None}, False),
|
||||
({}, False),
|
||||
])
|
||||
def test_what_appdetails_calls_free(data, expected):
|
||||
assert hc.free_per_appdetails(data) is expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("client, status", [
|
||||
(FakeClient(data={"name": "CS2", "is_free": True}), hc.OK),
|
||||
(FakeClient(data={"name": "CS2", "is_free": False, "price_overview": {"initial": 999}}), hc.NOT_FREE),
|
||||
(FakeClient(data={"name": "CS2", "is_free": True, "release_date": {"coming_soon": True}}), hc.COMING_SOON),
|
||||
(FakeClient(status="unavailable"), hc.UNAVAILABLE),
|
||||
(FakeClient(status="not_found"), hc.NOT_FOUND_404),
|
||||
(FakeClient(status="network_error"), hc.NETWORK_ERROR),
|
||||
(FakeClient(data={}, head=404), hc.NOT_FOUND_404),
|
||||
(FakeClient(data={}, head=410), hc.NOT_FOUND_410),
|
||||
(FakeClient(data={}, head=200), hc.OK),
|
||||
(FakeClient(data={}, head=-1), hc.NETWORK_ERROR),
|
||||
])
|
||||
def test_appdetails_answers_map_to_a_verdict(client, status):
|
||||
result = hc.check_game_health(LINK, "Counter-Strike 2", client)
|
||||
assert result.status == status and result.appid == "730"
|
||||
assert result.should_remove is (status in hc.REMOVABLE)
|
||||
|
||||
|
||||
def test_a_link_without_an_appid_is_invalid_without_a_request():
|
||||
result = hc.check_game_health("https://example.com/", "x", object())
|
||||
assert result.status == hc.INVALID_FORMAT and result.should_remove
|
||||
@@ -0,0 +1,76 @@
|
||||
"""
|
||||
The leaderboards sample every online (top_online.py) or offline
|
||||
(top_offline.py) game's player count through the keyed Web API, and
|
||||
update_data.py refills incomplete records. These run each script end to end
|
||||
against a fake client: they write their page and save through save_main(),
|
||||
and the Web API loop never pauses.
|
||||
|
||||
Run from the repository root: python -m pytest scripts/tests
|
||||
"""
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
import top_offline
|
||||
import top_online
|
||||
import update_data
|
||||
from core import budget, data_store, fetcher
|
||||
|
||||
|
||||
def _rec(appid: str, type_game: str, players: str = "10") -> dict:
|
||||
return {"link": f"https://store.steampowered.com/app/{appid}/", "name": f"Game {appid}",
|
||||
"type_game": type_game, "status": "active", "current_players": players,
|
||||
"peak_today": players, "reviews": "90% (Very Positive)", "genre": "Action",
|
||||
"anti_cheat": "-", "notes": "Not reviewed yet", "last_updated": "2026-01-01T00:00:00Z"}
|
||||
|
||||
|
||||
class Players:
|
||||
def __init__(self, counts):
|
||||
self.counts = counts
|
||||
|
||||
def fetch_player_count(self, appid, key):
|
||||
return self.counts.get(appid)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def repo(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
os.makedirs("data")
|
||||
monkeypatch.delenv(budget.ENV, raising=False)
|
||||
monkeypatch.setattr(fetcher, "STEAM_API_KEY", "key")
|
||||
monkeypatch.setattr(fetcher.time, "sleep", lambda s: pytest.fail("the Web API loop slept"))
|
||||
data_store.save_main([_rec("10", "online"), _rec("20", "online"), _rec("30", "offline")],
|
||||
apply_human_overrides=False)
|
||||
return monkeypatch
|
||||
|
||||
|
||||
def _after() -> dict:
|
||||
return {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
|
||||
|
||||
|
||||
def test_top_online_samples_the_online_games_and_writes_its_page(repo):
|
||||
repo.setattr(top_online, "STEAM_API_KEY", "key")
|
||||
repo.setattr(fetcher, "get_client", lambda: Players({"10": 50000, "20": 3}))
|
||||
top_online.main()
|
||||
page = open(os.path.join("games", "top-online.md"), encoding="utf-8").read()
|
||||
assert page.index("Game 10") < page.index("Game 20") and "Game 30" not in page
|
||||
after = _after()
|
||||
assert (after["10"]["current_players"], after["30"]["current_players"]) == ("50,000", "10")
|
||||
|
||||
|
||||
def test_top_offline_samples_the_offline_games_and_writes_its_page(repo):
|
||||
repo.setattr(top_offline, "STEAM_API_KEY", "key")
|
||||
repo.setattr(fetcher, "get_client", lambda: Players({"30": 1200}))
|
||||
top_offline.main()
|
||||
page = open(os.path.join("games", "top-offline.md"), encoding="utf-8").read()
|
||||
assert "Game 30" in page and "Game 10" not in page
|
||||
assert _after()["30"]["current_players"] == "1,200"
|
||||
|
||||
|
||||
def test_update_data_refills_only_incomplete_records(repo):
|
||||
refilled: list[str] = []
|
||||
repo.setattr(fetcher, "get_client", lambda: object())
|
||||
repo.setattr(fetcher.OverrideGuard, "load", classmethod(lambda cls: None))
|
||||
repo.setattr(fetcher, "fetch_full", lambda g, client=None, guard=None: refilled.append(g["link"]))
|
||||
update_data.main()
|
||||
assert len(refilled) == 3 # the fixture's records lack most store fields
|
||||
@@ -132,6 +132,23 @@ def test_anticheat_items_survive_a_failed_batch(monkeypatch):
|
||||
assert sent["data_request"] == {"include_extra_details": True}
|
||||
|
||||
|
||||
def test_items_pass_the_data_request_through_and_tell_unknown_from_unanswered(monkeypatch):
|
||||
monkeypatch.setattr(steam_client, "STORE_ITEMS_BATCH", 2)
|
||||
answered = {"response": {"store_items": [
|
||||
{"id": 730, "success": 1, "reviews": {}},
|
||||
{"id": 1, "appid": 0, "success": 15}, # unknown to Steam
|
||||
]}}
|
||||
client, session = _client(_Resp(steam_client.STORE_ITEMS_URL, body=answered),
|
||||
_Resp(steam_client.STORE_ITEMS_URL, status_code=500))
|
||||
monkeypatch.setattr(steam_client, "MAX_RETRIES", 1)
|
||||
items, failed = client.fetch_items(["730", "1", "570"], {"include_reviews": True})
|
||||
assert list(items) == ["730"]
|
||||
assert failed == {"570"} # "1" was answered for: Steam does not know it
|
||||
sent = json.loads(session.requests[0]["params"]["input_json"])
|
||||
assert sent["data_request"] == {"include_reviews": True}
|
||||
assert sent["context"]["country_code"] == "US"
|
||||
|
||||
|
||||
def test_anticheat_items_give_up_after_failures_in_a_row(monkeypatch):
|
||||
monkeypatch.setattr(steam_client, "STORE_ITEMS_BATCH", 1)
|
||||
bad = [_Resp(steam_client.STORE_ITEMS_URL, status_code=403) for _ in range(2)]
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
"""
|
||||
update_reviews.py reads every game's review summary from GetItems in a
|
||||
hundred-odd requests. These pin what makes that safe to run daily: Steam's
|
||||
floored percentage is written in the stored shape, a count of 0 is never
|
||||
trusted on its own, an unanswered game keeps its value, last_updated moves
|
||||
only on a change or on the heartbeat that keeps the site's "stale" badge
|
||||
quiet, and a second run writes the same bytes.
|
||||
|
||||
Steam is never called: the client is a fake. Run from the repository root:
|
||||
python -m pytest scripts/tests
|
||||
"""
|
||||
import os
|
||||
import zlib
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
import update_reviews as ur
|
||||
from core import budget, data_store
|
||||
from core.fetcher import reviews_from_summary
|
||||
|
||||
STAMP = "2026-10-23T06:30:00Z"
|
||||
OLD = "2026-01-01T00:00:00Z"
|
||||
NOW = datetime(2026, 10, 23, 6, 30, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _recent(days: int = 1) -> str:
|
||||
return (datetime.now(timezone.utc) - timedelta(days=days)).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def _rec(appid: str, reviews: str, last_updated: str = OLD) -> dict:
|
||||
return {"link": f"https://store.steampowered.com/app/{appid}/", "name": f"Game {appid}",
|
||||
"reviews": reviews, "type_game": "online", "notes": "Not reviewed yet",
|
||||
"last_updated": last_updated}
|
||||
|
||||
|
||||
def _item(appid: str, count=None, pct=None, label=None) -> dict:
|
||||
item = {"id": int(appid), "success": 1}
|
||||
if count is not None:
|
||||
summary = {"review_count": count, "review_score_label": label}
|
||||
if pct is not None:
|
||||
summary["percent_positive"] = pct
|
||||
item["reviews"] = {"summary_filtered": summary}
|
||||
return item
|
||||
|
||||
|
||||
class FakeClient:
|
||||
"""GetItems items by appid (absent: Steam does not know it), the appids
|
||||
whose batch failed, and appreviews' query_summary by appid (None: failed)."""
|
||||
|
||||
def __init__(self, items=None, failed=(), appreviews=None):
|
||||
self.items, self.failed = items or {}, set(failed)
|
||||
self.appreviews = appreviews or {}
|
||||
self.asked: list[str] = []
|
||||
|
||||
def fetch_items(self, appids, data_request, give_up_after=10):
|
||||
assert data_request == {"include_reviews": True}
|
||||
return ({a: self.items[a] for a in appids if a in self.items and a not in self.failed},
|
||||
{a for a in appids if a in self.failed})
|
||||
|
||||
def fetch_reviews(self, appid):
|
||||
self.asked.append(appid)
|
||||
return self.appreviews.get(appid)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def repo(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
os.makedirs("data")
|
||||
monkeypatch.setattr(ur, "now_iso", lambda: STAMP)
|
||||
monkeypatch.delenv(budget.ENV, raising=False)
|
||||
monkeypatch.setattr("core.fetcher.time.sleep", lambda s: None)
|
||||
|
||||
def run(records=None, client=None, argv=()):
|
||||
if records is not None:
|
||||
data_store.save_main(records, apply_human_overrides=False)
|
||||
monkeypatch.setattr(ur, "get_client", lambda: client)
|
||||
ur.main(list(argv))
|
||||
return {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
|
||||
return run
|
||||
|
||||
|
||||
def _bytes() -> dict:
|
||||
return {n: open(os.path.join("data", n), "rb").read()
|
||||
for n in sorted(os.listdir("data")) if n.endswith((".jsonl", ".json"))}
|
||||
|
||||
|
||||
# ──────────── The stored shape ────────────
|
||||
|
||||
@pytest.mark.parametrize("pos, total, expected", [
|
||||
(29, 100, "29% (Mixed)"), # floor(29/100*100) in floating point is 28
|
||||
(2, 3, "66% (Mixed)"),
|
||||
(999, 1000, "99% (Mixed)"),
|
||||
(0, 0, "No reviews"),
|
||||
])
|
||||
def test_appreviews_is_floored_in_integers(pos, total, expected):
|
||||
summary = {"total_positive": pos, "total_reviews": total, "review_score_desc": "Mixed"}
|
||||
assert reviews_from_summary(summary) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stored, item, expected", [
|
||||
("86% (Very Positive)", _item("1", 9943066, 85, "Very Positive"), (ur.SET, "85% (Very Positive)")),
|
||||
("N/A", _item("1", 3, 100, "3 user reviews"), (ur.SET, "100% (3 user reviews)")),
|
||||
("N/A", _item("1", 12, 50, ""), (ur.SET, "50%")),
|
||||
("No reviews", _item("1", 0, 0, "No user reviews"), (ur.KEEP, None)),
|
||||
("93% (Very Positive)", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
|
||||
("N/A", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
|
||||
("", _item("1", 0), (ur.ASK, None)),
|
||||
("50% (Mixed)", None, (ur.UNKNOWN, None)),
|
||||
("50% (Mixed)", _item("1"), (ur.UNKNOWN, None)), # no summary
|
||||
("50% (Mixed)", _item("1", 40, None, "Mixed"), (ur.UNKNOWN, None)), # count without a share
|
||||
("50% (Mixed)", _item("1", 40, 101, "Mixed"), (ur.UNKNOWN, None)),
|
||||
("50% (Mixed)", {"reviews": {"summary_filtered": {"percent_positive": 50}}}, (ur.UNKNOWN, None)),
|
||||
])
|
||||
def test_what_an_item_says_to_do(stored, item, expected):
|
||||
assert ur.decide(stored, item) == expected
|
||||
|
||||
|
||||
def test_the_heartbeat_spreads_over_a_week_and_reads_a_bad_stamp_as_due():
|
||||
game = _rec("730", "85% (Very Positive)")
|
||||
days = ur.HEARTBEAT_DAYS + zlib.crc32(b"730") % ur.HEARTBEAT_JITTER_DAYS
|
||||
assert 14 <= days <= 20
|
||||
game["last_updated"] = (NOW - timedelta(days=days - 1)).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
assert not ur.heartbeat_due(game, NOW)
|
||||
game["last_updated"] = (NOW - timedelta(days=days)).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
assert ur.heartbeat_due(game, NOW)
|
||||
game["last_updated"] = ""
|
||||
assert ur.heartbeat_due(game, NOW)
|
||||
|
||||
|
||||
# ──────────── A run ────────────
|
||||
|
||||
def test_a_changed_summary_is_written_and_stamped(repo):
|
||||
client = FakeClient(items={"730": _item("730", 100, 85, "Very Positive")})
|
||||
out = repo([_rec("730", "86% (Very Positive)", _recent())], client)
|
||||
assert out["730"]["reviews"] == "85% (Very Positive)"
|
||||
assert out["730"]["last_updated"] == STAMP
|
||||
assert client.asked == []
|
||||
|
||||
|
||||
def test_an_unchanged_game_keeps_its_stamp_until_its_heartbeat(repo):
|
||||
recent, old = _recent(), _recent(days=40)
|
||||
client = FakeClient(items={"730": _item("730", 100, 85, "Very Positive"),
|
||||
"570": _item("570", 100, 80, "Very Positive")})
|
||||
out = repo([_rec("730", "85% (Very Positive)", recent), _rec("570", "80% (Very Positive)", old)], client)
|
||||
assert out["730"]["last_updated"] == recent
|
||||
assert out["570"]["last_updated"] == STAMP # read, and due its heartbeat
|
||||
|
||||
|
||||
def test_a_second_run_writes_the_same_bytes(repo):
|
||||
client = FakeClient(items={"730": _item("730", 100, 85, "Very Positive"),
|
||||
"570": _item("570", 0, 0, "No user reviews")},
|
||||
appreviews={"570": {"total_reviews": 4, "total_positive": 3,
|
||||
"review_score_desc": "4 user reviews"}})
|
||||
repo([_rec("730", "86% (Very Positive)"), _rec("570", "N/A")], client)
|
||||
first = _bytes()
|
||||
out = repo(None, client)
|
||||
assert _bytes() == first
|
||||
assert out["570"]["reviews"] == "75% (4 user reviews)"
|
||||
|
||||
|
||||
def test_a_zero_is_checked_on_appreviews_before_a_percentage_goes(repo):
|
||||
zero = _item("845070", 0, 0, "No user reviews")
|
||||
failing = FakeClient(items={"845070": zero}, appreviews={"845070": None})
|
||||
out = repo([_rec("845070", "93% (Very Positive)")], failing)
|
||||
assert out["845070"]["reviews"] == "93% (Very Positive)"
|
||||
assert out["845070"]["last_updated"] == OLD # not read: no heartbeat either
|
||||
assert failing.asked == ["845070"]
|
||||
|
||||
answered = FakeClient(items={"845070": zero}, appreviews={"845070": {
|
||||
"total_reviews": 5843, "total_positive": 5318, "review_score_desc": "Very Positive"}})
|
||||
assert repo(None, answered)["845070"]["reviews"] == "91% (Very Positive)"
|
||||
|
||||
gone = FakeClient(items={"845070": zero}, appreviews={"845070": {"total_reviews": 0}})
|
||||
assert repo(None, gone)["845070"]["reviews"] == "No reviews"
|
||||
|
||||
|
||||
def test_no_reviews_stays_without_asking(repo):
|
||||
client = FakeClient(items={"4263530": _item("4263530", 0, 0, "No user reviews")})
|
||||
out = repo([_rec("4263530", "No reviews", _recent())], client)
|
||||
assert out["4263530"]["reviews"] == "No reviews" and client.asked == []
|
||||
|
||||
|
||||
def test_an_unanswered_game_keeps_everything(repo):
|
||||
client = FakeClient(items={"730": _item("730", 100, 85, "Very Positive"),
|
||||
"570": _item("570", 100, 70, "Mostly Positive")}, failed={"570"})
|
||||
out = repo([_rec("730", "86% (Very Positive)"), _rec("570", "80% (Very Positive)")], client)
|
||||
assert out["570"]["reviews"] == "80% (Very Positive)" and out["570"]["last_updated"] == OLD
|
||||
assert out["730"]["reviews"] == "85% (Very Positive)"
|
||||
|
||||
|
||||
def test_no_answer_at_all_writes_nothing(repo):
|
||||
repo([_rec("730", "86% (Very Positive)"), _rec("570", "80% (Very Positive)")], FakeClient())
|
||||
before = _bytes()
|
||||
repo(None, FakeClient(failed={"730", "570"}))
|
||||
assert _bytes() == before
|
||||
|
||||
|
||||
def test_appreviews_is_capped_and_stops_at_the_deadline(repo, monkeypatch):
|
||||
zero = {a: _item(a, 0, 0, "No user reviews") for a in ("10", "20", "30")}
|
||||
summary = {"total_reviews": 1, "total_positive": 1, "review_score_desc": "1 user reviews"}
|
||||
client = FakeClient(items=zero, appreviews={a: summary for a in zero})
|
||||
monkeypatch.setattr(ur, "ASK_CAP", 2)
|
||||
out = repo([_rec(a, "N/A") for a in zero], client)
|
||||
assert len(client.asked) == 2
|
||||
assert sum(g["reviews"] == "100% (1 user reviews)" for g in out.values()) == 2
|
||||
|
||||
client = FakeClient(items=zero, appreviews={a: summary for a in zero})
|
||||
monkeypatch.setenv(budget.ENV, "0.000001")
|
||||
repo(None, client)
|
||||
assert client.asked == []
|
||||
|
||||
|
||||
def test_a_dry_run_writes_nothing(repo):
|
||||
repo([_rec("730", "86% (Very Positive)")], FakeClient())
|
||||
before = _bytes()
|
||||
repo(None, FakeClient(items={"730": _item("730", 100, 85, "Very Positive")}), argv=["--dry-run"])
|
||||
assert _bytes() == before
|
||||
@@ -12,6 +12,7 @@ Output: games/top-offline.md (10 columns, same shape as top-online.md).
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from datetime import datetime, timezone
|
||||
from core.budget import Deadline
|
||||
from core.constants import STEAM_API_KEY, GAMES_DIR, TOP_OFFLINE_LIMIT
|
||||
from core.data_store import load_main, save_main, extract_appid
|
||||
from core.fetcher import update_players_only
|
||||
@@ -58,6 +59,7 @@ def main():
|
||||
if not STEAM_API_KEY:
|
||||
print("No Steam API key (set STEAM_API_KEY).")
|
||||
sys.exit(1)
|
||||
deadline = Deadline.from_env()
|
||||
games = load_main()
|
||||
offline = [g for g in games if _is_offline(g) and g.get("status", "active") == "active"]
|
||||
if not offline:
|
||||
@@ -68,7 +70,7 @@ def main():
|
||||
for g in offline:
|
||||
aid = extract_appid(g.get("link", ""))
|
||||
if aid: prev[aid] = _ppc(g.get("current_players", "0"))
|
||||
update_players_only(games, type_filter="offline")
|
||||
update_players_only(games, type_filter="offline", deadline=deadline)
|
||||
# Re-filter after update (status may have changed).
|
||||
offline = [g for g in games if _is_offline(g) and g.get("status", "active") == "active"]
|
||||
offline.sort(key=lambda g: _ppc(g.get("current_players", "0")), reverse=True)
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from datetime import datetime, timezone
|
||||
from core.budget import Deadline
|
||||
from core.constants import STEAM_API_KEY, GAMES_DIR, TOP_ONLINE_LIMIT
|
||||
from core.data_store import load_main, save_main, extract_appid
|
||||
from core.fetcher import update_players_only
|
||||
@@ -39,6 +40,7 @@ def _stars(r):
|
||||
|
||||
def main():
|
||||
if not STEAM_API_KEY: print("No key."); sys.exit(1)
|
||||
deadline = Deadline.from_env()
|
||||
games = load_main()
|
||||
online = [g for g in games if g.get("type_game","").lower()=="online"
|
||||
and g.get("status","active")=="active"]
|
||||
@@ -48,7 +50,7 @@ def main():
|
||||
for g in online:
|
||||
aid = extract_appid(g.get("link",""))
|
||||
if aid: prev[aid] = _ppc(g.get("current_players","0"))
|
||||
update_players_only(games)
|
||||
update_players_only(games, deadline=deadline)
|
||||
# Re-filter after update (status may have changed)
|
||||
online = [g for g in games if g.get("type_game","").lower()=="online"
|
||||
and g.get("status","active")=="active"]
|
||||
|
||||
@@ -2,14 +2,16 @@
|
||||
"""Full data update – enriches incomplete records."""
|
||||
import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from core.budget import Deadline
|
||||
from core.data_store import load_main, save_main, migrate_record
|
||||
from core.fetcher import update_all_full
|
||||
|
||||
def main():
|
||||
deadline = Deadline.from_env()
|
||||
games = load_main()
|
||||
print(f"Loaded {len(games)} games")
|
||||
for g in games: migrate_record(g)
|
||||
update_all_full(games)
|
||||
update_all_full(games, deadline=deadline)
|
||||
save_main(games)
|
||||
print(f"\n✓ Saved {len(games)} games")
|
||||
|
||||
|
||||
+175
-10
@@ -1,15 +1,180 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Update reviews only. Every 2 days."""
|
||||
import sys, os
|
||||
"""
|
||||
Refresh every game's review summary from Steam's own, a hundred-odd requests a day.
|
||||
|
||||
This used to ask store.steampowered.com/appreviews about one game at a time:
|
||||
two seconds a game, five and a half hours at 9,800 games, and on 2026-10-23 it
|
||||
ran into GitHub's six-hour limit and was stopped with nothing saved.
|
||||
IStoreBrowseService/GetItems answers for fifty games a request with the summary
|
||||
the store shows (core/store_items.review_summary): the whole catalogue in about
|
||||
two minutes.
|
||||
|
||||
The rules:
|
||||
- A summary with reviews is written "85% (Very Positive)": Steam's floored
|
||||
percentage and its label, the vocabulary appreviews has ("3 user
|
||||
reviews", "Mixed").
|
||||
- A count of 0 is never written on GetItems' word alone: its summary has
|
||||
said 0 for a game with thousands of reviews. A game stored as "No
|
||||
reviews" stays so; one stored with a percentage, "N/A" or nothing is asked
|
||||
on appreviews instead (at most ASK_CAP a run, in an order that changes
|
||||
daily), and that answer is written, "No reviews" included.
|
||||
- No item, no usable summary, a failed batch or a failed appreviews request
|
||||
leaves the stored value alone. When GetItems answers for no game at all,
|
||||
nothing is written.
|
||||
- last_updated moves when the value changes, and on a heartbeat: a game
|
||||
whose summary was read and whose stamp is 14 to 20 days old (crc32
|
||||
jitter, so the stamps spread out) is stamped again. The site calls a
|
||||
record older than 30 days stale (web/src/lib/validation.ts, /health),
|
||||
and this job used to stamp every game on every run; the heartbeat keeps
|
||||
that meaning - read by the pipeline lately - without rewriting every
|
||||
record every day.
|
||||
- Everything is written by save_main().
|
||||
|
||||
Usage (from the repository root):
|
||||
python scripts/update_reviews.py
|
||||
python scripts/update_reviews.py --limit 200 --dry-run
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import zlib
|
||||
from collections import Counter
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from core.data_store import load_main, save_main
|
||||
from core.fetcher import update_reviews_only
|
||||
|
||||
def main():
|
||||
from core.budget import Deadline
|
||||
from core.data_store import extract_appid, load_main, migrate_record, now_iso, save_main
|
||||
from core.fetcher import process_batch, reviews_from_summary, reviews_text
|
||||
from core.normalize import normalize_records
|
||||
from core.rotation import daily_order, day_number
|
||||
from core.steam_client import get_client
|
||||
from core.store_items import review_summary
|
||||
|
||||
ASK_CAP = 300
|
||||
HEARTBEAT_DAYS = 14
|
||||
HEARTBEAT_JITTER_DAYS = 7
|
||||
NO_REVIEWS = "No reviews"
|
||||
|
||||
# What decide() says to do with one game.
|
||||
SET = "set" # write this value
|
||||
ASK = "ask" # ask appreviews: GetItems said 0
|
||||
KEEP = "keep" # read, and nothing to write
|
||||
UNKNOWN = "unknown" # not read: no item or no usable summary
|
||||
|
||||
|
||||
def decide(stored: str, item: Optional[dict]) -> tuple[str, Optional[str]]:
|
||||
"""What GetItems' item says to do with a game whose reviews are `stored`."""
|
||||
found = review_summary(item)
|
||||
if found is None:
|
||||
return UNKNOWN, None
|
||||
count, pct, label = found
|
||||
if count > 0:
|
||||
return SET, reviews_text(count, pct, label)
|
||||
return (KEEP, None) if stored == NO_REVIEWS else (ASK, None)
|
||||
|
||||
|
||||
def heartbeat_due(game: dict, now: datetime) -> bool:
|
||||
"""Whether a game read today is due a new stamp although nothing changed:
|
||||
its last_updated is 14 to 20 days old (by its appid), or unreadable."""
|
||||
appid = extract_appid(game.get("link", "")) or ""
|
||||
days = HEARTBEAT_DAYS + zlib.crc32(appid.encode()) % HEARTBEAT_JITTER_DAYS
|
||||
try:
|
||||
stamp = datetime.strptime(game.get("last_updated") or "", "%Y-%m-%dT%H:%M:%SZ")
|
||||
except ValueError:
|
||||
return True
|
||||
return now - stamp.replace(tzinfo=timezone.utc) >= timedelta(days=days)
|
||||
|
||||
|
||||
def record(game: dict, value: Optional[str], now: datetime, tally: Counter) -> None:
|
||||
"""A game whose summary was read: write `value` (None: nothing new) and
|
||||
stamp it as the rules say."""
|
||||
if value is not None and game.get("reviews") != value:
|
||||
game["reviews"] = value
|
||||
game["last_updated"] = now_iso()
|
||||
tally["changed"] += 1
|
||||
elif heartbeat_due(game, now):
|
||||
game["last_updated"] = now_iso()
|
||||
tally["heartbeat"] += 1
|
||||
|
||||
|
||||
def refresh(games: list[dict], client, deadline: Optional[Deadline] = None,
|
||||
day: Optional[int] = None, now: Optional[datetime] = None) -> Counter:
|
||||
"""Refresh the reviews of `games` in place. Returns a tally: changed,
|
||||
heartbeat, unknown, unanswered (batches GetItems failed), asked,
|
||||
ask_failed, ask_skipped (over the cap or past the deadline), and
|
||||
no_answer when GetItems answered for nobody."""
|
||||
now = now or datetime.now(timezone.utc)
|
||||
by_id = {}
|
||||
for game in games:
|
||||
appid = extract_appid(game.get("link", ""))
|
||||
if appid:
|
||||
by_id[appid] = game
|
||||
items, failed = client.fetch_items(list(by_id), {"include_reviews": True})
|
||||
tally: Counter = Counter(unanswered=len(failed))
|
||||
if by_id and len(failed) == len(by_id):
|
||||
tally["no_answer"] = 1
|
||||
return tally
|
||||
|
||||
asks: list[dict] = []
|
||||
for appid, game in by_id.items():
|
||||
if appid in failed:
|
||||
continue
|
||||
kind, value = decide(game.get("reviews", ""), items.get(appid))
|
||||
if kind == UNKNOWN:
|
||||
tally["unknown"] += 1
|
||||
elif kind == ASK:
|
||||
asks.append(game)
|
||||
else:
|
||||
record(game, value, now, tally)
|
||||
|
||||
def ask(game: dict) -> None:
|
||||
summary = client.fetch_reviews(extract_appid(game["link"]))
|
||||
if summary is None:
|
||||
tally["ask_failed"] += 1
|
||||
else:
|
||||
record(game, reviews_from_summary(summary), now, tally)
|
||||
|
||||
asks = daily_order(asks, day_number() if day is None else day)
|
||||
done = process_batch(asks[:ASK_CAP], ask, "Reviews from appreviews", deadline=deadline) if asks else 0
|
||||
tally["asked"] = done
|
||||
tally["ask_skipped"] = len(asks) - done
|
||||
return tally
|
||||
|
||||
|
||||
def main(argv: Optional[list[str]] = None) -> None:
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0].strip())
|
||||
ap.add_argument("--limit", type=int, help="at most this many games (for trying it out)")
|
||||
ap.add_argument("--dry-run", action="store_true", help="ask Steam and report, write nothing")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
deadline = Deadline.from_env()
|
||||
games = load_main()
|
||||
print(f"{len(games)} games – updating reviews...")
|
||||
update_reviews_only(games)
|
||||
save_main(games)
|
||||
print(f"✓ Done.")
|
||||
for g in games:
|
||||
migrate_record(g)
|
||||
# Compared as stored, like the other refresh jobs.
|
||||
normalize_records(games)
|
||||
targets = games if args.limit is None else games[: args.limit]
|
||||
print(f"{len(games)} games; reading the review summaries of {len(targets)} from Steam")
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
tally = refresh(targets, get_client(), deadline=deadline)
|
||||
if tally["no_answer"]:
|
||||
print("::warning::GetItems answered for no game: nothing written.")
|
||||
return
|
||||
print(f"\nReviews: {tally['changed']} changed, {tally['heartbeat']} re-stamped (heartbeat), "
|
||||
f"{tally['unknown']} without a usable summary; appreviews asked about {tally['asked']} "
|
||||
f"({tally['ask_failed']} failed, {tally['ask_skipped']} left for another day)")
|
||||
if tally["unanswered"]:
|
||||
print(f"::warning::GetItems did not answer for {tally['unanswered']} games; "
|
||||
f"they keep their reviews until the next run.")
|
||||
|
||||
if args.dry_run:
|
||||
print("Dry run: nothing written.")
|
||||
return
|
||||
save_main(games)
|
||||
print(f"✓ Saved {len(games)} games")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in new issue
Block a user