fix(reviews): stop depending on appreviews, which now redirects

The first run of the new Update Reviews on main asked appreviews about the
12 games GetItems counted at 0, and all 12 failed: since 2026-10-23
store.steampowered.com/appreviews/<id>?json=1 answers 302 to the store
front, from GitHub's runners as from anywhere else (it worked on 10-17).
The 38 games the ingest added that day were listed with "N/A".

- fetch_full() falls back to the GetItems summary when appreviews gives
  none (reviews_from_store_api()), so an ingest gets the value the daily
  refresh would write.
- update_reviews: a game never read ("N/A", nothing) with a count of 0
  becomes "No reviews" without a question; the first count GetItems has
  replaces it. A stored percentage is still never replaced by a 0 without a
  second source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 committed 2026-10-23 22:44:57 +07:00
1 parent 3c64923e1f
commit 11fb7a5b43
7 files changed
+101 -20

No files matched your search

+7 -1
View File
@@ -27,7 +27,13 @@ build. Repo public-facing version `4.3.3` → `4.4.0`.
days and five and a half hours. The percentage is now Steam's own, which
rounds down: about 4,000 games show one point less once (Counter-Strike 2
"85%" instead of "86%"), exactly as on their store pages. A summary that
says 0 reviews is double-checked game by game before it replaces anything.
says 0 reviews is double-checked game by game before it replaces a
percentage.
- **New games get their reviews again.** Steam's per-game reviews endpoint
has answered with a redirect to the store front since 2026-10-23, so the 38
games added that day were listed with "N/A". A new game now takes its
summary from the store API when that endpoint gives none, and a game never
read becomes "No reviews" until Steam counts its first one.
- **The daily purge needs two answers to remove a game.** Steam's store API
screens every game in a few minutes, and a game is removed only when
appdetails agrees; "unavailable" is asked twice, five minutes apart. A run
+10 -5
View File
@@ -312,11 +312,16 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
GitHub's six-hour limit with nothing saved. Rules: the percentage is Steam's,
FLOORED (`reviews_from_summary()` floors appreviews in integers too, so an
ingest and the refresh agree; until v4.4.0 it was rounded, one point higher
on ~40% of games); a GetItems count of 0 is never written - its summary has
said 0 for a game with thousands (845070) - so a stored percentage or "N/A"
is asked on appreviews (`ASK_CAP` 300 a run, daily order) and only that
answer is written; no item, no usable summary or a failed batch keeps the
value. A record's `last_updated` moves on a change and on a HEARTBEAT: a
on ~40% of games); a GetItems count of 0 never replaces a percentage - its
summary has said 0 for a game with thousands (845070) - so such a game is
asked on appreviews (`ASK_CAP` 300 a run, daily order) and keeps its value
when that gives no answer; "N/A" (never read) with a count of 0 becomes
"No reviews"; no item, no usable summary or a failed batch keeps the value.
**appreviews has answered with a 302 to the store front since 2026-10-23**
(from here and from GitHub's runners; it worked on 10-17), so `fetch_full()`
falls back to the same GetItems summary (`reviews_from_store_api()`) and an
ingest no longer leaves "N/A". A record's `last_updated` moves on a change
and on a HEARTBEAT: a
game read whose stamp is 14-20 days old (crc32 jitter) is stamped again.
The site calls a record older than 30 days stale (`web/src/lib/validation.ts`,
`/health`), and this job used to stamp every game on every run; the
+12
View File
@@ -145,6 +145,18 @@ Rules worth keeping (CLAUDE.md has them in full):
`queue: max`, `cancel-in-progress: false`, `ref: main` on every checkout, a
`timeout-minutes`, or a budget less than 20 minutes below it.
## Found after the merge: appreviews redirects
The first run of the new Update Reviews on `main` (2026-10-23, 2 m 31 s, 4,540
games changed - exactly the ones whose `last_updated` moved) asked `appreviews`
about the 12 games GetItems counted at 0, and all 12 failed. From this machine
and from GitHub's runners alike, `store.steampowered.com/appreviews/<id>?json=1`
now answers 302 to the store front, whatever the headers; it answered on
10-17. The 38 games the ingest added on 10-23 were the first to get "N/A".
So the ingest falls back to the GetItems summary, and a game never read
("N/A") with a count of 0 becomes "No reviews"; a stored percentage is still
kept until a second source confirms the 0.
## Load, before and after
| | Before | After |
+17 -2
View File
@@ -24,7 +24,7 @@ from .overrides import OverrideGuard
from .peaks import raise_peak
from .rotation import daily_order, day_number
from .scraper import scrape_store_page
from .store_items import paid_dlc_from_items, parse_store_item
from .store_items import paid_dlc_from_items, parse_store_item, review_summary
# Steam category descriptions that imply online multiplayer. Bias toward
@@ -165,6 +165,16 @@ def apply_reviews(game: dict, summary: dict) -> dict:
return game
def reviews_from_store_api(client, appid: str) -> Optional[str]:
"""A game's review summary from IStoreBrowseService/GetItems, as stored
("No reviews" for a count of 0), or None when it could not be read - the
ingest's answer when appreviews gives none. The daily refresh
(update_reviews.py) reads the same summary, so the two agree."""
items, failed = client.fetch_items([appid], {"include_reviews": True})
found = None if appid in failed else review_summary(items.get(appid))
return None if found is None else reviews_text(*found)
def apply_players(game: dict, count: int) -> dict:
game["current_players"] = f"{count:,}"
# A sample of 0 says nothing about a peak; anything else may beat it.
@@ -281,10 +291,15 @@ def fetch_full(game: dict, client=None, fetch_players=True,
if data:
apply_details(game, data)
# 2) Reviews
# 2) Reviews: appreviews, else Steam's store API - appreviews has answered
# with a redirect to the store front since 2026-10-23.
summary = c.fetch_reviews(appid)
if summary:
apply_reviews(game, summary)
else:
value = reviews_from_store_api(c, appid)
if value is not None:
game["reviews"] = value
# 3) Players
if fetch_players and STEAM_API_KEY:
+37 -2
View File
@@ -28,16 +28,28 @@ def _page(name="Anti-Cheat Expert (ACE)", heading="Uses Kernel Level Anti-Cheat"
class FakeClient:
def __init__(self, page=None, status=None, items=None):
"""`reviews` is appreviews' query_summary (None: it redirected), and
`review_items` GetItems' answer to include_reviews (None: the batch failed)."""
def __init__(self, page=None, status=None, items=None, reviews=None, review_items=None):
self.page = page
self.status = status or ("ok" if page is not None else "network_error")
self.items = items
self.reviews, self.review_items = reviews, review_items
self.asked_items = 0
def fetch_app_details(self, appid):
return dict(DETAILS)
def fetch_reviews(self, appid):
return None
return self.reviews
def fetch_items(self, appids, data_request, give_up_after=10):
assert data_request == {"include_reviews": True}
self.asked_items += 1
if self.review_items is None:
return {}, set(appids)
return {a: self.review_items[a] for a in appids if a in self.review_items}, set()
def fetch_store_page_full(self, appid):
return (self.status, self.page if self.status == "ok" else None)
@@ -83,6 +95,29 @@ def test_fetch_full_never_replaces_another_product():
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("EAC", "EAC", False)
def _reviews_item(count, pct=0, label=""):
return {"2507950": {"id": 2507950, "success": 1, "reviews": {"summary_filtered": {
"review_count": count, "percent_positive": pct, "review_score_label": label}}}}
def test_reviews_come_from_appreviews_when_it_answers():
client = FakeClient(page=_page(), reviews={"total_reviews": 3, "total_positive": 2,
"review_score_desc": "3 user reviews"})
assert _fetch(client=client)["reviews"] == "66% (3 user reviews)"
assert client.asked_items == 0
def test_reviews_come_from_the_store_api_when_appreviews_does_not_answer():
# Since 2026-10-23 appreviews answers with a redirect to the store front.
client = FakeClient(page=_page(), review_items=_reviews_item(52341, 80, "Very Positive"))
assert _fetch(client=client)["reviews"] == "80% (Very Positive)"
assert _fetch(client=FakeClient(page=_page(), review_items=_reviews_item(0)))["reviews"] == "No reviews"
def test_reviews_stay_unknown_when_neither_answers():
assert _fetch(client=FakeClient(page=_page()))["reviews"] == "N/A"
def test_apply_details_writes_no_anti_cheat():
game = make_skeleton(LINK)
fetcher.apply_details(game, VAC_DETAILS)
+5 -4
View File
@@ -104,8 +104,9 @@ def test_appreviews_is_floored_in_integers(pos, total, expected):
("N/A", _item("1", 12, 50, ""), (ur.SET, "50%")),
("No reviews", _item("1", 0, 0, "No user reviews"), (ur.KEEP, None)),
("93% (Very Positive)", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
("N/A", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
("", _item("1", 0), (ur.ASK, None)),
("100% (3 user reviews)", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
("N/A", _item("1", 0, 0, "No user reviews"), (ur.SET, "No reviews")), # never read: no question
("", _item("1", 0), (ur.SET, "No reviews")),
("50% (Mixed)", None, (ur.UNKNOWN, None)),
("50% (Mixed)", _item("1"), (ur.UNKNOWN, None)), # no summary
("50% (Mixed)", _item("1", 40, None, "Mixed"), (ur.UNKNOWN, None)), # count without a share
@@ -156,7 +157,7 @@ def test_a_second_run_writes_the_same_bytes(repo):
first = _bytes()
out = repo(None, client)
assert _bytes() == first
assert out["570"]["reviews"] == "75% (4 user reviews)"
assert out["570"]["reviews"] == "No reviews" and client.asked == []
def test_a_zero_is_checked_on_appreviews_before_a_percentage_goes(repo):
@@ -201,7 +202,7 @@ def test_appreviews_is_capped_and_stops_at_the_deadline(repo, monkeypatch):
summary = {"total_reviews": 1, "total_positive": 1, "review_score_desc": "1 user reviews"}
client = FakeClient(items=zero, appreviews={a: summary for a in zero})
monkeypatch.setattr(ur, "ASK_CAP", 2)
out = repo([_rec(a, "N/A") for a in zero], client)
out = repo([_rec(a, "50% (Mixed)") for a in zero], client)
assert len(client.asked) == 2
assert sum(g["reviews"] == "100% (1 user reviews)" for g in out.values()) == 2
+13 -6
View File
@@ -13,11 +13,13 @@ The rules:
- A summary with reviews is written "85% (Very Positive)": Steam's floored
percentage and its label, the vocabulary appreviews has ("3 user
reviews", "Mixed").
- A count of 0 is never written on GetItems' word alone: its summary has
said 0 for a game with thousands of reviews. A game stored as "No
reviews" stays so; one stored with a percentage, "N/A" or nothing is asked
on appreviews instead (at most ASK_CAP a run, in an order that changes
daily), and that answer is written, "No reviews" included.
- A count of 0 never replaces a percentage on GetItems' word alone: its
summary has said 0 for a game with thousands of reviews. Such a game is
asked on appreviews instead (at most ASK_CAP a run, in an order that
changes daily), and keeps its percentage when appreviews gives no answer
- which, since 2026-10-23, it does not: it redirects to the store front.
A game stored as "N/A" or nothing, never read, gets "No reviews": the
first count GetItems has replaces it.
- No item, no usable summary, a failed batch or a failed appreviews request
leaves the stored value alone. When GetItems answers for no game at all,
nothing is written.
@@ -36,6 +38,7 @@ Usage (from the repository root):
"""
import argparse
import os
import re
import sys
import zlib
from collections import Counter
@@ -56,6 +59,7 @@ ASK_CAP = 300
HEARTBEAT_DAYS = 14
HEARTBEAT_JITTER_DAYS = 7
NO_REVIEWS = "No reviews"
PERCENTAGE = re.compile(r"\d{1,3}%")
# What decide() says to do with one game.
SET = "set" # write this value
@@ -72,7 +76,10 @@ def decide(stored: str, item: Optional[dict]) -> tuple[str, Optional[str]]:
count, pct, label = found
if count > 0:
return SET, reviews_text(count, pct, label)
return (KEEP, None) if stored == NO_REVIEWS else (ASK, None)
if stored == NO_REVIEWS:
return KEEP, None
# A percentage is checked before a 0 replaces it; "N/A" was never known.
return (ASK, None) if PERCENTAGE.match(stored or "") else (SET, NO_REVIEWS)
def heartbeat_due(game: dict, now: datetime) -> bool: