fix(reviews): stop depending on appreviews, which now redirects
The first run of the new Update Reviews on main asked appreviews about the
12 games GetItems counted at 0, and all 12 failed: since 2026-10-23
store.steampowered.com/appreviews/<id>?json=1 answers 302 to the store
front, from GitHub's runners as from anywhere else (it worked on 10-17).
The 38 games the ingest added that day were listed with "N/A".
- fetch_full() falls back to the GetItems summary when appreviews gives
none (reviews_from_store_api()), so an ingest gets the value the daily
refresh would write.
- update_reviews: a game never read ("N/A", nothing) with a count of 0
becomes "No reviews" without a question; the first count GetItems has
replaces it. A stored percentage is still never replaced by a 0 without a
second source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3c64923e1f
commit
11fb7a5b43
7 files changed
+101
-20
No files matched your search
+7
-1
@@ -27,7 +27,13 @@ build. Repo public-facing version `4.3.3` → `4.4.0`.
|
||||
days and five and a half hours. The percentage is now Steam's own, which
|
||||
rounds down: about 4,000 games show one point less once (Counter-Strike 2
|
||||
"85%" instead of "86%"), exactly as on their store pages. A summary that
|
||||
says 0 reviews is double-checked game by game before it replaces anything.
|
||||
says 0 reviews is double-checked game by game before it replaces a
|
||||
percentage.
|
||||
- **New games get their reviews again.** Steam's per-game reviews endpoint
|
||||
has answered with a redirect to the store front since 2026-10-23, so the 38
|
||||
games added that day were listed with "N/A". A new game now takes its
|
||||
summary from the store API when that endpoint gives none, and a game never
|
||||
read becomes "No reviews" until Steam counts its first one.
|
||||
- **The daily purge needs two answers to remove a game.** Steam's store API
|
||||
screens every game in a few minutes, and a game is removed only when
|
||||
appdetails agrees; "unavailable" is asked twice, five minutes apart. A run
|
||||
|
||||
@@ -312,11 +312,16 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
GitHub's six-hour limit with nothing saved. Rules: the percentage is Steam's,
|
||||
FLOORED (`reviews_from_summary()` floors appreviews in integers too, so an
|
||||
ingest and the refresh agree; until v4.4.0 it was rounded, one point higher
|
||||
on ~40% of games); a GetItems count of 0 is never written - its summary has
|
||||
said 0 for a game with thousands (845070) - so a stored percentage or "N/A"
|
||||
is asked on appreviews (`ASK_CAP` 300 a run, daily order) and only that
|
||||
answer is written; no item, no usable summary or a failed batch keeps the
|
||||
value. A record's `last_updated` moves on a change and on a HEARTBEAT: a
|
||||
on ~40% of games); a GetItems count of 0 never replaces a percentage - its
|
||||
summary has said 0 for a game with thousands (845070) - so such a game is
|
||||
asked on appreviews (`ASK_CAP` 300 a run, daily order) and keeps its value
|
||||
when that gives no answer; "N/A" (never read) with a count of 0 becomes
|
||||
"No reviews"; no item, no usable summary or a failed batch keeps the value.
|
||||
**appreviews has answered with a 302 to the store front since 2026-10-23**
|
||||
(from here and from GitHub's runners; it worked on 10-17), so `fetch_full()`
|
||||
falls back to the same GetItems summary (`reviews_from_store_api()`) and an
|
||||
ingest no longer leaves "N/A". A record's `last_updated` moves on a change
|
||||
and on a HEARTBEAT: a
|
||||
game read whose stamp is 14-20 days old (crc32 jitter) is stamped again.
|
||||
The site calls a record older than 30 days stale (`web/src/lib/validation.ts`,
|
||||
`/health`), and this job used to stamp every game on every run; the
|
||||
|
||||
@@ -145,6 +145,18 @@ Rules worth keeping (CLAUDE.md has them in full):
|
||||
`queue: max`, `cancel-in-progress: false`, `ref: main` on every checkout, a
|
||||
`timeout-minutes`, or a budget less than 20 minutes below it.
|
||||
|
||||
## Found after the merge: appreviews redirects
|
||||
|
||||
The first run of the new Update Reviews on `main` (2026-10-23, 2 m 31 s, 4,540
|
||||
games changed - exactly the ones whose `last_updated` moved) asked `appreviews`
|
||||
about the 12 games GetItems counted at 0, and all 12 failed. From this machine
|
||||
and from GitHub's runners alike, `store.steampowered.com/appreviews/<id>?json=1`
|
||||
now answers 302 to the store front, whatever the headers; it answered on
|
||||
10-17. The 38 games the ingest added on 10-23 were the first to get "N/A".
|
||||
So the ingest falls back to the GetItems summary, and a game never read
|
||||
("N/A") with a count of 0 becomes "No reviews"; a stored percentage is still
|
||||
kept until a second source confirms the 0.
|
||||
|
||||
## Load, before and after
|
||||
|
||||
| | Before | After |
|
||||
|
||||
+17
-2
@@ -24,7 +24,7 @@ from .overrides import OverrideGuard
|
||||
from .peaks import raise_peak
|
||||
from .rotation import daily_order, day_number
|
||||
from .scraper import scrape_store_page
|
||||
from .store_items import paid_dlc_from_items, parse_store_item
|
||||
from .store_items import paid_dlc_from_items, parse_store_item, review_summary
|
||||
|
||||
|
||||
# Steam category descriptions that imply online multiplayer. Bias toward
|
||||
@@ -165,6 +165,16 @@ def apply_reviews(game: dict, summary: dict) -> dict:
|
||||
return game
|
||||
|
||||
|
||||
def reviews_from_store_api(client, appid: str) -> Optional[str]:
|
||||
"""A game's review summary from IStoreBrowseService/GetItems, as stored
|
||||
("No reviews" for a count of 0), or None when it could not be read - the
|
||||
ingest's answer when appreviews gives none. The daily refresh
|
||||
(update_reviews.py) reads the same summary, so the two agree."""
|
||||
items, failed = client.fetch_items([appid], {"include_reviews": True})
|
||||
found = None if appid in failed else review_summary(items.get(appid))
|
||||
return None if found is None else reviews_text(*found)
|
||||
|
||||
|
||||
def apply_players(game: dict, count: int) -> dict:
|
||||
game["current_players"] = f"{count:,}"
|
||||
# A sample of 0 says nothing about a peak; anything else may beat it.
|
||||
@@ -281,10 +291,15 @@ def fetch_full(game: dict, client=None, fetch_players=True,
|
||||
if data:
|
||||
apply_details(game, data)
|
||||
|
||||
# 2) Reviews
|
||||
# 2) Reviews: appreviews, else Steam's store API - appreviews has answered
|
||||
# with a redirect to the store front since 2026-10-23.
|
||||
summary = c.fetch_reviews(appid)
|
||||
if summary:
|
||||
apply_reviews(game, summary)
|
||||
else:
|
||||
value = reviews_from_store_api(c, appid)
|
||||
if value is not None:
|
||||
game["reviews"] = value
|
||||
|
||||
# 3) Players
|
||||
if fetch_players and STEAM_API_KEY:
|
||||
|
||||
@@ -28,16 +28,28 @@ def _page(name="Anti-Cheat Expert (ACE)", heading="Uses Kernel Level Anti-Cheat"
|
||||
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, page=None, status=None, items=None):
|
||||
"""`reviews` is appreviews' query_summary (None: it redirected), and
|
||||
`review_items` GetItems' answer to include_reviews (None: the batch failed)."""
|
||||
|
||||
def __init__(self, page=None, status=None, items=None, reviews=None, review_items=None):
|
||||
self.page = page
|
||||
self.status = status or ("ok" if page is not None else "network_error")
|
||||
self.items = items
|
||||
self.reviews, self.review_items = reviews, review_items
|
||||
self.asked_items = 0
|
||||
|
||||
def fetch_app_details(self, appid):
|
||||
return dict(DETAILS)
|
||||
|
||||
def fetch_reviews(self, appid):
|
||||
return None
|
||||
return self.reviews
|
||||
|
||||
def fetch_items(self, appids, data_request, give_up_after=10):
|
||||
assert data_request == {"include_reviews": True}
|
||||
self.asked_items += 1
|
||||
if self.review_items is None:
|
||||
return {}, set(appids)
|
||||
return {a: self.review_items[a] for a in appids if a in self.review_items}, set()
|
||||
|
||||
def fetch_store_page_full(self, appid):
|
||||
return (self.status, self.page if self.status == "ok" else None)
|
||||
@@ -83,6 +95,29 @@ def test_fetch_full_never_replaces_another_product():
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("EAC", "EAC", False)
|
||||
|
||||
|
||||
def _reviews_item(count, pct=0, label=""):
|
||||
return {"2507950": {"id": 2507950, "success": 1, "reviews": {"summary_filtered": {
|
||||
"review_count": count, "percent_positive": pct, "review_score_label": label}}}}
|
||||
|
||||
|
||||
def test_reviews_come_from_appreviews_when_it_answers():
|
||||
client = FakeClient(page=_page(), reviews={"total_reviews": 3, "total_positive": 2,
|
||||
"review_score_desc": "3 user reviews"})
|
||||
assert _fetch(client=client)["reviews"] == "66% (3 user reviews)"
|
||||
assert client.asked_items == 0
|
||||
|
||||
|
||||
def test_reviews_come_from_the_store_api_when_appreviews_does_not_answer():
|
||||
# Since 2026-10-23 appreviews answers with a redirect to the store front.
|
||||
client = FakeClient(page=_page(), review_items=_reviews_item(52341, 80, "Very Positive"))
|
||||
assert _fetch(client=client)["reviews"] == "80% (Very Positive)"
|
||||
assert _fetch(client=FakeClient(page=_page(), review_items=_reviews_item(0)))["reviews"] == "No reviews"
|
||||
|
||||
|
||||
def test_reviews_stay_unknown_when_neither_answers():
|
||||
assert _fetch(client=FakeClient(page=_page()))["reviews"] == "N/A"
|
||||
|
||||
|
||||
def test_apply_details_writes_no_anti_cheat():
|
||||
game = make_skeleton(LINK)
|
||||
fetcher.apply_details(game, VAC_DETAILS)
|
||||
|
||||
@@ -104,8 +104,9 @@ def test_appreviews_is_floored_in_integers(pos, total, expected):
|
||||
("N/A", _item("1", 12, 50, ""), (ur.SET, "50%")),
|
||||
("No reviews", _item("1", 0, 0, "No user reviews"), (ur.KEEP, None)),
|
||||
("93% (Very Positive)", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
|
||||
("N/A", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
|
||||
("", _item("1", 0), (ur.ASK, None)),
|
||||
("100% (3 user reviews)", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
|
||||
("N/A", _item("1", 0, 0, "No user reviews"), (ur.SET, "No reviews")), # never read: no question
|
||||
("", _item("1", 0), (ur.SET, "No reviews")),
|
||||
("50% (Mixed)", None, (ur.UNKNOWN, None)),
|
||||
("50% (Mixed)", _item("1"), (ur.UNKNOWN, None)), # no summary
|
||||
("50% (Mixed)", _item("1", 40, None, "Mixed"), (ur.UNKNOWN, None)), # count without a share
|
||||
@@ -156,7 +157,7 @@ def test_a_second_run_writes_the_same_bytes(repo):
|
||||
first = _bytes()
|
||||
out = repo(None, client)
|
||||
assert _bytes() == first
|
||||
assert out["570"]["reviews"] == "75% (4 user reviews)"
|
||||
assert out["570"]["reviews"] == "No reviews" and client.asked == []
|
||||
|
||||
|
||||
def test_a_zero_is_checked_on_appreviews_before_a_percentage_goes(repo):
|
||||
@@ -201,7 +202,7 @@ def test_appreviews_is_capped_and_stops_at_the_deadline(repo, monkeypatch):
|
||||
summary = {"total_reviews": 1, "total_positive": 1, "review_score_desc": "1 user reviews"}
|
||||
client = FakeClient(items=zero, appreviews={a: summary for a in zero})
|
||||
monkeypatch.setattr(ur, "ASK_CAP", 2)
|
||||
out = repo([_rec(a, "N/A") for a in zero], client)
|
||||
out = repo([_rec(a, "50% (Mixed)") for a in zero], client)
|
||||
assert len(client.asked) == 2
|
||||
assert sum(g["reviews"] == "100% (1 user reviews)" for g in out.values()) == 2
|
||||
|
||||
|
||||
@@ -13,11 +13,13 @@ The rules:
|
||||
- A summary with reviews is written "85% (Very Positive)": Steam's floored
|
||||
percentage and its label, the vocabulary appreviews has ("3 user
|
||||
reviews", "Mixed").
|
||||
- A count of 0 is never written on GetItems' word alone: its summary has
|
||||
said 0 for a game with thousands of reviews. A game stored as "No
|
||||
reviews" stays so; one stored with a percentage, "N/A" or nothing is asked
|
||||
on appreviews instead (at most ASK_CAP a run, in an order that changes
|
||||
daily), and that answer is written, "No reviews" included.
|
||||
- A count of 0 never replaces a percentage on GetItems' word alone: its
|
||||
summary has said 0 for a game with thousands of reviews. Such a game is
|
||||
asked on appreviews instead (at most ASK_CAP a run, in an order that
|
||||
changes daily), and keeps its percentage when appreviews gives no answer
|
||||
- which, since 2026-10-23, it does not: it redirects to the store front.
|
||||
A game stored as "N/A" or nothing, never read, gets "No reviews": the
|
||||
first count GetItems has replaces it.
|
||||
- No item, no usable summary, a failed batch or a failed appreviews request
|
||||
leaves the stored value alone. When GetItems answers for no game at all,
|
||||
nothing is written.
|
||||
@@ -36,6 +38,7 @@ Usage (from the repository root):
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import zlib
|
||||
from collections import Counter
|
||||
@@ -56,6 +59,7 @@ ASK_CAP = 300
|
||||
HEARTBEAT_DAYS = 14
|
||||
HEARTBEAT_JITTER_DAYS = 7
|
||||
NO_REVIEWS = "No reviews"
|
||||
PERCENTAGE = re.compile(r"\d{1,3}%")
|
||||
|
||||
# What decide() says to do with one game.
|
||||
SET = "set" # write this value
|
||||
@@ -72,7 +76,10 @@ def decide(stored: str, item: Optional[dict]) -> tuple[str, Optional[str]]:
|
||||
count, pct, label = found
|
||||
if count > 0:
|
||||
return SET, reviews_text(count, pct, label)
|
||||
return (KEEP, None) if stored == NO_REVIEWS else (ASK, None)
|
||||
if stored == NO_REVIEWS:
|
||||
return KEEP, None
|
||||
# A percentage is checked before a 0 replaces it; "N/A" was never known.
|
||||
return (ASK, None) if PERCENTAGE.match(stored or "") else (SET, NO_REVIEWS)
|
||||
|
||||
|
||||
def heartbeat_due(game: dict, now: datetime) -> bool:
|
||||
|
||||
Reference in new issue
Block a user