Merge pull request #210 from poli0981/fix/reviews-without-appreviews

fix(reviews): stop depending on appreviews, which now redirects
This commit is contained in:
poli0981 authored and GitHub committed 2026-10-23 22:47:55 +07:00
commit 8c81afab57
7 files changed
+101 -20

No files matched your search

+7 -1
View File
@@ -27,7 +27,13 @@ build. Repo public-facing version `4.3.3` → `4.4.0`.
days and five and a half hours. The percentage is now Steam's own, which
rounds down: about 4,000 games show one point less once (Counter-Strike 2
"85%" instead of "86%"), exactly as on their store pages. A summary that
says 0 reviews is double-checked game by game before it replaces anything.
says 0 reviews is double-checked game by game before it replaces a
percentage.
- **New games get their reviews again.** Steam's per-game reviews endpoint
has answered with a redirect to the store front since 2026-10-23, so the 38
games added that day were listed with "N/A". A new game now takes its
summary from the store API when that endpoint gives none, and a game never
read becomes "No reviews" until Steam counts its first one.
- **The daily purge needs two answers to remove a game.** Steam's store API
screens every game in a few minutes, and a game is removed only when
appdetails agrees; "unavailable" is asked twice, five minutes apart. A run
+10 -5
View File
@@ -312,11 +312,16 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
GitHub's six-hour limit with nothing saved. Rules: the percentage is Steam's,
FLOORED (`reviews_from_summary()` floors appreviews in integers too, so an
ingest and the refresh agree; until v4.4.0 it was rounded, one point higher
on ~40% of games); a GetItems count of 0 is never written - its summary has
said 0 for a game with thousands (845070) - so a stored percentage or "N/A"
is asked on appreviews (`ASK_CAP` 300 a run, daily order) and only that
answer is written; no item, no usable summary or a failed batch keeps the
value. A record's `last_updated` moves on a change and on a HEARTBEAT: a
on ~40% of games); a GetItems count of 0 never replaces a percentage - its
summary has said 0 for a game with thousands (845070) - so such a game is
asked on appreviews (`ASK_CAP` 300 a run, daily order) and keeps its value
when that gives no answer; "N/A" (never read) with a count of 0 becomes
"No reviews"; no item, no usable summary or a failed batch keeps the value.
**appreviews has answered with a 302 to the store front since 2026-10-23**
(from here and from GitHub's runners; it worked on 10-17), so `fetch_full()`
falls back to the same GetItems summary (`reviews_from_store_api()`) and an
ingest no longer leaves "N/A". A record's `last_updated` moves on a change
and on a HEARTBEAT: a
game read whose stamp is 14-20 days old (crc32 jitter) is stamped again.
The site calls a record older than 30 days stale (`web/src/lib/validation.ts`,
`/health`), and this job used to stamp every game on every run; the
+12
View File
@@ -145,6 +145,18 @@ Rules worth keeping (CLAUDE.md has them in full):
`queue: max`, `cancel-in-progress: false`, `ref: main` on every checkout, a
`timeout-minutes`, or a budget less than 20 minutes below it.
## Found after the merge: appreviews redirects
The first run of the new Update Reviews on `main` (2026-10-23, 2 m 31 s, 4,540
games changed - exactly the ones whose `last_updated` moved) asked `appreviews`
about the 12 games GetItems counted at 0, and all 12 failed. From this machine
and from GitHub's runners alike, `store.steampowered.com/appreviews/<id>?json=1`
now answers 302 to the store front, whatever the headers; it answered on
10-17. The 38 games the ingest added on 10-23 were the first to get "N/A".
So the ingest falls back to the GetItems summary, and a game never read
("N/A") with a count of 0 becomes "No reviews"; a stored percentage is still
kept until a second source confirms the 0.
## Load, before and after
| | Before | After |
+17 -2
View File
@@ -24,7 +24,7 @@ from .overrides import OverrideGuard
from .peaks import raise_peak
from .rotation import daily_order, day_number
from .scraper import scrape_store_page
from .store_items import paid_dlc_from_items, parse_store_item
from .store_items import paid_dlc_from_items, parse_store_item, review_summary
# Steam category descriptions that imply online multiplayer. Bias toward
@@ -165,6 +165,16 @@ def apply_reviews(game: dict, summary: dict) -> dict:
return game
def reviews_from_store_api(client, appid: str) -> Optional[str]:
"""A game's review summary from IStoreBrowseService/GetItems, as stored
("No reviews" for a count of 0), or None when it could not be read - the
ingest's answer when appreviews gives none. The daily refresh
(update_reviews.py) reads the same summary, so the two agree."""
items, failed = client.fetch_items([appid], {"include_reviews": True})
found = None if appid in failed else review_summary(items.get(appid))
return None if found is None else reviews_text(*found)
def apply_players(game: dict, count: int) -> dict:
game["current_players"] = f"{count:,}"
# A sample of 0 says nothing about a peak; anything else may beat it.
@@ -281,10 +291,15 @@ def fetch_full(game: dict, client=None, fetch_players=True,
if data:
apply_details(game, data)
# 2) Reviews
# 2) Reviews: appreviews, else Steam's store API - appreviews has answered
# with a redirect to the store front since 2026-10-23.
summary = c.fetch_reviews(appid)
if summary:
apply_reviews(game, summary)
else:
value = reviews_from_store_api(c, appid)
if value is not None:
game["reviews"] = value
# 3) Players
if fetch_players and STEAM_API_KEY:
+37 -2
View File
@@ -28,16 +28,28 @@ def _page(name="Anti-Cheat Expert (ACE)", heading="Uses Kernel Level Anti-Cheat"
class FakeClient:
def __init__(self, page=None, status=None, items=None):
"""`reviews` is appreviews' query_summary (None: it redirected), and
`review_items` GetItems' answer to include_reviews (None: the batch failed)."""
def __init__(self, page=None, status=None, items=None, reviews=None, review_items=None):
self.page = page
self.status = status or ("ok" if page is not None else "network_error")
self.items = items
self.reviews, self.review_items = reviews, review_items
self.asked_items = 0
def fetch_app_details(self, appid):
return dict(DETAILS)
def fetch_reviews(self, appid):
return None
return self.reviews
def fetch_items(self, appids, data_request, give_up_after=10):
assert data_request == {"include_reviews": True}
self.asked_items += 1
if self.review_items is None:
return {}, set(appids)
return {a: self.review_items[a] for a in appids if a in self.review_items}, set()
def fetch_store_page_full(self, appid):
return (self.status, self.page if self.status == "ok" else None)
@@ -83,6 +95,29 @@ def test_fetch_full_never_replaces_another_product():
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("EAC", "EAC", False)
def _reviews_item(count, pct=0, label=""):
return {"2507950": {"id": 2507950, "success": 1, "reviews": {"summary_filtered": {
"review_count": count, "percent_positive": pct, "review_score_label": label}}}}
def test_reviews_come_from_appreviews_when_it_answers():
client = FakeClient(page=_page(), reviews={"total_reviews": 3, "total_positive": 2,
"review_score_desc": "3 user reviews"})
assert _fetch(client=client)["reviews"] == "66% (3 user reviews)"
assert client.asked_items == 0
def test_reviews_come_from_the_store_api_when_appreviews_does_not_answer():
# Since 2026-10-23 appreviews answers with a redirect to the store front.
client = FakeClient(page=_page(), review_items=_reviews_item(52341, 80, "Very Positive"))
assert _fetch(client=client)["reviews"] == "80% (Very Positive)"
assert _fetch(client=FakeClient(page=_page(), review_items=_reviews_item(0)))["reviews"] == "No reviews"
def test_reviews_stay_unknown_when_neither_answers():
assert _fetch(client=FakeClient(page=_page()))["reviews"] == "N/A"
def test_apply_details_writes_no_anti_cheat():
game = make_skeleton(LINK)
fetcher.apply_details(game, VAC_DETAILS)
+5 -4
View File
@@ -104,8 +104,9 @@ def test_appreviews_is_floored_in_integers(pos, total, expected):
("N/A", _item("1", 12, 50, ""), (ur.SET, "50%")),
("No reviews", _item("1", 0, 0, "No user reviews"), (ur.KEEP, None)),
("93% (Very Positive)", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
("N/A", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
("", _item("1", 0), (ur.ASK, None)),
("100% (3 user reviews)", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
("N/A", _item("1", 0, 0, "No user reviews"), (ur.SET, "No reviews")), # never read: no question
("", _item("1", 0), (ur.SET, "No reviews")),
("50% (Mixed)", None, (ur.UNKNOWN, None)),
("50% (Mixed)", _item("1"), (ur.UNKNOWN, None)), # no summary
("50% (Mixed)", _item("1", 40, None, "Mixed"), (ur.UNKNOWN, None)), # count without a share
@@ -156,7 +157,7 @@ def test_a_second_run_writes_the_same_bytes(repo):
first = _bytes()
out = repo(None, client)
assert _bytes() == first
assert out["570"]["reviews"] == "75% (4 user reviews)"
assert out["570"]["reviews"] == "No reviews" and client.asked == []
def test_a_zero_is_checked_on_appreviews_before_a_percentage_goes(repo):
@@ -201,7 +202,7 @@ def test_appreviews_is_capped_and_stops_at_the_deadline(repo, monkeypatch):
summary = {"total_reviews": 1, "total_positive": 1, "review_score_desc": "1 user reviews"}
client = FakeClient(items=zero, appreviews={a: summary for a in zero})
monkeypatch.setattr(ur, "ASK_CAP", 2)
out = repo([_rec(a, "N/A") for a in zero], client)
out = repo([_rec(a, "50% (Mixed)") for a in zero], client)
assert len(client.asked) == 2
assert sum(g["reviews"] == "100% (1 user reviews)" for g in out.values()) == 2
+13 -6
View File
@@ -13,11 +13,13 @@ The rules:
- A summary with reviews is written "85% (Very Positive)": Steam's floored
percentage and its label, the vocabulary appreviews has ("3 user
reviews", "Mixed").
- A count of 0 is never written on GetItems' word alone: its summary has
said 0 for a game with thousands of reviews. A game stored as "No
reviews" stays so; one stored with a percentage, "N/A" or nothing is asked
on appreviews instead (at most ASK_CAP a run, in an order that changes
daily), and that answer is written, "No reviews" included.
- A count of 0 never replaces a percentage on GetItems' word alone: its
summary has said 0 for a game with thousands of reviews. Such a game is
asked on appreviews instead (at most ASK_CAP a run, in an order that
changes daily), and keeps its percentage when appreviews gives no answer
- which, since 2026-10-23, it does not: it redirects to the store front.
A game stored as "N/A" or nothing, never read, gets "No reviews": the
first count GetItems has replaces it.
- No item, no usable summary, a failed batch or a failed appreviews request
leaves the stored value alone. When GetItems answers for no game at all,
nothing is written.
@@ -36,6 +38,7 @@ Usage (from the repository root):
"""
import argparse
import os
import re
import sys
import zlib
from collections import Counter
@@ -56,6 +59,7 @@ ASK_CAP = 300
HEARTBEAT_DAYS = 14
HEARTBEAT_JITTER_DAYS = 7
NO_REVIEWS = "No reviews"
PERCENTAGE = re.compile(r"\d{1,3}%")
# What decide() says to do with one game.
SET = "set" # write this value
@@ -72,7 +76,10 @@ def decide(stored: str, item: Optional[dict]) -> tuple[str, Optional[str]]:
count, pct, label = found
if count > 0:
return SET, reviews_text(count, pct, label)
return (KEEP, None) if stored == NO_REVIEWS else (ASK, None)
if stored == NO_REVIEWS:
return KEEP, None
# A percentage is checked before a 0 replaces it; "N/A" was never known.
return (ASK, None) if PERCENTAGE.match(stored or "") else (SET, NO_REVIEWS)
def heartbeat_due(game: dict, now: datetime) -> bool: