Merge pull request #210 from poli0981/fix/reviews-without-appreviews
fix(reviews): stop depending on appreviews, which now redirects
This commit is contained in:
7 files changed
+101
-20
No files matched your search
+7
-1
@@ -27,7 +27,13 @@ build. Repo public-facing version `4.3.3` → `4.4.0`.
|
||||
days and five and a half hours. The percentage is now Steam's own, which
|
||||
rounds down: about 4,000 games show one point less once (Counter-Strike 2
|
||||
"85%" instead of "86%"), exactly as on their store pages. A summary that
|
||||
says 0 reviews is double-checked game by game before it replaces anything.
|
||||
says 0 reviews is double-checked game by game before it replaces a
|
||||
percentage.
|
||||
- **New games get their reviews again.** Steam's per-game reviews endpoint
|
||||
has answered with a redirect to the store front since 2026-10-23, so the 38
|
||||
games added that day were listed with "N/A". A new game now takes its
|
||||
summary from the store API when that endpoint gives none, and a game never
|
||||
read becomes "No reviews" until Steam counts its first one.
|
||||
- **The daily purge needs two answers to remove a game.** Steam's store API
|
||||
screens every game in a few minutes, and a game is removed only when
|
||||
appdetails agrees; "unavailable" is asked twice, five minutes apart. A run
|
||||
|
||||
@@ -312,11 +312,16 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
GitHub's six-hour limit with nothing saved. Rules: the percentage is Steam's,
|
||||
FLOORED (`reviews_from_summary()` floors appreviews in integers too, so an
|
||||
ingest and the refresh agree; until v4.4.0 it was rounded, one point higher
|
||||
on ~40% of games); a GetItems count of 0 is never written - its summary has
|
||||
said 0 for a game with thousands (845070) - so a stored percentage or "N/A"
|
||||
is asked on appreviews (`ASK_CAP` 300 a run, daily order) and only that
|
||||
answer is written; no item, no usable summary or a failed batch keeps the
|
||||
value. A record's `last_updated` moves on a change and on a HEARTBEAT: a
|
||||
on ~40% of games); a GetItems count of 0 never replaces a percentage - its
|
||||
summary has said 0 for a game with thousands (845070) - so such a game is
|
||||
asked on appreviews (`ASK_CAP` 300 a run, daily order) and keeps its value
|
||||
when that gives no answer; "N/A" (never read) with a count of 0 becomes
|
||||
"No reviews"; no item, no usable summary or a failed batch keeps the value.
|
||||
**appreviews has answered with a 302 to the store front since 2026-10-23**
|
||||
(from here and from GitHub's runners; it worked on 10-17), so `fetch_full()`
|
||||
falls back to the same GetItems summary (`reviews_from_store_api()`) and an
|
||||
ingest no longer leaves "N/A". A record's `last_updated` moves on a change
|
||||
and on a HEARTBEAT: a
|
||||
game read whose stamp is 14-20 days old (crc32 jitter) is stamped again.
|
||||
The site calls a record older than 30 days stale (`web/src/lib/validation.ts`,
|
||||
`/health`), and this job used to stamp every game on every run; the
|
||||
|
||||
@@ -145,6 +145,18 @@ Rules worth keeping (CLAUDE.md has them in full):
|
||||
`queue: max`, `cancel-in-progress: false`, `ref: main` on every checkout, a
|
||||
`timeout-minutes`, or a budget less than 20 minutes below it.
|
||||
|
||||
## Found after the merge: appreviews redirects
|
||||
|
||||
The first run of the new Update Reviews on `main` (2026-10-23, 2 m 31 s, 4,540
|
||||
games changed - exactly the ones whose `last_updated` moved) asked `appreviews`
|
||||
about the 12 games GetItems counted at 0, and all 12 failed. From this machine
|
||||
and from GitHub's runners alike, `store.steampowered.com/appreviews/<id>?json=1`
|
||||
now answers 302 to the store front, whatever the headers; it answered on
|
||||
10-17. The 38 games the ingest added on 10-23 were the first to get "N/A".
|
||||
So the ingest falls back to the GetItems summary, and a game never read
|
||||
("N/A") with a count of 0 becomes "No reviews"; a stored percentage is still
|
||||
kept until a second source confirms the 0.
|
||||
|
||||
## Load, before and after
|
||||
|
||||
| | Before | After |
|
||||
|
||||
+17
-2
@@ -24,7 +24,7 @@ from .overrides import OverrideGuard
|
||||
from .peaks import raise_peak
|
||||
from .rotation import daily_order, day_number
|
||||
from .scraper import scrape_store_page
|
||||
from .store_items import paid_dlc_from_items, parse_store_item
|
||||
from .store_items import paid_dlc_from_items, parse_store_item, review_summary
|
||||
|
||||
|
||||
# Steam category descriptions that imply online multiplayer. Bias toward
|
||||
@@ -165,6 +165,16 @@ def apply_reviews(game: dict, summary: dict) -> dict:
|
||||
return game
|
||||
|
||||
|
||||
def reviews_from_store_api(client, appid: str) -> Optional[str]:
|
||||
"""A game's review summary from IStoreBrowseService/GetItems, as stored
|
||||
("No reviews" for a count of 0), or None when it could not be read - the
|
||||
ingest's answer when appreviews gives none. The daily refresh
|
||||
(update_reviews.py) reads the same summary, so the two agree."""
|
||||
items, failed = client.fetch_items([appid], {"include_reviews": True})
|
||||
found = None if appid in failed else review_summary(items.get(appid))
|
||||
return None if found is None else reviews_text(*found)
|
||||
|
||||
|
||||
def apply_players(game: dict, count: int) -> dict:
|
||||
game["current_players"] = f"{count:,}"
|
||||
# A sample of 0 says nothing about a peak; anything else may beat it.
|
||||
@@ -281,10 +291,15 @@ def fetch_full(game: dict, client=None, fetch_players=True,
|
||||
if data:
|
||||
apply_details(game, data)
|
||||
|
||||
# 2) Reviews
|
||||
# 2) Reviews: appreviews, else Steam's store API - appreviews has answered
|
||||
# with a redirect to the store front since 2026-10-23.
|
||||
summary = c.fetch_reviews(appid)
|
||||
if summary:
|
||||
apply_reviews(game, summary)
|
||||
else:
|
||||
value = reviews_from_store_api(c, appid)
|
||||
if value is not None:
|
||||
game["reviews"] = value
|
||||
|
||||
# 3) Players
|
||||
if fetch_players and STEAM_API_KEY:
|
||||
|
||||
@@ -28,16 +28,28 @@ def _page(name="Anti-Cheat Expert (ACE)", heading="Uses Kernel Level Anti-Cheat"
|
||||
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, page=None, status=None, items=None):
|
||||
"""`reviews` is appreviews' query_summary (None: it redirected), and
|
||||
`review_items` GetItems' answer to include_reviews (None: the batch failed)."""
|
||||
|
||||
def __init__(self, page=None, status=None, items=None, reviews=None, review_items=None):
|
||||
self.page = page
|
||||
self.status = status or ("ok" if page is not None else "network_error")
|
||||
self.items = items
|
||||
self.reviews, self.review_items = reviews, review_items
|
||||
self.asked_items = 0
|
||||
|
||||
def fetch_app_details(self, appid):
|
||||
return dict(DETAILS)
|
||||
|
||||
def fetch_reviews(self, appid):
|
||||
return None
|
||||
return self.reviews
|
||||
|
||||
def fetch_items(self, appids, data_request, give_up_after=10):
|
||||
assert data_request == {"include_reviews": True}
|
||||
self.asked_items += 1
|
||||
if self.review_items is None:
|
||||
return {}, set(appids)
|
||||
return {a: self.review_items[a] for a in appids if a in self.review_items}, set()
|
||||
|
||||
def fetch_store_page_full(self, appid):
|
||||
return (self.status, self.page if self.status == "ok" else None)
|
||||
@@ -83,6 +95,29 @@ def test_fetch_full_never_replaces_another_product():
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("EAC", "EAC", False)
|
||||
|
||||
|
||||
def _reviews_item(count, pct=0, label=""):
|
||||
return {"2507950": {"id": 2507950, "success": 1, "reviews": {"summary_filtered": {
|
||||
"review_count": count, "percent_positive": pct, "review_score_label": label}}}}
|
||||
|
||||
|
||||
def test_reviews_come_from_appreviews_when_it_answers():
|
||||
client = FakeClient(page=_page(), reviews={"total_reviews": 3, "total_positive": 2,
|
||||
"review_score_desc": "3 user reviews"})
|
||||
assert _fetch(client=client)["reviews"] == "66% (3 user reviews)"
|
||||
assert client.asked_items == 0
|
||||
|
||||
|
||||
def test_reviews_come_from_the_store_api_when_appreviews_does_not_answer():
|
||||
# Since 2026-10-23 appreviews answers with a redirect to the store front.
|
||||
client = FakeClient(page=_page(), review_items=_reviews_item(52341, 80, "Very Positive"))
|
||||
assert _fetch(client=client)["reviews"] == "80% (Very Positive)"
|
||||
assert _fetch(client=FakeClient(page=_page(), review_items=_reviews_item(0)))["reviews"] == "No reviews"
|
||||
|
||||
|
||||
def test_reviews_stay_unknown_when_neither_answers():
|
||||
assert _fetch(client=FakeClient(page=_page()))["reviews"] == "N/A"
|
||||
|
||||
|
||||
def test_apply_details_writes_no_anti_cheat():
|
||||
game = make_skeleton(LINK)
|
||||
fetcher.apply_details(game, VAC_DETAILS)
|
||||
|
||||
@@ -104,8 +104,9 @@ def test_appreviews_is_floored_in_integers(pos, total, expected):
|
||||
("N/A", _item("1", 12, 50, ""), (ur.SET, "50%")),
|
||||
("No reviews", _item("1", 0, 0, "No user reviews"), (ur.KEEP, None)),
|
||||
("93% (Very Positive)", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
|
||||
("N/A", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
|
||||
("", _item("1", 0), (ur.ASK, None)),
|
||||
("100% (3 user reviews)", _item("1", 0, 0, "No user reviews"), (ur.ASK, None)),
|
||||
("N/A", _item("1", 0, 0, "No user reviews"), (ur.SET, "No reviews")), # never read: no question
|
||||
("", _item("1", 0), (ur.SET, "No reviews")),
|
||||
("50% (Mixed)", None, (ur.UNKNOWN, None)),
|
||||
("50% (Mixed)", _item("1"), (ur.UNKNOWN, None)), # no summary
|
||||
("50% (Mixed)", _item("1", 40, None, "Mixed"), (ur.UNKNOWN, None)), # count without a share
|
||||
@@ -156,7 +157,7 @@ def test_a_second_run_writes_the_same_bytes(repo):
|
||||
first = _bytes()
|
||||
out = repo(None, client)
|
||||
assert _bytes() == first
|
||||
assert out["570"]["reviews"] == "75% (4 user reviews)"
|
||||
assert out["570"]["reviews"] == "No reviews" and client.asked == []
|
||||
|
||||
|
||||
def test_a_zero_is_checked_on_appreviews_before_a_percentage_goes(repo):
|
||||
@@ -201,7 +202,7 @@ def test_appreviews_is_capped_and_stops_at_the_deadline(repo, monkeypatch):
|
||||
summary = {"total_reviews": 1, "total_positive": 1, "review_score_desc": "1 user reviews"}
|
||||
client = FakeClient(items=zero, appreviews={a: summary for a in zero})
|
||||
monkeypatch.setattr(ur, "ASK_CAP", 2)
|
||||
out = repo([_rec(a, "N/A") for a in zero], client)
|
||||
out = repo([_rec(a, "50% (Mixed)") for a in zero], client)
|
||||
assert len(client.asked) == 2
|
||||
assert sum(g["reviews"] == "100% (1 user reviews)" for g in out.values()) == 2
|
||||
|
||||
|
||||
@@ -13,11 +13,13 @@ The rules:
|
||||
- A summary with reviews is written "85% (Very Positive)": Steam's floored
|
||||
percentage and its label, the vocabulary appreviews has ("3 user
|
||||
reviews", "Mixed").
|
||||
- A count of 0 is never written on GetItems' word alone: its summary has
|
||||
said 0 for a game with thousands of reviews. A game stored as "No
|
||||
reviews" stays so; one stored with a percentage, "N/A" or nothing is asked
|
||||
on appreviews instead (at most ASK_CAP a run, in an order that changes
|
||||
daily), and that answer is written, "No reviews" included.
|
||||
- A count of 0 never replaces a percentage on GetItems' word alone: its
|
||||
summary has said 0 for a game with thousands of reviews. Such a game is
|
||||
asked on appreviews instead (at most ASK_CAP a run, in an order that
|
||||
changes daily), and keeps its percentage when appreviews gives no answer
|
||||
- which, since 2026-10-23, it does not: it redirects to the store front.
|
||||
A game stored as "N/A" or nothing, never read, gets "No reviews": the
|
||||
first count GetItems has replaces it.
|
||||
- No item, no usable summary, a failed batch or a failed appreviews request
|
||||
leaves the stored value alone. When GetItems answers for no game at all,
|
||||
nothing is written.
|
||||
@@ -36,6 +38,7 @@ Usage (from the repository root):
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import zlib
|
||||
from collections import Counter
|
||||
@@ -56,6 +59,7 @@ ASK_CAP = 300
|
||||
HEARTBEAT_DAYS = 14
|
||||
HEARTBEAT_JITTER_DAYS = 7
|
||||
NO_REVIEWS = "No reviews"
|
||||
PERCENTAGE = re.compile(r"\d{1,3}%")
|
||||
|
||||
# What decide() says to do with one game.
|
||||
SET = "set" # write this value
|
||||
@@ -72,7 +76,10 @@ def decide(stored: str, item: Optional[dict]) -> tuple[str, Optional[str]]:
|
||||
count, pct, label = found
|
||||
if count > 0:
|
||||
return SET, reviews_text(count, pct, label)
|
||||
return (KEEP, None) if stored == NO_REVIEWS else (ASK, None)
|
||||
if stored == NO_REVIEWS:
|
||||
return KEEP, None
|
||||
# A percentage is checked before a 0 replaces it; "N/A" was never known.
|
||||
return (ASK, None) if PERCENTAGE.match(stored or "") else (SET, NO_REVIEWS)
|
||||
|
||||
|
||||
def heartbeat_due(game: dict, now: datetime) -> bool:
|
||||
|
||||
Reference in new issue
Block a user