fix(pipeline): "Free To Play" is not a genre

appdetails writes "Free To Play"; SKIP_GENRE_TAGS said "Free to Play", and
the exact match let 1,357 of 12,854 games be stored with it as their genre
(the third most common), while the fallback to the first name could pick
it as well.

- fetcher.pick_genre() compares casefolded and never falls back to a price
  model or a release state (NOT_GENRES); with nothing else it says
  "Uncategorized". ingest uses it.
- Refresh Store Data replaces a stored one from appdetails: 150 games a
  day, first after the page repairs, never under an override. It is the one
  manual-field value the job writes; no person chose it. A dry run on the
  first 150 (2026-10-30): Casual 91, Indie 24, Strategy 11, Simulation 11,
  Uncategorized 10, three others; all answered, no override.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 committed 2026-10-30 17:43:03 +07:00
1 parent 9ff338c734
commit bca818ee19
8 files changed
+199 -28

No files matched your search

+8 -1
View File
@@ -10,7 +10,8 @@ The catalogue now takes free games in every language, not only the ones
that support English: about 1,300 more, most of them Chinese, Japanese or
Korean, arrive through the review queue over the next days. The site and
the apps get a language filter, search that matches across scripts and
widths, and CSV files that Excel opens correctly. Desktop and Android
widths, and CSV files that Excel opens correctly; and 1,357 games filed under
"Free To Play" get a real genre. Desktop and Android
bumped `2.3.0` → `2.4.0`; repo public-facing version `4.5.1` → `4.6.0`.
The desktop app offers 2.4.0 through its updater once the release is
published; on Android, install the new APK over the old one.
@@ -46,6 +47,12 @@ published; on Android, install the new APK over the old one.
intact (it now starts with a byte order mark).
- A long Chinese or Japanese description is cut at the end of a sentence
where it can be, and never ends in `、…` or `。…`.
- **"Free To Play" is no longer a genre.** Steam spells it with a capital T,
the list of names never to use as a genre had a small one, and 1,357 games
were filed under "Free To Play" - the third most common genre, and one that
says nothing about a catalogue of free games. New games get their real
genre, and the daily store refresh gives the 1,357 theirs from Steam, 150 a
day (about nine days). A genre someone set by hand is never touched.
- A queued store link written in full-width digits (`5118700`) is read
as the appid it is; digits of any other script are refused. Such an appid
used to slip through, and the next daily refresh would have stopped on it.
+10 -1
View File
@@ -73,6 +73,13 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
field an active override covers, or a retired entry's value, would be undone
by `apply_overrides()` and bump `last_updated` on every run.
One stored genre is not a judgement: a value in `NOT_GENRES` ("Free To
Play", "Early Access"; `constants.py`), a price model or a release state.
Until v4.6.0 the skip list matched appdetails' names case-sensitively and
1,357 games were stored with "Free To Play". `fetcher.pick_genre()` compares
casefolded and never falls back to one, and Refresh Store Data replaces a
stored one (`is_stray_genre()`) from appdetails, guarded like any writer.
- **`data/overrides/<appid>.json` is human intent, and it is permanent.**
`apply_overrides()` is called from inside `save_main()` — the single choke
point through which all 15 data-writing scripts pass, and the only code that
@@ -236,7 +243,9 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
The slice is visited in `rotation.daily_order()`, so a run that meets its
time budget leaves a different tail each day. It
replaces values only with NON-EMPTY ones, never writes `MANUAL_FIELDS` or
`notes`, trusts `has_paid_dlc` only from a real store page (an age gate has
`notes` - except a stray genre (above), read from appdetails every day,
first after the page repairs, at most `GENRE_DAILY_CAP` a day, and left
alone under an override - trusts `has_paid_dlc` only from a real store page (an age gate has
no language table or tags) or a gated game's DLC prices, compares every
value as stored (normalised, `description` truncated - and a blurb merely
cut off somewhere else is written without counting as news: later, now
+15 -5
View File
@@ -150,12 +150,22 @@ A window counts as done only when its log says "walked past the window"; a
run that reaches its limit is run again. Windows before 2019 get their
non-English games when they are first swept.
## Left for later
## Found on the way: genre "Free To Play"
- **Genre "Free To Play".** `SKIP_GENRE_TAGS` holds `"Free to Play"`, but
appdetails writes `"Free To Play"` and the comparison is case-sensitive:
1,084 records have that genre, 809 of them added since 2026-10-01. `genre`
is a human-judgement field, so the fix needs its own decision.
`SKIP_GENRE_TAGS` held `"Free to Play"`, but appdetails writes `"Free To
Play"` and the comparison was case-sensitive, and the fallback took the first
name anyway: 1,357 of 12,854 games had it as their genre. Not a language
problem, but fixed in the same release:
- `fetcher.pick_genre()` compares casefolded and never falls back to a price
model or a release state (`NOT_GENRES`); with nothing else it says
"Uncategorized".
- Refresh Store Data replaces a stored "Free To Play" from appdetails, 150
games a day, first after the page repairs, never under an override. `genre`
is a human-judgement field, and this is its one scheduled writer: no person
chose "Free To Play".
## Left for later
- **Games not sold in the US.** Every request goes out as the US; covering
region-locked games needs another country or runner.
- **CJK fonts and `lang`.** CJK text renders in the system's font (a Linux
+5
View File
@@ -47,6 +47,11 @@ SKIP_GENRE_TAGS = frozenset({
"Cross-Platform Multiplayer", "In-App Purchases",
"Massively Multiplayer", "Competitive",
})
# Not genres at all: a price model and a release state. Every game here is
# free, so "Free To Play" tells a reader nothing. Both are on the skip list
# above; this is what a genre may never FALL BACK to (fetcher.pick_genre), and
# what Refresh Store Data replaces when it finds one stored.
NOT_GENRES = frozenset({"Free to Play", "Early Access"})
# ──────────── Anti-Cheat patterns ────────────
# A key or a pattern matches a name only as whole words (core/anticheat.py
+35 -4
View File
@@ -14,7 +14,7 @@ from typing import Optional
from .anticheat import apply_anti_cheat, from_categories
from .budget import Deadline, report
from .constants import (
STEAM_API_KEY, SKIP_GENRE_TAGS,
STEAM_API_KEY, SKIP_GENRE_TAGS, NOT_GENRES,
BATCH_SIZE, BATCH_PAUSE_MIN, BATCH_PAUSE_MAX,
)
from .steam_client import get_client
@@ -58,6 +58,39 @@ def _infer_type_game(categories: list[dict]) -> str:
return ""
# ──────────── Genre ────────────
_SKIP_GENRES = frozenset(t.casefold() for t in SKIP_GENRE_TAGS)
_NOT_GENRES = frozenset(t.casefold() for t in NOT_GENRES)
def is_stray_genre(genre) -> bool:
"""A stored genre that is not a genre ("Free To Play"): the scraper's
leftover, never a choice a person made."""
return isinstance(genre, str) and genre.strip().casefold() in _NOT_GENRES
def pick_genre(genres: list) -> str:
"""The genre for appdetails' `genres` names: the first not on the skip
list, else the first that is a genre at all, else "Uncategorized".
Case-insensitive. appdetails writes "Free To Play" and the skip list said
"Free to Play"; the exact match let 1,357 games be stored with it as their
genre (2026-10-30), and the fallback to the first name could pick it too.
"""
names = [g.strip() for g in genres if isinstance(g, str) and g.strip()]
kept = [g for g in names if g.casefold() not in _SKIP_GENRES]
if kept:
return kept[0]
real = [g for g in names if g.casefold() not in _NOT_GENRES]
return real[0] if real else "Uncategorized"
def appdetails_genres(data: dict) -> list:
"""The genre names appdetails lists, in its order."""
return [g.get("description", "") for g in data.get("genres") or [] if isinstance(g, dict)]
# ──────────── Apply functions ────────────
def apply_details(game: dict, data: dict) -> dict:
@@ -77,9 +110,7 @@ def apply_details(game: dict, data: dict) -> dict:
game["header_image"] = img
if is_empty(game.get("genre")):
raw = [g["description"] for g in data.get("genres", [])]
filt = [g for g in raw if g not in SKIP_GENRE_TAGS]
game["genre"] = filt[0] if filt else (raw[0] if raw else "Uncategorized")
game["genre"] = pick_genre(appdetails_genres(data))
if is_empty(game.get("developer")):
devs = data.get("developers", [])
+52 -16
View File
@@ -41,7 +41,12 @@ The rules, because these fields were fill-if-empty until now:
- A value is only ever replaced by a NON-EMPTY one. A failed request, an age
gate, a delisted app or a changed page layout leaves the stored value alone.
- MANUAL_FIELDS are never written (genre, type_game, anti_cheat, safe, ...),
and neither is `notes`: health checks and "no longer free" belong to
with one exception: a stored genre that is not a genre ("Free To Play",
left by a case-sensitive skip list until v4.6.0) is replaced by the one
appdetails gives (fetcher.pick_genre), unless an override owns the field.
No person chose it. Such games are in the details group every day, first
after the page repairs, at most GENRE_DAILY_CAP a day.
Nor is `notes` written: health checks and "no longer free" belong to
purge_unhealthy.py and mark_dead_games.py. A game appdetails no longer
calls free is only listed, in a ::warning:: - purge removes a game only
when Steam's store API flags it too, and this is where the opposite case
@@ -78,11 +83,14 @@ from core.data_store import (
ELLIPSIS, extract_appid, is_empty, load_main, migrate_record, now_iso, save_main,
truncate_description,
)
from core.fetcher import process_batch, scrape_store_fields
from core.fetcher import (
appdetails_genres, is_stray_genre, pick_genre, process_batch, scrape_store_fields,
)
from core.health_checker import free_per_appdetails
from core.normalize import (
clean_list, clean_text, decode_entities, normalize_records, normalize_release_date,
)
from core.overrides import OverrideGuard
# The rotation (crc32 slices) is shared with refresh_peaks.py.
from core.rotation import cycle_for, daily_order, day_number, due
from core.steam_client import get_client
@@ -97,6 +105,10 @@ DETAILS_DAILY = 500
# 2026-10-10; the cap is for the day something breaks for every game at once,
# so a two-hour job cannot turn into a whole-catalogue crawl.
MISSING_DAILY_CAP = 150
# Games with a stray genre ("Free To Play") whose appdetails are read a day on
# top of the rotation: 1,357 on 2026-10-30, so about nine days, then only the
# odd one a new rule lets through.
GENRE_DAILY_CAP = 150
PAGE_FIELDS = ("tags", "languages", "language_details", "has_paid_dlc")
DETAILS_FIELDS = (
@@ -215,18 +227,28 @@ def _fuller_cut(stored, fresh: str) -> bool:
return bool(fresh_stem) and fresh != stored and stem.startswith(fresh_stem)
def apply_app_details(game: dict, data: dict) -> list[str]:
def apply_app_details(game: dict, data: dict, guard: Optional[OverrideGuard] = None) -> list[str]:
changed: list[str] = []
for field, value in details_values(data).items():
if field == "description" and _fuller_cut(game.get("description"), value):
game["description"] = value
continue
_replace(game, field, value, changed)
# The one manual field this job writes, and only a value no person chose:
# a genre that is not a genre. Left alone when appdetails names no genre,
# and when an override owns the field.
genres = appdetails_genres(data)
if is_stray_genre(game.get("genre")) and genres:
genre = pick_genre(genres)
appid = extract_appid(game.get("link", ""))
if guard is None or guard.allows(appid, "genre", genre):
_replace(game, "genre", genre, changed)
return changed
def refresh_game(game: dict, client, page: bool, details: bool,
not_free: Optional[list] = None) -> list[str]:
not_free: Optional[list] = None,
guard: Optional[OverrideGuard] = None) -> list[str]:
"""Refresh one game in place. Returns the fields that changed. A game
whose appdetails no longer call it free is added to `not_free`."""
appid = extract_appid(game.get("link", ""))
@@ -237,7 +259,7 @@ def refresh_game(game: dict, client, page: bool, details: bool,
if details:
data = client.fetch_app_details(appid)
if data:
changed += apply_app_details(game, data)
changed += apply_app_details(game, data, guard)
if not_free is not None and not free_per_appdetails(data):
not_free.append(f"{appid} {game.get('name', '')[:40]}")
if page:
@@ -261,23 +283,34 @@ def build_plan(games: list[dict], day: int, want_page: bool = True,
page_cycle: int = PAGE_CYCLE_DAYS,
details_cycle: int = DETAILS_CYCLE_DAYS) -> list[tuple]:
"""[(game, page, details)] for one day: the games with empty store fields
first (page group, every day), then the day's rotation slices.
first (page group, every day), then the games with a stray genre
(appdetails, every day), then the day's rotation slices.
Both parts come in an order that changes daily (rotation.daily_order):
the repairs are capped at MISSING_DAILY_CAP, so a residue Steam never
fills cannot hold the same places forever, and a run that meets its time
budget leaves a different part of the slice for the next one.
missing_only keeps just the repairs, store page only.
All of it comes in an order that changes daily (rotation.daily_order):
the repairs are capped (MISSING_DAILY_CAP, GENRE_DAILY_CAP), so a residue
Steam never fills cannot hold the same places forever, and a run that
meets its time budget leaves a different part of the slice for the next one.
missing_only keeps just the store-field repairs, store page only.
"""
order = daily_order(games, day)
repairs: list[dict] = []
if want_page:
repairs = daily_order([g for g in games if missing_store_fields(g)], day)[:MISSING_DAILY_CAP]
repairs = [g for g in order if missing_store_fields(g)][:MISSING_DAILY_CAP]
if missing_only:
return [(g, True, False) for g in repairs]
plan = [(g, True, want_details and due(g, details_cycle, day)) for g in repairs]
strays: list[dict] = []
if want_details:
strays = [g for g in order if is_stray_genre(g.get("genre"))][:GENRE_DAILY_CAP]
stray_ids = {id(g) for g in strays}
plan = [(g, True, want_details and (due(g, details_cycle, day) or id(g) in stray_ids))
for g in repairs]
taken = {id(g) for g in repairs}
for g in daily_order(games, day):
for g in strays:
if id(g) not in taken:
plan.append((g, want_page and due(g, page_cycle, day), True))
taken.add(id(g))
for g in order:
if id(g) in taken:
continue
page = want_page and due(g, page_cycle, day)
@@ -321,13 +354,16 @@ def main(argv: Optional[list[str]] = None) -> None:
n_page = sum(1 for _, p, _ in plan if p)
n_details = sum(1 for _, _, d in plan if d)
n_missing = sum(1 for g, p, _ in plan if p and missing_store_fields(g))
n_stray = sum(1 for g, _, d in plan if d and is_stray_genre(g.get("genre")))
print(f"{len(games)} games; day {day}: store page for {n_page} "
f"(slice {day % page_cycle}/{page_cycle}, {n_missing} with empty store fields), "
f"appdetails for {n_details} (slice {day % details_cycle}/{details_cycle})")
f"appdetails for {n_details} (slice {day % details_cycle}/{details_cycle}, "
f"{n_stray} with a genre that is not one)")
if not plan:
return
client = get_client()
guard = OverrideGuard.load()
flags = {id(g): (page, details) for g, page, details in plan}
fields: Counter = Counter()
not_free: list[str] = []
@@ -336,7 +372,7 @@ def main(argv: Optional[list[str]] = None) -> None:
def refresh(game: dict) -> None:
nonlocal touched
page, details = flags[id(game)]
changed = refresh_game(game, client, page=page, details=details, not_free=not_free)
changed = refresh_game(game, client, page=page, details=details, not_free=not_free, guard=guard)
if changed:
touched += 1
fields.update(changed)
+34 -1
View File
@@ -3,11 +3,13 @@ fetch_full() builds every new game (ingest_new.py) and refills incomplete ones
(update_data.py, refetch_all.py). These pin how it treats the anti-cheat
fields: filled from the developer's disclosure on the page it already fetched,
else from appdetails' VAC category, never replaced, and never where an
override would undo it.
override would undo it. And how it picks a genre: never a price model.
Steam is never called: the client is a fake. Run from the repository root:
python -m pytest scripts/tests
"""
import pytest
from core import fetcher
from core.data_store import make_skeleton
from core.overrides import OverrideGuard
@@ -124,6 +126,37 @@ def test_apply_details_writes_no_anti_cheat():
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("-", "", None)
@pytest.mark.parametrize("genres, genre", [
(["Action", "Free To Play"], "Action"),
# appdetails capitalises the T; the skip list used to miss it.
(["Casual", "Free To Play", "Indie"], "Casual"),
(["Free To Play", "Massively Multiplayer"], "Massively Multiplayer"),
(["Free To Play", "Early Access"], "Uncategorized"),
(["Free To Play"], "Uncategorized"),
(["Indie", "Simulation"], "Simulation"),
([], "Uncategorized"),
([" RPG ", None, ""], "RPG"),
])
def test_pick_genre_never_picks_a_price_model(genres, genre):
assert fetcher.pick_genre(genres) == genre
def test_apply_details_reads_genres_case_insensitively():
game = make_skeleton(LINK)
game["genre"] = ""
fetcher.apply_details(game, {"genres": [{"id": "37", "description": "Free To Play"},
{"id": "4", "description": "Casual"}]})
assert game["genre"] == "Casual"
@pytest.mark.parametrize("genre, stray", [
("Free To Play", True), ("free to play", True), (" Early Access ", True),
("Casual", False), ("", False), (None, False), (["Free To Play"], False),
])
def test_a_stray_genre_is_one_no_person_chose(genre, stray):
assert fetcher.is_stray_genre(genre) is stray
def test_the_guard_reaches_fetch_full():
# A restored game keeps its override, here "nobody has verified the kernel
# flag": filling it would be undone by save_main() on every run.
+40
View File
@@ -344,6 +344,22 @@ def test_missing_only_takes_just_the_repairs():
assert [(g["link"], p, d) for g, p, d in plan] == [(games[1]["link"], True, False)]
def test_stray_genres_are_read_every_day_capped(monkeypatch):
monkeypatch.setattr(rsd, "GENRE_DAILY_CAP", 2)
games = [_game(str(10 * i), genre="Free To Play") for i in range(1, 6)] + [_game("730")]
plan = rsd.build_plan(games, day=3)
strays = [(g, d) for g, _, d in plan if g["genre"] == "Free To Play" and d and not rsd.due(
g, rsd.DETAILS_CYCLE_DAYS, 3)]
assert len(strays) <= 2
assert all(d for _, d in strays)
assert sum(1 for g, _, _ in plan if g["link"].endswith("/730/")) <= 1
def test_a_page_repair_with_a_stray_genre_reads_its_details_too():
plan = rsd.build_plan([_game("4156010", tags=[], genre="Free To Play")], day=0)
assert [(p, d) for _, p, d in plan] == [(True, True)]
def test_a_details_only_run_repairs_no_page():
plan = rsd.build_plan([_game("4156010", tags=[])], day=0, want_page=False)
assert not any(p for _, p, _ in plan)
@@ -388,6 +404,30 @@ def test_a_blurb_that_now_fits_further_is_written_but_is_not_news():
assert game["description"] == rsd.truncate_description(blurb)
GENRES = {"genres": [{"description": "Casual"}, {"description": "Free To Play"},
{"description": "Indie"}]}
def test_a_stray_genre_is_replaced_by_the_one_appdetails_gives():
game = _game("4156010", genre="Free To Play")
assert "genre" in rsd.apply_app_details(game, GENRES)
assert game["genre"] == "Casual"
def test_a_genre_a_person_could_have_chosen_is_never_touched():
game = _game("4156010", genre="Puzzle")
assert "genre" not in rsd.apply_app_details(game, GENRES)
assert game["genre"] == "Puzzle"
def test_a_stray_genre_stays_without_genres_or_under_an_override():
game = _game("4156010", genre="Free To Play")
assert "genre" not in rsd.apply_app_details(game, {"genres": []})
guard = rsd.OverrideGuard({"4156010": {"fields": {"genre": {"value": "Free To Play"}}}})
assert "genre" not in rsd.apply_app_details(game, GENRES, guard)
assert game["genre"] == "Free To Play"
def test_a_rewritten_blurb_is_a_change():
game = _game(description="Old words, cut short…")
assert "description" in rsd.apply_app_details(game, {"short_description": "New words entirely."})