feat(purge): remove a game only when two of Steam's answers agree

The purge asked appdetails about every game, 5.3 hours at 9,847, and one
answer was enough to remove one - appdetails has answered success:false
for live games in bursts (2026-09-27).

GetItems now screens the whole catalogue in a few minutes (unknown,
hidden, unlisted, coming soon, not is_free; never type), and
check_game_health() must confirm each flag; "unavailable" is asked again
five minutes later. A failed batch or a network error never reads as
gone, blocklisted games are not asked about, and more than 500 flags or
100 removals stops the run before anything is written. Removals are
logged as before. A notice names earlier removals Steam has since undone.

The dry run on the real catalogue: 8 flagged, 3 removed (the three
unlisted games), 5 kept because appdetails calls them free. Daily now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 committed 2026-10-23 22:21:59 +07:00
1 parent 49f0c56ed7
commit 74e1845fc4
4 files changed
+453 -43

No files matched your search

+27 -2
View File
@@ -1,20 +1,45 @@
name : Purge Unhealthy Games
# Takes out the games that left the store, stopped being free or are not out
# yet - only when two of Steam's answers agree: IStoreBrowseService/GetItems
# screens the whole catalogue in a hundred-odd requests, and appdetails
# confirms each game it flags (scripts/purge_unhealthy.py has the rules and
# the guards). A few minutes, so it runs daily. It replaces both the old
# purge (appdetails for every game, 5.3 hours) and Check Dead Links, a 5.5-hour
# HEAD sweep that never found anything: Steam sends a delisted app's page to
# the store front, which answers 200.
on :
schedule : [ { cron : '0 5 * * 1' } ]
schedule : [ { cron : '0 7 * * *' } ]
workflow_dispatch :
inputs :
dry_run :
description : 'Dry run: screen and confirm, remove nothing'
type : boolean
default : false
permissions : { contents : write }
# queue: max - a run that finds another data job running waits its turn
# (first in, first out) instead of evicting the run already waiting.
concurrency :
group : data-write
cancel-in-progress : false
queue : max
jobs :
purge :
runs-on : ubuntu-latest
timeout-minutes : 60
steps :
-
uses : actions/checkout@v6
# main as it is when the job STARTS: a run that waited its turn must
# work on what the job before it committed.
with : { ref : main }
-
uses : actions/setup-python@v6
with : { python-version : '3.15' }
-
run : chmod +x ./bash/purge.sh && ./bash/purge.sh
env : { STEAM_API_KEY : '${{ secrets.STEAM_API_KEY }}' }
env :
STEAM_API_KEY : '${{ secrets.STEAM_API_KEY }}'
JOB_BUDGET_MINUTES : '40'
PURGE_DRY_RUN : ${{ inputs.dry_run }}
+7
View File
@@ -1,5 +1,12 @@
#!/bin/bash
set -euo pipefail
pip install --quiet --require-hashes --only-binary :all: -r requirements.txt
# PURGE_DRY_RUN is the manual run's "dry run" box: screen and confirm, remove nothing.
# A guard that trips (too many flagged or confirmed) exits non-zero before
# anything is written, so set -e stops here and nothing is committed.
if [[ "${PURGE_DRY_RUN:-}" == "true" ]]; then
python scripts/purge_unhealthy.py --dry-run
exit 0
fi
python scripts/purge_unhealthy.py
bash "$(dirname "$0")/commit_push.sh" "Purge unhealthy [$(date +'%Y-%m-%d')]"
+205 -41
View File
@@ -1,48 +1,212 @@
#!/usr/bin/env python3
"""Purge unhealthy games."""
import sys, os, random, time
"""
Take out the games that left the store, stopped being free or are not out yet. Daily.
A game is removed only when two of Steam's answers agree:
1. the screen - IStoreBrowseService/GetItems for the whole catalogue, a
hundred-odd requests (core/store_items.screen_item): Steam does not know
the app, it is hidden or unlisted, coming soon, or has no is_free flag;
2. the confirmation - appdetails for each flagged game, through the same
check_game_health() the ingest uses. An "unavailable" (or 404) answer is
asked again RECHECK_AFTER seconds later: appdetails has answered
success:false for live games in bursts (14 of them on 2026-09-27).
This used to ask appdetails about every game, one at a time: 5.3 hours at
9,800 games, with one answer enough to remove one.
The rules:
- A batch GetItems did not answer flags nothing, and neither does a network
error at the confirmation: unknown is never "gone".
- Blocklisted games are left to save_main(), which drops them anyway, and
are not asked about.
- More than --max-candidates flagged, or more than --max-removals confirmed,
stops the run with an error BEFORE anything is written: that is a broken
answer from Steam, not a wave of delistings.
- Removals are logged to scripts/removed_games.jsonl as they always were
(reason and status_code from core/health_checker.py): the Worker's
reconcile reads that file.
- Games removed in the past that GetItems now sees free, listed and released
are listed in a ::notice::, so a removal Steam has since undone can be
brought back by queueing its link again.
Usage (from the repository root):
python scripts/purge_unhealthy.py
python scripts/purge_unhealthy.py --dry-run
python scripts/purge_unhealthy.py --max-removals 300 # a deliberate mass removal
"""
import argparse
import os
import sys
import time
from collections import Counter
from typing import Optional
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from core.constants import REMOVED_JSONL, BATCH_SIZE, BATCH_PAUSE_MIN, BATCH_PAUSE_MAX
from core.data_store import load_main, save_main, load_jsonl, save_jsonl, dedup_removed, now_iso
from core.blocklist import blocked_appids
from core.budget import Deadline, expired
from core.constants import REMOVED_JSONL
from core.data_store import (
dedup_removed, extract_appid, load_jsonl, load_main, now_iso, save_jsonl, save_main,
)
from core.health_checker import (
COMING_SOON, NOT_FOUND_404, NOT_FOUND_410, NOT_FREE, UNAVAILABLE, check_game_health,
)
from core.steam_client import get_client
from core.health_checker import check_game_health
from core.store_items import screen_item
def main():
MAX_CANDIDATES = 500
MAX_REMOVALS = 100
RECHECK_AFTER = 300 # seconds
SCREEN_REQUEST = {"include_release": True}
# appdetails answers worth asking twice: the ones a bad moment produces.
RECHECKED = frozenset({UNAVAILABLE, NOT_FOUND_404, NOT_FOUND_410})
# Removal reasons a game can come back from: what Steam says, not a person.
RECOVERABLE = frozenset({UNAVAILABLE, NOT_FOUND_404, NOT_FOUND_410, NOT_FREE, COMING_SOON})
def screen(games: list[dict], items: dict, failed: set, blocked: set) -> list[tuple[dict, str]]:
"""The games Steam's store flags, with why. A game without an appid is
flagged too: check_game_health() calls its link invalid without a request."""
out = []
for game in games:
appid = extract_appid(game.get("link", ""))
if not appid:
out.append((game, "invalid"))
elif appid not in blocked and appid not in failed:
why = screen_item(items.get(appid))
if why:
out.append((game, why))
return out
def confirm(candidates: list[tuple[dict, str]], client, deadline: Optional[Deadline] = None,
log=print) -> tuple[list, Counter]:
"""Ask appdetails about each flagged game. Returns ([(game, HealthResult)]
to remove, tally). An unavailable answer is asked again RECHECK_AFTER
seconds after the first ask; past the deadline nothing more is asked and
the rest stay."""
removals: list = []
recheck: list[dict] = []
tally: Counter = Counter()
first_ask = None
def judge(game: dict, result) -> None:
if result.should_remove:
removals.append((game, result))
elif result.is_healthy:
tally["kept"] += 1
log(f" ✓ kept {result.appid} {result.name[:40]}: appdetails says it is healthy")
else:
tally["network"] += 1
for game, why in candidates:
if expired(deadline):
tally["unasked"] += 1
continue
first_ask = time.monotonic() if first_ask is None else first_ask
result = check_game_health(game.get("link", ""), game.get("name", ""), client)
if result.status in RECHECKED:
recheck.append(game)
else:
judge(game, result)
if recheck:
wait = RECHECK_AFTER - (time.monotonic() - first_ask)
if wait > 0:
log(f" {len(recheck)} answered as unavailable; asking again in {wait:.0f} s")
time.sleep(wait)
for game in recheck:
if expired(deadline):
tally["unasked"] += 1
continue
judge(game, check_game_health(game.get("link", ""), game.get("name", ""), client))
return removals, tally
def recoverable(removed_rows: list[dict], live: set, blocked: set, client) -> list[str]:
"""Earlier removals Steam has since undone: "appid name" for each game
removed for a store reason that GetItems now sees free, listed and
released, and that is neither back in the catalogue nor blocklisted."""
names: dict[str, str] = {}
for row in removed_rows:
appid = str(row.get("appid") or extract_appid(row.get("link", "")) or "")
if appid and row.get("status_code") in RECOVERABLE and appid not in live | blocked:
names[appid] = (row.get("name") or "").strip()
if not names:
return []
items, failed = client.fetch_items(list(names), SCREEN_REQUEST)
return [f"{a} {names[a]}".strip() for a in names
if a not in failed and a in items and screen_item(items[a]) is None]
def main(argv: Optional[list[str]] = None) -> int:
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0].strip(),
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--dry-run", action="store_true", help="screen and confirm, write nothing")
ap.add_argument("--max-candidates", type=int, default=MAX_CANDIDATES,
help=f"stop when the screen flags more games than this (default {MAX_CANDIDATES})")
ap.add_argument("--max-removals", type=int, default=MAX_REMOVALS,
help=f"stop when more games than this are confirmed (default {MAX_REMOVALS})")
args = ap.parse_args(argv)
deadline = Deadline.from_env()
games = load_main()
if not games: print("Empty."); return
print(f"Scanning {len(games)} games...\n")
if not games:
print("Empty.")
return 0
client = get_client()
remove_set, log = set(), []
stats = {"ok": 0, "rm": 0, "net": 0}
total = len(games)
for bs in range(0, total, BATCH_SIZE):
be = min(bs + BATCH_SIZE, total)
print(f"── Batch {bs//BATCH_SIZE+1} ──")
for i in range(bs, be):
g = games[i]
name = g.get("name") or g.get("link", "?")
r = check_game_health(g.get("link",""), name, client)
if r.is_healthy:
stats["ok"] += 1
elif r.should_remove:
print(f" [{i+1}/{total}] ❌ {name[:50]} → {r.reason}")
remove_set.add(i)
log.append({"link":g.get("link",""),"name":r.name or name,
"appid":r.appid or "","reason":r.reason,
"status_code":r.status,"removed_at":now_iso()})
stats["rm"] += 1
else:
stats["net"] += 1
if be < total:
time.sleep(random.uniform(BATCH_PAUSE_MIN, BATCH_PAUSE_MAX))
if remove_set:
cleaned = [g for i,g in enumerate(games) if i not in remove_set]
save_main(cleaned)
existing = load_jsonl(REMOVED_JSONL)
save_jsonl(REMOVED_JSONL, dedup_removed(existing + log))
print(f"\nPurged {len(remove_set)}, remaining {len(cleaned)}")
else:
print("\nAll healthy.")
print(f"✅{stats['ok']} ❌{stats['rm']} ⚠{stats['net']}")
blocked = blocked_appids()
ids = [a for a in (extract_appid(g.get("link", "")) for g in games) if a and a not in blocked]
print(f"Screening {len(ids)} games with Steam's store API...")
items, failed = client.fetch_items(ids, SCREEN_REQUEST)
if ids and len(failed) == len(ids):
print("::warning::GetItems answered for no game: nothing screened, nothing removed.")
return 0
if failed:
print(f"::warning::GetItems did not answer for {len(failed)} games; they are screened again tomorrow.")
if __name__ == "__main__": main()
candidates = screen(games, items, failed, blocked)
flags = Counter(why for _, why in candidates)
print(f"{len(candidates)} flagged: " + (", ".join(f"{k} {v}" for k, v in flags.most_common()) or "none"))
if len(candidates) > args.max_candidates:
print(f"::error::The screen flagged {len(candidates)} games, more than --max-candidates "
f"{args.max_candidates}. That is a broken answer, not a wave of delistings: nothing "
f"was asked or written. Run with a higher --max-candidates if it is real.")
return 2
removals, tally = confirm(candidates, client, deadline)
for game, result in removals:
print(f" ❌ {result.appid} {(result.name or game.get('name', ''))[:50]} → {result.reason}")
print(f"Confirmed {len(removals)}; kept {tally['kept']} that appdetails calls healthy, "
f"{tally['network']} unanswered, {tally['unasked']} not asked (budget)")
if len(removals) > args.max_removals:
print(f"::error::{len(removals)} removals, more than --max-removals {args.max_removals}: "
f"nothing written. Run with a higher --max-removals if they are real.")
return 2
back = recoverable(load_jsonl(REMOVED_JSONL), set(ids), blocked, client)
if back:
print(f"::notice::{len(back)} removed game(s) are free and on the store again: "
f"{'; '.join(back[:20])}{' …' if len(back) > 20 else ''}. Queue a link in "
f"scripts/temp_info.jsonl to bring one back.")
if args.dry_run:
print("Dry run: nothing written.")
return 0
if not removals:
print("All healthy.")
return 0
gone = {id(game) for game, _ in removals}
save_main([g for g in games if id(g) not in gone])
log = [{"link": g.get("link", ""), "name": r.name or g.get("name", ""), "appid": r.appid or "",
"reason": r.reason, "status_code": r.status, "removed_at": now_iso()}
for g, r in removals]
save_jsonl(REMOVED_JSONL, dedup_removed(load_jsonl(REMOVED_JSONL) + log))
print(f"Purged {len(removals)}, remaining {len(games) - len(removals)}")
return 0
if __name__ == "__main__":
sys.exit(main())
+214
View File
@@ -0,0 +1,214 @@
"""
purge_unhealthy.py removes a game only when two of Steam's answers agree: the
store API's screen of the whole catalogue, and appdetails for each game it
flags. These pin the rules that make a daily, unattended removal safe: what
nobody answered is never "gone", appdetails' unavailable is asked twice,
blocklisted games are not asked about, a broken answer stops the run before
anything is written, and removals are logged as the Worker's reconcile reads
them.
Steam is never called: the client and appdetails are fakes. Run from the
repository root: python -m pytest scripts/tests
"""
import json
import os
import pytest
import purge_unhealthy as pu
from core import budget, data_store
from core.health_checker import (
NETWORK_ERROR, NOT_FREE, OK, UNAVAILABLE, HealthResult,
)
OLD = "2026-01-01T00:00:00Z"
FREE = {"visible": True, "is_free": True, "release": {"steam_release_date": 1}}
def _rec(appid: str, name: str = "") -> dict:
return {"link": f"https://store.steampowered.com/app/{appid}/", "name": name or f"Game {appid}",
"type_game": "online", "notes": "Not reviewed yet", "last_updated": OLD}
def _item(appid: str, **fields) -> dict:
return dict(FREE, id=int(appid), success=1, **fields)
class FakeClient:
"""GetItems items by appid (absent: unknown to Steam) and failed appids."""
def __init__(self, items=None, failed=()):
self.items, self.failed = items or {}, set(failed)
self.asked: list[list[str]] = []
def fetch_items(self, appids, data_request, give_up_after=10):
assert data_request == {"include_release": True}
self.asked.append(list(appids))
return ({a: self.items[a] for a in appids if a in self.items and a not in self.failed},
{a for a in appids if a in self.failed})
@pytest.fixture
def repo(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
os.makedirs("data")
os.makedirs("scripts")
monkeypatch.delenv(budget.ENV, raising=False)
sleeps: list[float] = []
monkeypatch.setattr(pu.time, "sleep", sleeps.append)
answers: dict[str, list[str]] = {}
health_calls: list[str] = []
def fake_health(link, name="", client=None):
appid = data_store.extract_appid(link)
health_calls.append(appid)
status = answers[appid].pop(0)
return HealthResult(status, appid, name, link)
monkeypatch.setattr(pu, "check_game_health", fake_health)
monkeypatch.setattr(pu, "blocked_appids", lambda: set())
class Repo:
def run(self, records=None, client=None, health=None, argv=()):
if records is not None:
data_store.save_main(records, apply_human_overrides=False)
answers.clear()
answers.update({a: list(v) for a, v in (health or {}).items()})
monkeypatch.setattr(pu, "get_client", lambda: client)
code = pu.main(list(argv))
return code, {data_store.extract_appid(g["link"]) for g in data_store.load_main()}
r = Repo()
r.sleeps, r.health_calls = sleeps, health_calls
r.block = lambda ids: monkeypatch.setattr(pu, "blocked_appids", lambda: set(ids))
return r
def _removed() -> list[dict]:
path = os.path.join("scripts", "removed_games.jsonl")
return [json.loads(line) for line in open(path, encoding="utf-8")] if os.path.exists(path) else []
def _bytes() -> dict:
files = [os.path.join("data", n) for n in sorted(os.listdir("data")) if n.endswith((".jsonl", ".json"))]
files += [os.path.join("scripts", n) for n in sorted(os.listdir("scripts"))]
return {f: open(f, "rb").read() for f in files}
def test_an_empty_catalogue_asks_nothing(repo):
client = FakeClient()
assert repo.run(None, client) == (0, set())
assert client.asked == []
def test_a_healthy_catalogue_asks_appdetails_nothing(repo):
client = FakeClient(items={"730": _item("730"), "570": _item("570")})
code, live = repo.run([_rec("730"), _rec("570")], client)
assert (code, live) == (0, {"730", "570"})
assert repo.health_calls == [] and _removed() == []
def test_a_game_both_answers_flag_is_removed_and_logged(repo):
client = FakeClient(items={"730": _item("730"), "4991340": _item("4991340", unlisted=True, is_free=False)})
code, live = repo.run([_rec("730"), _rec("4991340", "Might & Poison Mushroom")], client,
health={"4991340": [NOT_FREE]})
assert (code, live) == (0, {"730"})
[row] = _removed()
assert (row["appid"], row["status_code"], row["reason"]) == ("4991340", "not_free", "No longer free-to-play")
assert row["name"] == "Might & Poison Mushroom" and row["removed_at"]
def test_a_flag_appdetails_contradicts_is_kept(repo):
client = FakeClient(items={"2878670": _item("2878670", is_free=False)})
code, live = repo.run([_rec("2878670", "Spoop Troop")], client, health={"2878670": [OK]})
assert (code, live) == (0, {"2878670"}) and _removed() == []
def test_unavailable_is_asked_again_before_a_game_goes(repo):
client = FakeClient(items={}) # answered, and unknown to Steam
code, live = repo.run([_rec("10"), _rec("20")], client,
health={"10": [UNAVAILABLE, UNAVAILABLE], "20": [UNAVAILABLE, OK]})
assert code == 0 and live == {"20"}
assert repo.health_calls == ["10", "20", "10", "20"]
assert len(repo.sleeps) == 1 and 0 < repo.sleeps[0] <= pu.RECHECK_AFTER
assert [r["status_code"] for r in _removed()] == ["unavailable"]
def test_what_nobody_answered_is_never_gone(repo):
client = FakeClient(items={"730": _item("730")}, failed={"570"})
code, live = repo.run([_rec("730"), _rec("570"), _rec("440")], client,
health={"440": [NETWORK_ERROR]})
assert (code, live) == (0, {"730", "570", "440"})
assert repo.health_calls == ["440"] # 570's batch failed: not even asked
def test_no_answer_at_all_writes_nothing(repo):
repo.run([_rec("730")], FakeClient(items={"730": _item("730")}))
before = _bytes()
code, live = repo.run(None, FakeClient(failed={"730"}))
assert (code, live) == (0, {"730"}) and _bytes() == before
def test_a_blocklisted_game_is_not_asked_about(repo):
repo.block({"570"})
client = FakeClient(items={"730": _item("730")})
repo.run([_rec("730"), _rec("570")], client)
assert "570" not in client.asked[0] and repo.health_calls == []
def test_too_many_flags_stop_the_run_before_any_request(repo):
repo.run([_rec("730"), _rec("570"), _rec("440")], FakeClient(items={a: _item(a) for a in ("730", "570", "440")}))
before = _bytes()
code, _ = repo.run(None, FakeClient(items={}), argv=["--max-candidates", "2"])
assert code == 2 and repo.health_calls == [] and _bytes() == before
def test_too_many_removals_stop_the_run_before_anything_is_written(repo):
repo.run([_rec("730"), _rec("570")], FakeClient(items={"730": _item("730"), "570": _item("570")}))
before = _bytes()
client = FakeClient(items={"730": _item("730", is_free=False), "570": _item("570", is_free=False)})
code, live = repo.run(None, client, health={"730": [NOT_FREE], "570": [NOT_FREE]},
argv=["--max-removals", "1"])
assert code == 2 and live == {"730", "570"} and _bytes() == before
def test_a_dry_run_writes_nothing(repo):
repo.run([_rec("730")], FakeClient(items={"730": _item("730")}))
before = _bytes()
code, _ = repo.run(None, FakeClient(items={"730": _item("730", unlisted=True)}),
health={"730": [NOT_FREE]}, argv=["--dry-run"])
assert code == 0 and _bytes() == before
def test_a_removal_steam_has_undone_is_named(repo, capsys):
with open(os.path.join("scripts", "removed_games.jsonl"), "w", encoding="utf-8") as f:
for appid, code in (("4214820", "unavailable"), ("620", "not_free"), ("99", "admin_removed")):
f.write(json.dumps({"appid": appid, "name": f"Old {appid}", "status_code": code,
"removed_at": OLD}) + "\n")
client = FakeClient(items={"730": _item("730"), "4214820": _item("4214820"),
"620": _item("620", is_free=False), "99": _item("99")})
repo.run([_rec("730")], client)
out = capsys.readouterr().out
assert "::notice::1 removed game(s)" in out and "4214820 Old 4214820" in out
assert "620" not in out.split("::notice::")[1] and "Old 99" not in out
def test_the_budget_leaves_the_rest_unasked(repo, monkeypatch):
client = FakeClient(items={"730": _item("730", unlisted=True)})
monkeypatch.setenv(budget.ENV, "0.000001")
code, live = repo.run([_rec("730")], client, health={"730": [NOT_FREE]})
assert (code, live) == (0, {"730"}) and repo.health_calls == []
@pytest.mark.parametrize("item, why", [
(None, "unknown"),
({"visible": False, "is_free": True}, "hidden"),
({"visible": True, "unlisted": True}, "unlisted"),
({"visible": True, "is_free": True, "release": {"is_coming_soon": True}}, "coming_soon"),
({"visible": True}, "not_free"),
({"is_free": True, "type": 2}, None), # a free mod: type is not read
(FREE, None),
])
def test_what_the_screen_flags(item, why):
from core.store_items import screen_item
assert screen_item(item) == why