fix(pipeline): read anti-cheat from Steam's own disclosure; retire the extension's paths

None of the 4,704 games added since September had an anti-cheat. The only
code that read the anti-cheat a developer declares on Steam was the browser
extension (archived; last push 2026-06-25); the pipeline matched appdetails'
categories, which only ever say "Valve Anti-Cheat enabled".

- core/anticheat.py (new, pure): the store page's div.anticheat_section
  (its HEADING decides kernel, never the class; a Boot Protection row is a
  flag) and GetItems' extra_details.anticheat; labels from
  ANTI_CHEAT_PATTERNS, else Steam's name cut at 40 characters; same_family()
  ("EAC" = "Easy Anti-Cheat", "ACE" = "Anti-Cheat Expert (ACE)"); and
  apply_anti_cheat(), the one rule.
- fetch_full() fills, never replaces, from the page it already fetches (or
  GetItems for a gated page), else the VAC category, now found in any
  position (with a stored "" only the first category was checked).
- refresh_anticheat.py (new): GetItems for every game (~160 requests), a
  store page only for a disclosure without a name. Steam wins; an override
  wins over Steam and is not fought (OverrideGuard); no disclosure never
  clears; a failed batch is unknown. Runs at the end of the daily Refresh
  Store Data job with its own commit; only=anticheat runs it alone. That job
  now checks out main.
- The queue takes links only: merge_extension_data(), EXTENSION_FIELDS,
  ARRAY_FIELDS and the schema.ts mirrors go. translate_notes() becomes
  clean_notes(): the extension's leading "Has paid DLC" and "[hover-add]"
  are dropped (517 notes, once, never news).
- refetch_all.py keeps "-" instead of resetting it to ""; a retired override
  whose was equals its value no longer bumps last_updated on every save.
- "denuvo" joins Denuvo AC on both sides; anti-cheat.test.ts holds the TS
  patterns and enum to constants.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 committed 2026-10-16 09:38:39 +07:00
1 parent 9fc8be83ad
commit 3d0e22cd76
37 files changed
+1553 -296

No files matched your search

+12 -2
View File
@@ -9,6 +9,13 @@ name: Refresh Store Data
# scripts/refresh_store_data.py explains the rotation and the rules (never
# blanks a value, never touches MANUAL_FIELDS).
#
# Then the anti-cheat sweep, committed separately: every game's anti-cheat as
# its developer declares it on Steam, about 160 GetItems requests plus a store
# page for each disclosure Steam does not name there (~70), a few minutes.
# scripts/refresh_anticheat.py has its rules: Steam wins, an override wins
# over Steam, and no disclosure never clears a value. `only: anticheat` runs
# just the sweep.
#
# 16:00 UTC is the quiet part of this repository's day. Scheduled runs here
# start two to five hours late, the morning data jobs (reviews, dead links,
# purge) run for two to four hours, and the data-write group keeps only ONE
@@ -20,11 +27,11 @@ on:
workflow_dispatch:
inputs:
only:
description: 'Refresh one group only; leave empty for both'
description: 'One part only: a store group, or the anti-cheat sweep; leave empty for all'
required: false
default: ''
type: choice
options: ['', 'page', 'details']
options: ['', 'page', 'details', 'anticheat']
permissions: {contents: write}
@@ -37,7 +44,10 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
# main, not the triggering commit: a run that waited its turn in
# data-write must work on the data as it is now, not as it was queued.
- uses: actions/checkout@v6
with: {ref: main}
- uses: actions/setup-python@v6
with: {python-version: '3.12'}
- run: chmod +x ./bash/refresh_store.sh && ./bash/refresh_store.sh
+2 -2
View File
@@ -4,8 +4,8 @@
# A data job can run for hours (update-reviews, check-dead-links,
# purge-unhealthy), and `concurrency: data-write` serialises only the
# WORKFLOWS. main still moves underneath one: a merged pull request, an /admin
# approval or override the Worker commits, the extension's push to
# scripts/temp_info.jsonl. A plain `git push` is then rejected ("fetch first")
# approval, removal or override the Worker commits. A plain `git push` is then
# rejected ("fetch first")
# and the whole run is thrown away - purge-unhealthy lost four hours of work
# that way on 2026-09-21.
#
+12 -3
View File
@@ -1,6 +1,15 @@
#!/bin/bash
set -euo pipefail
pip install --quiet -r requirements.txt
# REFRESH_ONLY: "page" or "details" from a manual run; empty = both groups.
python scripts/refresh_store_data.py ${REFRESH_ONLY:+--only "$REFRESH_ONLY"}
bash "$(dirname "$0")/commit_push.sh" "Refresh store data [$(date +'%Y-%m-%d')]"
# REFRESH_ONLY from a manual run: "page" or "details" runs that group of the
# store refresh, "anticheat" only the anti-cheat sweep; empty = everything.
# Each part commits on its own, so a failure in the sweep never costs the
# store refresh its push (and set -e still stops at a crash in either).
if [ "${REFRESH_ONLY:-}" != "anticheat" ]; then
python scripts/refresh_store_data.py ${REFRESH_ONLY:+--only "$REFRESH_ONLY"}
bash "$(dirname "$0")/commit_push.sh" "Refresh store data [$(date +'%Y-%m-%d')]"
fi
if [ -z "${REFRESH_ONLY:-}" ] || [ "${REFRESH_ONLY:-}" = "anticheat" ]; then
python scripts/refresh_anticheat.py
bash "$(dirname "$0")/commit_push.sh" "Refresh anti-cheat [$(date +'%Y-%m-%d')]"
fi
+289
View File
@@ -0,0 +1,289 @@
"""
A game's anti-cheat as its developer declares it on Steam.
Steam asks a developer to say which anti-cheat a game uses and whether it runs
in the kernel, and shows the answer in two places (measured 2026-10-16):
GetItems IStoreBrowseService/GetItems with data_request.include_extra_details:
extra_details.anticheat = {kernel_mode, uninstall_completely,
boot_protection, boot_protection_name, name, name_loc}. False
booleans are left out, so {} is an anti-cheat outside the kernel;
no `anticheat` at all is no disclosure. Steam's own list of
products comes with a `name` (Delta Force: "Anti-Cheat Expert
(ACE)"); a developer's own wording mostly does not (PUBG: only
kernel_mode). It answers for the 'Adult Only' games whose store
page needs a login.
The page one div.anticheat_section in the Features block:
<div class="anticheat_section DRM_notice" >
<div>Uses Kernel Level Anti-Cheat</div>
<div class="anticheat_name">Easy Anti-Cheat<span class="anticheat_uninstalls"> - Requires manual removal after game uninstall</span></div>
</div>
The HEADING says whether it is kernel level ("Uses Kernel Level
Anti-Cheat" / "Uses Anti-Cheat Software"), never the class:
DRM_notice also marks an anti-cheat outside the kernel that needs
Secure Boot. A "Boot Protection" row is that requirement, not a
product. Names can be empty, free text, or end in a space.
No disclosure does not mean no anti-cheat - Warframe, Overwatch 2 and NARAKA
declare none - so nothing here ever clears a stored value.
apply_anti_cheat() is the one rule every writer uses: fetch_full() fills an
empty game, refresh_anticheat.py also replaces a value Steam contradicts, and
an override (core/overrides.py) beats both.
"""
import re
from dataclasses import dataclass
from html import unescape
from typing import Optional
from .constants import ANTI_CHEAT_PATTERNS
from .data_store import ELLIPSIS, extract_appid, is_empty
from .normalize import clean_text
FIELDS = ("anti_cheat", "anti_cheat_note", "is_kernel_ac")
@dataclass(frozen=True)
class Disclosure:
"""What Steam says. `name` is "" when Steam does not name the product;
`kernel` is None only for a page whose heading this module cannot read."""
name: str = ""
kernel: Optional[bool] = None
boot_protection: bool = False
# ──────────── The store page ────────────
# The heading, then the name rows that follow it, each with its own </div>.
_RE_SECTION = re.compile(
r'<div\s+class\s*=\s*"anticheat_section(?:\s[^"]*)?"[^>]*>\s*'
r'<div>([^<]*)</div>'
r'((?:\s*<div\s+class\s*=\s*"anticheat_name"[^>]*>.*?</div>)*)',
re.DOTALL | re.IGNORECASE,
)
_RE_NAME_ROW = re.compile(
r'<div\s+class\s*=\s*"anticheat_name"[^>]*>(.*?)</div>', re.DOTALL | re.IGNORECASE)
_RE_UNINSTALLS = re.compile(
r'<span\s+class\s*=\s*"anticheat_uninstalls"[^>]*>.*?</span>', re.DOTALL | re.IGNORECASE)
_RE_TAG = re.compile(r"<[^>]+>")
_HEADINGS = {
"uses kernel level anti-cheat": True,
"uses anti-cheat software": False,
}
BOOT_PROTECTION = "boot protection"
def _text(html: str) -> str:
return clean_text(unescape(_RE_TAG.sub("", html)))
def parse_page(html: Optional[str]) -> Optional[Disclosure]:
"""The disclosure on an English store page, or None when it has none."""
m = _RE_SECTION.search(html or "")
if not m:
return None
kernel = _HEADINGS.get(_text(m.group(1)).lower())
name, boot = "", False
for row in _RE_NAME_ROW.findall(m.group(2)):
text = _text(_RE_UNINSTALLS.sub("", row))
if text.lower() == BOOT_PROTECTION:
boot = True
elif text and not name:
name = text
return Disclosure(name=name, kernel=kernel, boot_protection=boot)
# ──────────── Steam's JSON ────────────
def from_store_item(item: Optional[dict]) -> Optional[Disclosure]:
"""The disclosure in one GetItems item, or None when it has none."""
details = (item or {}).get("extra_details")
ac = details.get("anticheat") if isinstance(details, dict) else None
if not isinstance(ac, dict):
return None
name = ac.get("name")
return Disclosure(
name=clean_text(name) if isinstance(name, str) else "",
kernel=bool(ac.get("kernel_mode")),
boot_protection=bool(ac.get("boot_protection")),
)
VAC = Disclosure(name="VAC (Valve Anti-Cheat)", kernel=False)
def from_categories(categories) -> Optional[Disclosure]:
"""VAC from appdetails' "Valve Anti-Cheat enabled" category (id 8), in any
position. Named as Steam names VAC in a disclosure, so a game that later
declares it ends with the same note. VAC never runs in the kernel."""
for cat in categories or []:
if not isinstance(cat, dict):
continue
desc = str(cat.get("description") or "").lower()
if cat.get("id") == 8 or any(p in desc for p in ANTI_CHEAT_PATTERNS["VAC"]):
return VAC
return None
def combine(api: Optional[Disclosure], page: Optional[Disclosure]) -> Optional[Disclosure]:
"""GetItems decides whether there is a disclosure and whether it is kernel
level; the page supplies the name GetItems leaves out."""
if api is None:
return page
if page is None or api.name:
return api
return Disclosure(name=page.name, kernel=api.kernel,
boot_protection=api.boot_protection or page.boot_protection)
# ──────────── Labels ────────────
def _alnum(text: str) -> str:
return "".join(ch for ch in str(text).casefold() if ch.isalnum())
_RE_ACRONYM = re.compile(r"\(([A-Za-z0-9]{2,6})\)")
def canonical(text: str) -> Optional[str]:
"""The ANTI_CHEAT_PATTERNS key a name belongs to: one of its patterns, or
the key's own name ("EAC", "KSS"). None for anything else."""
low = str(text).lower()
flat = _alnum(text)
for key, patterns in ANTI_CHEAT_PATTERNS.items():
if flat == _alnum(key) or any(p in low for p in patterns):
return key
return None
# Longest label. A developer can type a whole paragraph where the product name
# goes ("Frontier uses SCUE4 as its anti-cheat, which comes pre-built ...", 300
# characters), and the label is a badge on every list; the note keeps it all.
LABEL_MAX = 40
def label(name: str) -> str:
"""The stored `anti_cheat` for a name Steam shows: the short key for a
known product, else Steam's own name, cut on a word at LABEL_MAX."""
key = canonical(name)
if key:
return key
text = clean_text(name)
if len(text) <= LABEL_MAX:
return text
cut = text[:LABEL_MAX - 1]
if cut.rfind(" ") >= LABEL_MAX // 2:
cut = cut[:cut.rfind(" ")]
return cut.rstrip(" ,;:.-") + ELLIPSIS
def _aliases(text: str) -> set[str]:
out = {_alnum(text)}
out.update(m.lower() for m in _RE_ACRONYM.findall(str(text)))
return out - {""}
def same_family(a: str, b: str) -> bool:
"""Whether two anti-cheat values name the same product: "Easy Anti-Cheat"
and "EAC", "ACE" and "Anti-Cheat Expert (ACE)", "CrackProof®" and
"CrackProof"."""
if clean_text(str(a)).casefold() == clean_text(str(b)).casefold():
return True
ca, cb = canonical(a), canonical(b)
if ca or cb:
return ca == cb
fa, fb = _alnum(a), _alnum(b)
for text, short, full in ((a, fa, fb), (b, fb, fa)):
# A label label() cut short, and the name it was cut from.
if str(text).endswith(ELLIPSIS) and len(short) >= 12 and full.startswith(short):
return True
return bool(_aliases(a) & _aliases(b))
# ──────────── Notes ────────────
KERNEL, NON_KERNEL = "[Kernel]", "[Non-Kernel]"
def note_for(name: str, kernel: Optional[bool], boot_protection: bool = False) -> str:
"""The note the extension wrote and the 71 games before this module carry:
"Easy Anti-Cheat [Kernel]", plus "; Boot Protection" when Steam says so."""
note = name
if kernel is not None:
note = f"{name} {KERNEL if kernel else NON_KERNEL}"
return f"{note}; Boot Protection" if boot_protection else note
def _with_suffix(note, kernel: bool) -> Optional[str]:
"""The note with its kernel suffix set to `kernel`, or None when the note
carries no suffix (a sentence a person wrote is not ours to rewrite)."""
if not isinstance(note, str):
return None
for suffix in (KERNEL, NON_KERNEL):
idx = note.find(f" {suffix}")
if idx >= 0:
return note[:idx] + f" {KERNEL if kernel else NON_KERNEL}" + note[idx + 1 + len(suffix):]
return None
# ──────────── The rule ────────────
def apply_anti_cheat(game: dict, found: Optional[Disclosure], replace: bool = False,
guard=None) -> list[str]:
"""Bring one record's anti_cheat, anti_cheat_note and is_kernel_ac in line
with `found`, in place. Returns the fields whose value changed.
- Nothing found, or found without a name: nothing changes.
- No anti-cheat stored yet: all three are written.
- The same product: an empty note is filled and the kernel flag follows
Steam (only an empty one unless `replace`); a note that carries a
"[Kernel]"/"[Non-Kernel]" suffix gets the new one, other notes stay.
- Another product: all three are replaced, only when `replace`.
- `guard` (overrides.OverrideGuard) refuses writes an override would
undo. With the label locked to another product nothing is written; a
locked kernel flag decides the suffix of a note written here.
"""
if found is None or not found.name:
return []
appid = extract_appid(game.get("link", "")) or ""
def allowed(field: str, value) -> bool:
return guard is None or guard.allows(appid, field, value)
changed: list[str] = []
def put(field: str, value) -> None:
if game.get(field) != value and allowed(field, value):
game[field] = value
changed.append(field)
new_label = label(found.name)
stored = game.get("anti_cheat")
fresh = is_empty(stored)
other = not fresh and not (same_family(stored, found.name) or same_family(stored, new_label))
if other and not replace:
return []
if (fresh or other) and not allowed("anti_cheat", new_label):
return [] # an override decided the product ("-" included)
kernel_before = game.get("is_kernel_ac")
if found.kernel is not None and (fresh or other or replace or kernel_before is None):
put("is_kernel_ac", found.kernel)
# The note's suffix says what is STORED: Steam's flag, or the one an
# override locked (none at all for a locked null).
kernel_now = game.get("is_kernel_ac")
suffix = kernel_now if isinstance(kernel_now, bool) else None
note = game.get("anti_cheat_note")
if fresh or other:
put("anti_cheat", new_label)
if other or is_empty(note) or _with_suffix(note, True) is not None:
put("anti_cheat_note", note_for(found.name, suffix, found.boot_protection))
elif is_empty(note):
put("anti_cheat_note", note_for(found.name, suffix, found.boot_protection))
elif isinstance(suffix, bool):
swapped = _with_suffix(note, suffix)
if swapped is not None:
put("anti_cheat_note", swapped)
return changed
+2 -2
View File
@@ -5,8 +5,8 @@ data/blocklist.jsonl holds one line per blocked game. It is written by the
Worker (/admin/remove appends lines, Restore deletes one) and by
scripts/delete_game.py, and only READ here. Like data/overrides/, it is human
intent that every write re-imposes: save_main() drops a blocked record before
anything else happens to it, so no writer - the ingest of a link the browser
extension queued again, a full refetch, a purge - can put the game back. The
anything else happens to it, so no writer - the ingest of a link queued again,
a full refetch, a purge - can put the game back. The
only way back is to delete the line (Restore in /admin), which also queues the
link for ingest_new.py in the same commit.
+2 -12
View File
@@ -61,7 +61,8 @@ ANTI_CHEAT_PATTERNS: dict[str, list[str]] = {
"Ricochet": ["ricochet anti-cheat", "ricochet"],
"mHyprot": ["mhyprot", "mhyprot2"],
"FACEIT AC": ["faceit anti-cheat", "faceit ac"],
"Denuvo AC": ["denuvo anti-cheat"],
# Steam's own name for it, in a disclosure, is just "Denuvo".
"Denuvo AC": ["denuvo anti-cheat", "denuvo"],
"KSS": ["kss", "krafton security"],
"NetEase GS": ["netease game security"],
"Hyperion": ["hyperion", "byfron"],
@@ -91,17 +92,6 @@ MACHINE_NOTE_MARKERS = (
"💀 Delisted",
"⚠ No longer free!",
)
ARRAY_FIELDS = frozenset({
"platforms", "languages", "language_details", "tags",
"developer", "publisher",
})
EXTENSION_FIELDS = frozenset({
"name", "genre", "type_game", "has_paid_dlc",
"developer", "publisher", "release_date", "description",
"header_image", "anti_cheat", "anti_cheat_note", "is_kernel_ac",
"platforms", "languages", "language_details", "tags",
"notes", "safe",
})
TOP_ONLINE_LIMIT = 100
TOP_OFFLINE_LIMIT = 100
+9 -63
View File
@@ -5,7 +5,6 @@ Optimizations:
- _is_empty() simplified with early returns (no set lookup)
- save_jsonl() uses json.dumps directly (faster than jsonlines writer for large files)
- build_index() O(n) single pass
- merge_extension_data() avoids repeated key lookups
"""
import glob as _glob
import hashlib
@@ -18,7 +17,7 @@ from typing import Optional
from .constants import (
DATA_DIR, LEGACY_JSONL, TEMP_JSONL, REMOVED_JSONL,
MAX_RECORDS_PER_FILE, SHARD_PREFIX,
MANUAL_FIELDS, ARRAY_FIELDS, DESCRIPTION_MAX,
DESCRIPTION_MAX,
BLOCKLIST_MAX_DROP,
)
from .normalize import normalize_records
@@ -268,11 +267,11 @@ def save_main(records: list[dict], apply_human_overrides: bool = True,
"""Save records to sharded JSONL files (MAX_RECORDS_PER_FILE per shard).
Human overrides are re-imposed here, immediately before the write. This is
the single choke point for the dataset - all 14 data-writing scripts call
the single choke point for the dataset - every data-writing script calls
save_main() and nothing else writes data/data_*.jsonl - so applying them
here is what makes a correction survive refetch_all.py and
normalize_genres.py --apply without either script knowing this layer
exists. See core/overrides.py.
here is what makes a correction survive refetch_all.py,
refresh_anticheat.py and normalize_genres.py --apply without those scripts
knowing this layer exists. See core/overrides.py.
The blocklist (core/blocklist.py) is applied FIRST, the same way: a game
the maintainer removed is left out of every write, whichever script made
@@ -283,9 +282,9 @@ def save_main(records: list[dict], apply_human_overrides: bool = True,
un-overridden values (an audit report). Never pass it from the pipeline.
The representation rules of core/normalize.py run first - one release
date shape, no invisible characters, no HTML entities, no Vietnamese
notes - so every writer gets them, the browser extension's ingest
included. Like the truncation, they never touch last_updated.
date shape, no invisible characters, no HTML entities, none of the retired
browser extension's note phrases - so every writer gets them. Like the
truncation, they never touch last_updated.
"""
# Before everything else, so no work is spent on records that are not
# written, and a blocked game's override reports as an orphan (its file
@@ -493,7 +492,7 @@ def is_info_complete(game: dict) -> bool:
return True
# ──────────── Extension data merge ────────────
# ──────────── Migration ────────────
def _normalize_to_list(val) -> list[str]:
if isinstance(val, list):
@@ -503,54 +502,6 @@ def _normalize_to_list(val) -> list[str]:
return []
# Pre-compute sets for O(1) membership tests
# Keys from extension that should NEVER be stored in data.jsonl
_SKIP_MERGE_KEYS = frozenset({
"link", "appid", "added_at",
# Extension-only classification (not stored server-side)
"free_type", "is_free", "is_dlc", "is_demo", "is_playtest",
"price", "auto_notes",
})
_NORMALIZE_LIST_KEYS = frozenset({"developer", "publisher"})
_DESC_KEYS = frozenset({"description", "desc"})
def merge_extension_data(game: dict, ext: dict) -> dict:
"""
Merge extension data into game record.
Optimized: single pass over ext keys, minimal dict lookups.
"""
for key, val in ext.items():
if key in _SKIP_MERGE_KEYS or is_empty(val):
continue
if key in _NORMALIZE_LIST_KEYS:
val = _normalize_to_list(val)
if key in _DESC_KEYS:
target = "description"
if is_empty(game.get(target)):
game[target] = val
continue
if key in MANUAL_FIELDS:
if is_empty(game.get(key)):
game[key] = val
continue
if key in ARRAY_FIELDS:
game[key] = val
continue
# EXTENSION_FIELDS or unknown: overwrite
game[key] = val
# Strip any ephemeral fields that may have leaked through
for k in ("free_type", "is_free", "is_dlc", "is_demo", "is_playtest", "price", "auto_notes"):
game.pop(k, None)
return game
def migrate_record(game: dict) -> dict:
"""Migrate v2.0 record to v2.2. Idempotent."""
# Add missing keys
@@ -570,9 +521,4 @@ def migrate_record(game: dict) -> dict:
game["description"] = game["desc"]
del game["desc"]
# Remove deprecated/ephemeral fields
for k in ("free_type", "is_free", "is_dlc", "is_demo",
"is_playtest", "price", "auto_notes"):
game.pop(k, None)
return game
+28 -32
View File
@@ -11,13 +11,15 @@ import random
import time
from typing import Optional
from .anticheat import apply_anti_cheat, from_categories
from .constants import (
STEAM_API_KEY, SKIP_GENRE_TAGS, ANTI_CHEAT_PATTERNS,
STEAM_API_KEY, SKIP_GENRE_TAGS,
BATCH_SIZE, BATCH_PAUSE_MIN, BATCH_PAUSE_MAX,
)
from .steam_client import get_client
from .data_store import extract_appid, now_iso, is_info_complete, is_empty
from .normalize import decode_entities, normalize_release_date
from .overrides import OverrideGuard
from .peaks import raise_peak
from .scraper import scrape_store_page
from .store_items import paid_dlc_from_items, parse_store_item
@@ -97,23 +99,11 @@ def apply_details(game: dict, data: dict) -> dict:
if p.get("linux"): plats.append("Linux")
game["platforms"] = plats
# Anti-cheat from categories
ac = game.get("anti_cheat", "-")
if is_empty(ac) or ac == "-":
for cat in data.get("categories", []):
desc_lower = cat.get("description", "").lower()
for label, pats in ANTI_CHEAT_PATTERNS.items():
if any(p in desc_lower for p in pats):
game["anti_cheat"] = label
if is_empty(game.get("anti_cheat_note")):
game["anti_cheat_note"] = cat["description"]
break
if game["anti_cheat"] != "-":
break
# Anti-cheat is not read here: fetch_full() takes it from the developer's
# disclosure on the store page, and from these categories only for VAC.
# Infer type_game from categories when not manually set. Manual prefills
# win because MANUAL_FIELDS gates the merge in merge_extension_data(),
# and ingest_new.py re-applies overrides after fetch_full.
# Infer type_game from categories when not set. It is a manual field: only
# an empty one is inferred, and an override still wins in save_main().
if is_empty(game.get("type_game")):
inferred = _infer_type_game(data.get("categories", []))
if inferred:
@@ -220,6 +210,7 @@ def scrape_store_fields(client, appid: str, details: Optional[dict] = None) -> O
(core/store_items.py), and has_paid_dlc from the prices of the game's DLC.
A page missing its language table or its tags gets only the missing part
from the API. A failed request, or a 404, reads as nothing at all.
`anti_cheat` is the page's disclosure, or the API's when the API was asked.
`details` is the game's appdetails when the caller already has them; they
are fetched here only if a DLC check needs them.
@@ -244,19 +235,24 @@ def scrape_store_fields(client, appid: str, details: Optional[dict] = None) -> O
scraped["language_details"] = api["language_details"]
if not scraped["tags"]:
scraped["tags"] = api["tags"]
if scraped.get("anti_cheat") is None:
scraped["anti_cheat"] = api.get("anti_cheat")
return scraped
# ──────────── fetch_full ────────────
def fetch_full(game: dict, client=None, fetch_players=True,
scrape=True, prefetched_details=None) -> dict:
scrape=True, prefetched_details=None, guard=None) -> dict:
"""
Fetch all data for one game.
Args:
prefetched_details: If health_checker already fetched appdetails,
pass it here to avoid a second API call.
guard: an overrides.OverrideGuard. The anti-cheat fields are written
only where no override would undo them (an override may hold
is_kernel_ac at null, which reads as empty).
"""
c = client or get_client()
appid = extract_appid(game.get("link", ""))
@@ -281,24 +277,23 @@ def fetch_full(game: dict, client=None, fetch_players=True,
if count is not None:
apply_players(game, count)
# 4) Store page (languages + tags + DLC pricing) – single request, or
# Steam's JSON API when the page is gated (scrape_store_fields).
# Always scrape if has_paid_dlc hasn't been set by extension,
# or if any of languages/tags/language_details are missing.
needs_scrape = (
is_empty(game.get("languages")) or
is_empty(game.get("tags")) or
is_empty(game.get("language_details")) or
not game.get("_dlc_checked") # internal flag: DLC not yet verified from HTML
)
if scrape and needs_scrape:
# 4) Store page (languages + tags + DLC pricing + anti-cheat disclosure) -
# a single request, or Steam's JSON API when the page is gated
# (scrape_store_fields).
disclosure = None
if scrape:
scraped = scrape_store_fields(c, appid, details=data)
if scraped:
apply_scraped(game, scraped)
game["_dlc_checked"] = True
disclosure = scraped.get("anti_cheat")
# 5) Anti-cheat, fill only (refresh_anticheat.py is what replaces): what the
# developer declares on Steam, else appdetails' VAC category.
apply_anti_cheat(game, disclosure, guard=guard)
if is_empty(game.get("anti_cheat")):
apply_anti_cheat(game, from_categories((data or {}).get("categories")), guard=guard)
# Housekeeping
game.pop("_dlc_checked", None) # internal flag, don't persist
if not game.get("type_game"):
game["type_game"] = "offline"
if not game.get("notes", "").strip():
@@ -337,7 +332,8 @@ def update_all_full(games, force=False):
if not to_fetch:
print("All up to date!")
return
process_batch(to_fetch, lambda g: fetch_full(g, client=client), "Fetching")
guard = OverrideGuard.load()
process_batch(to_fetch, lambda g: fetch_full(g, client=client, guard=guard), "Fetching")
def update_reviews_only(games):
+20 -15
View File
@@ -213,24 +213,29 @@ def decode_tags(tags):
# ──────────── notes ────────────
# The browser extension (poli0981/steam-f2p-extension) records "this game has
# paid DLC" as Vietnamese text in `notes`. The catalogue is English - nine
# records already said "Has paid DLC" - and has_paid_dlc carries the fact.
_NOTE_TRANSLATIONS = (
("Có DLC trả phí", "Has paid DLC"),
)
# The retired browser extension (poli0981/steam-f2p-extension) began `notes`
# with two phrases a reader gains nothing from: "Has paid DLC" (written in
# Vietnamese, then translated in v4.2.1) - a fact has_paid_dlc carries and the
# weekly store refresh keeps true, where the note went stale: 10 contradicted
# the flag and 791 games with paid DLC never had it - and "[hover-add]", how
# the game was added. Measured 2026-10-16, both only ever came first.
_RE_NOTE_ARTEFACT = re.compile(r"^(?:Has paid DLC|\[hover-add\])(?=$|;|\s)\s*;?\s*")
NOT_REVIEWED = "Not reviewed yet"
def translate_notes(value):
"""notes with the extension's Vietnamese phrases in English; the rest of the
note ("; Easy Anti-Cheat [Kernel]", "💀 Dead game ...") stays as it is."""
def clean_notes(value):
"""notes without the extension's leading phrases. The rest of the note
("Easy Anti-Cheat [Kernel]", "💀 Dead game ...") stays as it is; a note
that was nothing else reads "Not reviewed yet", as fetch_full() writes it
for a game nobody has looked at. An empty note stays empty."""
if not isinstance(value, str):
return value
nfc = unicodedata.normalize("NFC", value)
out = nfc
for vi, en in _NOTE_TRANSLATIONS:
out = out.replace(vi, en)
return out if out != nfc else value
out = value
while (m := _RE_NOTE_ARTEFACT.match(out)) is not None:
out = out[m.end():]
if out == value:
return value
return out if out.strip() else NOT_REVIEWED
# ──────────── Records ────────────
@@ -265,7 +270,7 @@ def normalize_record(game: dict) -> list[str]:
hit.append("text")
if "release_date" in game and _set(game, "release_date", normalize_release_date(game["release_date"])):
hit.append("release_date")
if "notes" in game and _set(game, "notes", translate_notes(game["notes"])):
if "notes" in game and _set(game, "notes", clean_notes(game["notes"])):
hit.append("notes")
return hit
+55 -14
View File
@@ -4,22 +4,25 @@ Human edits as standing instructions, not one-time mutations.
The problem this solves
-----------------------
MANUAL_FIELDS (anti_cheat, anti_cheat_note, is_kernel_ac, notes, type_game,
safe, genre) are protected only by "fill if empty": merge_extension_data()
skips them when the existing value is non-empty. Nothing anywhere records which
value a human chose and which a scraper guessed, so:
safe, genre) are protected only by "fill if empty": apply_details() and
fetch_full() skip them when the existing value is non-empty. Nothing anywhere
records which value a human chose and which a scraper guessed, so:
* `normalize_genres.py --apply` rewrites `genre` for every game and cannot
tell a correction from scraper output;
* `refresh_anticheat.py` replaces the three anti-cheat fields with what
Steam's disclosure says;
* `refetch_all.py` re-fetches everything;
and a hand-made correction is reverted with no error and no trace.
An override is therefore NOT applied once. It lives permanently in Git as
`data/overrides/<appid>.json`, and `apply_overrides()` is called from inside
`save_main()` — the single choke point through which all 14 data-writing
scripts pass, and the only code that writes `data/data_*.jsonl`. Whatever a
script did to a record, the human's decision is re-imposed as the last act
before bytes hit the disk. No script needs to know this layer exists.
`save_main()` — the single choke point through which every data-writing script
passes, and the only code that writes `data/data_*.jsonl`. Whatever a script
did to a record, the human's decision is re-imposed as the last act before
bytes hit the disk. No script needs to know this layer exists; one that writes
manual fields daily asks OverrideGuard first, so it does not fight it.
Retiring an override
--------------------
@@ -36,7 +39,7 @@ import os
from typing import Optional
from .constants import MANUAL_FIELDS, MACHINE_NOTE_MARKERS, OVERRIDES_DIR
from .normalize import translate_notes
from .normalize import clean_notes
OVERRIDE_SCHEMA = 1
@@ -167,6 +170,42 @@ def validate_value(field: str, value) -> Optional[str]:
return None
class OverrideGuard:
"""Which writes to a manual field an override would undo.
A script that writes a manual field into a record an override governs gains
nothing - save_main() puts the human value back - and loses a lot: the
record's last_updated moves on every run, and so do the shard bytes. Two
cases, decided exactly as apply_overrides() decides them:
- an ACTIVE entry (an object whose value passes validate_value()) owns
its field, whatever is written there;
- a RETIRED entry restores `was` whenever the stored value equals its
`value`, so writing that value would flip-flop every day.
"""
def __init__(self, docs: dict[str, dict]):
self._active: dict[str, set[str]] = {}
self._retired: dict[str, dict[str, tuple]] = {}
for appid, doc in docs.items():
for field, entry in (doc.get("fields") or {}).items():
if isinstance(entry, dict) and validate_value(field, entry.get("value")) is None:
self._active.setdefault(appid, set()).add(field)
for field, entry in (doc.get("retired") or {}).items():
if isinstance(entry, dict) and field in MANUAL_FIELDS:
self._retired.setdefault(appid, {})[field] = (entry.get("value"), entry.get("was"))
@classmethod
def load(cls) -> "OverrideGuard":
return cls(load_overrides())
def allows(self, appid: str, field: str, value) -> bool:
if field in self._active.get(str(appid), ()):
return False
retired = self._retired.get(str(appid), {}).get(field)
return not (retired and retired[0] == value and retired[1] != value)
def apply_overrides(games: list[dict], quiet: bool = False) -> ApplyStats:
"""Re-impose every human decision onto `games`, in place.
@@ -212,11 +251,11 @@ def apply_overrides(games: list[dict], quiet: bool = False) -> ApplyStats:
continue
if field == "notes":
# Translated as save_main() translates the stored notes: an
# override still saying "Có DLC trả phí" would otherwise put it
# back after every normalisation, bump last_updated and commit
# on every run.
value = preserve_machine_notes(game.get("notes", ""), translate_notes(value))
# Cleaned as save_main() cleans the stored notes: an override
# still starting "Has paid DLC" would otherwise put it back
# after every normalisation, bump last_updated and commit on
# every run.
value = preserve_machine_notes(game.get("notes", ""), clean_notes(value))
stats.enforced += 1
if game.get(field) != value:
@@ -230,7 +269,9 @@ def apply_overrides(games: list[dict], quiet: bool = False) -> ApplyStats:
# Restore only while the stored value is still the one this
# override put there. Once restored the two differ, so every later
# pass is a no-op and the entry can stay in the file as history.
if game.get(field) == entry.get("value"):
# An entry whose `was` IS its value restores nothing, and must not
# count as a change: it would bump last_updated on every save.
if game.get(field) == entry.get("value") and entry.get("was") != entry.get("value"):
game[field] = entry.get("was")
stats.restored += 1
touched = True
+5 -1
View File
@@ -10,6 +10,8 @@ Text is HTML-unescaped before it is stored: the tag regex used to keep
import re
from html import unescape as _unescape
from .anticheat import parse_page
# ──── Pre-compiled patterns ────
_RE_LANG_TABLE = re.compile(
@@ -174,11 +176,13 @@ def parse_tags(html: str) -> list[str]:
# ──── Combined ────
def scrape_store_page(html: str) -> dict:
"""Parse languages + DLC pricing + tags from one HTML string."""
"""Parse languages + DLC pricing + tags + the anti-cheat disclosure
(core/anticheat.py; None when the page has none) from one HTML string."""
lang = parse_language_table(html)
return {
"languages": lang["languages"],
"language_details": lang["language_details"],
"has_paid_dlc": parse_has_paid_dlc(html),
"tags": parse_tags(html),
"anti_cheat": parse_page(html),
}
+59 -21
View File
@@ -216,38 +216,76 @@ class SteamClient:
_, html = self.fetch_store_page_full(appid)
return html
def _store_items_batch(self, ids: list[str], data_request: dict) -> Optional[list[dict]]:
"""One GetItems request: the items Steam knows (success 1), or None if
the request failed. An id Steam does not know comes back as success 15
with appid 0."""
request = {
"ids": [{"appid": int(a)} for a in ids],
"context": {"language": "english", "country_code": "US", "steam_realm": 1},
"data_request": data_request,
}
resp = self._get(
STORE_ITEMS_URL,
params={"input_json": json.dumps(request)},
throttle_fn=self._throttle_api,
)
if not resp or resp.status_code != 200:
return None
try:
items = resp.json()["response"].get("store_items") or []
except (ValueError, KeyError, AttributeError):
return None
return [i for i in items if isinstance(i, dict) and i.get("success") == 1 and i.get("id")]
def fetch_store_items(self, appids) -> Optional[dict[str, dict]]:
"""IStoreBrowseService/GetItems: {appid: item} for the ids Steam knows.
Keyless, and it answers for games whose store page needs a login. An id
Steam does not know comes back as success 15 with appid 0, so items are
keyed by the id that was asked for and kept only with success 1. None
if any request failed, so a partial answer is never read as a whole one.
Keyless, and it answers for games whose store page needs a login. Items
are keyed by the id that was asked for. Tags, languages and the
anti-cheat disclosure (extra_details, core/anticheat.py) are asked for.
None if any request failed, so a partial answer is never read as a
whole one.
"""
ids = [str(a) for a in appids if str(a).isdigit()]
out: dict[str, dict] = {}
request = {"include_tag_count": 20, "include_supported_languages": True,
"include_extra_details": True}
for start in range(0, len(ids), STORE_ITEMS_BATCH):
request = {
"ids": [{"appid": int(a)} for a in ids[start:start + STORE_ITEMS_BATCH]],
"context": {"language": "english", "country_code": "US", "steam_realm": 1},
"data_request": {"include_tag_count": 20, "include_supported_languages": True},
}
resp = self._get(
STORE_ITEMS_URL,
params={"input_json": json.dumps(request)},
throttle_fn=self._throttle_api,
)
if not resp or resp.status_code != 200:
return None
try:
items = resp.json()["response"].get("store_items") or []
except (ValueError, KeyError, AttributeError):
items = self._store_items_batch(ids[start:start + STORE_ITEMS_BATCH], request)
if items is None:
return None
for item in items:
if isinstance(item, dict) and item.get("success") == 1 and item.get("id"):
out[str(item["id"])] = item
out[str(item["id"])] = item
return out
def fetch_anticheat_items(self, appids, give_up_after: int = 10) -> tuple[dict[str, dict], set[str]]:
"""GetItems asking only for extra_details - each game's anti-cheat
disclosure - for a whole catalogue: ({appid: item}, ids not answered).
Unlike fetch_store_items() a failed batch costs only its own ids, which
the caller must treat as unknown, never as "no disclosure". After
`give_up_after` failed batches in a row the rest is not asked at all.
"""
ids = [str(a) for a in appids if str(a).isdigit()]
out: dict[str, dict] = {}
failed: set[str] = set()
in_a_row = 0
for start in range(0, len(ids), STORE_ITEMS_BATCH):
chunk = ids[start:start + STORE_ITEMS_BATCH]
if in_a_row >= give_up_after:
failed.update(chunk)
continue
items = self._store_items_batch(chunk, {"include_extra_details": True})
if items is None:
failed.update(chunk)
in_a_row += 1
continue
in_a_row = 0
for item in items:
out[str(item["id"])] = item
return out, failed
def fetch_tag_names(self) -> dict[int, str]:
"""{tagid: English name} from IStoreService/GetTagList, fetched once per
client. A failure is not cached, and returns {} (no tag is named)."""
+4
View File
@@ -20,6 +20,8 @@ Everything here is pure; steam_client.py makes the requests.
"""
from typing import Iterable, Optional
from .anticheat import from_store_item
# Steam's ELanguage ids, named the way the store page names them. Derived on
# 2026-10-10 by pairing GetItems' rows with stored HTML scrapes of the same games
# (row order is the same; one game lists all 102 languages). 29 never appeared.
@@ -83,6 +85,7 @@ def parse_store_item(item: dict, tag_names: dict[int, str]) -> dict:
what tells apply_scraped()/apply_store_page() to leave the stored value
alone. Rows follow the page's rules - a language with none of the three
columns ticked is not listed, a tag at most once (case-insensitively).
anti_cheat is the developer's disclosure (core/anticheat.py), None for none.
"""
languages, details = [], []
for row in item.get("supported_languages") or []:
@@ -113,6 +116,7 @@ def parse_store_item(item: dict, tag_names: dict[int, str]) -> dict:
"language_details": details,
"tags": tags,
"has_paid_dlc": None,
"anti_cheat": from_store_item(item),
}
+2 -2
View File
@@ -5,8 +5,8 @@ Remove games from the catalogue - the command-line twin of /admin/remove.
Removing a game is not a one-time deletion. It adds a line to
data/blocklist.jsonl, a standing instruction that save_main() re-imposes on
every write (core/blocklist.py): the game leaves data/, is logged once in
scripts/removed_games.jsonl, and nothing - the browser extension, a refetch,
the discovery sweep - can bring it back. Restoring is done in /admin (Remove ->
scripts/removed_games.jsonl, and nothing - a queued link, a refetch, the
discovery sweep - can bring it back. Restoring is done in /admin (Remove ->
Removed -> Restore), because it also has to clear the review queue's record.
python scripts/delete_game.py 730 --reason not_a_game
+20 -21
View File
@@ -9,14 +9,15 @@ import sys, os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from core.blocklist import blocked_appids
from core.constants import REMOVED_JSONL, MANUAL_FIELDS
from core.constants import REMOVED_JSONL
from core.data_store import (
load_main, save_main, load_temp, clear_temp,
load_jsonl, save_jsonl, dedup_removed,
normalize_link, extract_appid, build_index,
make_skeleton, merge_extension_data, now_iso, is_empty,
make_skeleton, now_iso,
)
from core.fetcher import fetch_full
from core.overrides import OverrideGuard
from core.steam_client import get_client
from core.health_checker import check_game_health, NETWORK_ERROR
@@ -34,12 +35,16 @@ def _link_text(value) -> str:
def parse_entries(records):
"""The queued links, as [{"link": text}]. A line is a store link and
nothing else: /admin writes {"link": url}, and any other key on a line -
the retired browser extension sent a whole record - is ignored, so a
queued line can never decide a field. The game comes from Steam."""
entries = []
for rec in records:
if isinstance(rec, (str, int)) and not isinstance(rec, bool):
entries.append({"link": _link_text(rec)})
elif isinstance(rec, dict) and rec.get("link") not in (None, ""):
entries.append({**rec, "link": _link_text(rec["link"])})
entries.append({"link": _link_text(rec["link"])})
return entries
@@ -60,6 +65,9 @@ def main():
added = skipped_dup = skipped_bad = skipped_net = skipped_blocked = 0
rejected = []
client = get_client()
# A restored game keeps its overrides: fetch_full() must not write a
# manual field one of them would undo.
guard = OverrideGuard.load()
for entry in entries:
link = normalize_link(entry.get("link", ""))
@@ -82,7 +90,7 @@ def main():
# Health check (also fetches appdetails → cached in result.data)
print(f" 🔍 {appid}...", end=" ")
health = check_game_health(link, entry.get("name", ""), client)
health = check_game_health(link, "", client)
if health.status == NETWORK_ERROR:
print("⚠ network – skip")
@@ -96,10 +104,10 @@ def main():
skipped_bad += 1
continue
# The catalogue is games. Discovery only proposes appdetails type
# "game", but this queue is also fed by the extension and - with
# auto-approve on - by candidates nobody looked at, so a free DLC, demo
# or tool is refused HERE, where Steam's own answer is in hand. Not in
# health_checker: Purge uses it, and would start deleting records.
# "game", but this queue also takes a link a pull request adds and -
# with auto-approve on - candidates nobody looked at, so a free DLC,
# demo or tool is refused HERE, where Steam's own answer is in hand.
# Not in health_checker: Purge uses it, and would start deleting records.
app_type = str((health.data or {}).get("type") or "").strip().lower()
if app_type and app_type != "game":
print(f"❌ not a game (Steam type: {app_type})")
@@ -110,21 +118,12 @@ def main():
continue
print("✅")
# Build record
# Build the record from Steam (reusing the health check's appdetails,
# so no second call). Human values arrive through overrides, which
# save_main() re-imposes.
game = make_skeleton(link)
merge_extension_data(game, entry)
# Save manual overrides
overrides = {k: game[k] for k in MANUAL_FIELDS
if not is_empty(game.get(k))}
# Fetch (reuse health check data → no second appdetails call)
print(f" ➕ Fetching ({appid})...")
fetch_full(game, client=client, prefetched_details=health.data)
# Re-apply overrides
for k, v in overrides.items():
game[k] = v
fetch_full(game, client=client, prefetched_details=health.data, guard=guard)
games.append(game)
index[appid] = len(games) - 1
+8 -19
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""
Force re-fetch ALL games. MANUAL ONLY.
Backup → migrate → clear fetchable + ephemeral → re-fetch → cleanup → save.
Backup → migrate → clear fetchable → re-fetch → save.
"""
import sys, os, shutil
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
@@ -9,6 +9,7 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from core.constants import DATA_DIR, STEAM_API_KEY
from core.data_store import load_main, save_main, migrate_record, now_iso
from core.fetcher import fetch_full, process_batch
from core.overrides import OverrideGuard
from core.steam_client import get_client
CLEARABLE = {
@@ -18,6 +19,9 @@ CLEARABLE = {
"drm_notes", "has_paid_dlc",
"platforms", "languages", "language_details", "tags",
}
# Manual fields holding a scraper default rather than a decision: reset to the
# first value (the skeleton's), so the re-fetch fills them again. anti_cheat
# stays "-": it used to become "", which nothing else ever writes.
CONDITIONAL = {
"genre": ("", "N/A", "Uncategorized"),
"anti_cheat": ("-",),
@@ -26,10 +30,6 @@ CONDITIONAL = {
}
_LIST_FIELDS = {"developer", "publisher", "platforms", "languages", "language_details", "tags"}
# Extension-only ephemeral fields that should never be in data.jsonl
_EPHEMERAL = ("free_type", "is_free", "is_dlc", "is_demo",
"is_playtest", "price", "auto_notes", "desc")
def clear_fetchable(game):
"""Reset API-fetchable fields to empty. Preserves manual fields."""
@@ -42,22 +42,12 @@ def clear_fetchable(game):
game[f] = ""
for f, defaults in CONDITIONAL.items():
if game.get(f) in defaults:
game[f] = defaults[0] if (isinstance(defaults[0], bool) or defaults[0] is None) else ""
game[f] = defaults[0]
game["reviews"] = "N/A"
game["current_players"] = "N/A"
game["peak_today"] = "N/A"
game["metacritic"] = "N/A"
# Remove ephemeral fields from old data
for k in _EPHEMERAL:
game.pop(k, None)
def cleanup_record(game):
"""Final cleanup before save – remove any ephemeral fields that leaked."""
for k in _EPHEMERAL:
game.pop(k, None)
def main():
games = load_main()
@@ -83,12 +73,11 @@ def main():
print("⚠ No STEAM_API_KEY – players skipped")
client = get_client()
process_batch(games, lambda g: fetch_full(g, client=client), "Re-fetching")
guard = OverrideGuard.load()
process_batch(games, lambda g: fetch_full(g, client=client, guard=guard), "Re-fetching")
# Final cleanup + timestamp
stamp = now_iso()
for g in games:
cleanup_record(g)
g["last_updated"] = stamp
save_main(games)
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env python3
"""
Refresh every game's anti-cheat from what its developer declares on Steam. Daily.
core/anticheat.py explains where Steam shows the disclosure and the rule that
applies it. Each run:
1. asks GetItems (IStoreBrowseService, keyless) for the disclosure of EVERY
game, 50 at a time: about 160 requests, a minute and a half;
2. reads the store page of a disclosed game only when GetItems leaves out its
name - a developer's own wording, which Steam keeps to the page (PUBG's
"KSS (Krafton Security Services)");
3. applies the rule with Steam winning: an empty game is filled, and a value
Steam contradicts (another product, the other kernel flag) is replaced.
The rules:
- No disclosure never clears anything: Warframe, Overwatch 2 and NARAKA use
an anti-cheat and declare none. A disclosure that names nothing, not even
on the page, changes nothing either, and is listed.
- An override wins (core/overrides.py) and is not fought: a field it covers
is not written, and neither is a retired entry's value, which
apply_overrides() would undo on every run.
- A batch GetItems did not answer leaves its games alone: unknown, not "no
disclosure". After ten failed batches in a row the rest is not asked and
the run ends with a warning, not a failure; with no answer at all nothing
is written.
- last_updated moves only for a game whose fields changed, so a run that
learns nothing writes byte-identical shards. Everything goes through
save_main().
It runs at the end of the daily Refresh Store Data job (bash/refresh_store.sh)
and is committed on its own; that workflow's `only: anticheat` runs just this.
Usage (from the repository root):
python scripts/refresh_anticheat.py every game
python scripts/refresh_anticheat.py --appid 2507950 578080
python scripts/refresh_anticheat.py --limit 50 --dry-run
"""
import argparse
import os
import sys
from collections import Counter
from typing import Optional
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from core.anticheat import FIELDS, apply_anti_cheat, combine, from_store_item, parse_page
from core.data_store import extract_appid, is_empty, load_main, migrate_record, now_iso, save_main
from core.normalize import normalize_records
from core.overrides import OverrideGuard
from core.steam_client import get_client
GIVE_UP_AFTER = 10 # failed GetItems batches in a row
def _kernel(value) -> str:
return {True: "kernel", False: "not kernel"}.get(value, "kernel unknown")
def sweep(targets: list[dict], client, guard, log=print) -> tuple[Counter, list[str]]:
"""Apply Steam's disclosures to `targets` in place. Returns a tally -
disclosed, none, unknown, pages, filled, replaced, updated, unnamed - and
the unnamed games, as "appid name"."""
by_id: dict[str, dict] = {}
for game in targets:
appid = extract_appid(game.get("link", ""))
if appid:
by_id[appid] = game
items, failed = client.fetch_anticheat_items(list(by_id), give_up_after=GIVE_UP_AFTER)
tally: Counter = Counter(unknown=len(failed))
unnamed: list[str] = []
for appid, game in by_id.items():
if appid in failed:
continue
api = from_store_item(items.get(appid))
if api is None:
tally["none"] += 1
continue
tally["disclosed"] += 1
found = api
if not api.name:
# GetItems keeps a developer's own wording to the page.
tally["pages"] += 1
status, page = client.fetch_store_page_full(appid)
found = combine(api, parse_page(page) if status == "ok" else None)
if not found.name:
tally["unnamed"] += 1
unnamed.append(f"{appid} {game.get('name', '')[:40]}")
continue
before = {f: game.get(f) for f in FIELDS}
changed = apply_anti_cheat(game, found, replace=True, guard=guard)
if not changed:
continue
game["last_updated"] = now_iso()
if is_empty(before["anti_cheat"]):
kind = "filled"
elif "anti_cheat" in changed:
kind = "replaced"
else:
kind = "updated"
tally[kind] += 1
log(f" {appid} {game.get('name', '')[:40]}: {before['anti_cheat'] or '-'} "
f"({_kernel(before['is_kernel_ac'])}) -> {game['anti_cheat']} "
f"({_kernel(game.get('is_kernel_ac'))}) [{', '.join(changed)}]")
return tally, unnamed
def main(argv: Optional[list[str]] = None) -> None:
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0].strip())
ap.add_argument("--appid", nargs="+", help="only these games")
ap.add_argument("--limit", type=int, help="at most this many games (for trying it out)")
ap.add_argument("--dry-run", action="store_true", help="ask Steam and report, write nothing")
args = ap.parse_args(argv)
games = load_main()
for g in games:
migrate_record(g)
# Compared as stored, like refresh_store_data.py.
normalize_records(games)
targets = [g for g in games if extract_appid(g.get("link", ""))]
if args.appid:
wanted = {str(a) for a in args.appid}
targets = [g for g in targets if extract_appid(g["link"]) in wanted]
if args.limit is not None:
targets = targets[: args.limit]
print(f"{len(games)} games; asking Steam about the anti-cheat of {len(targets)}")
if not targets:
return
tally, unnamed = sweep(targets, get_client(), OverrideGuard.load())
print(f"\nSteam: {tally['disclosed']} disclose an anti-cheat, {tally['none']} declare none, "
f"{tally['unknown']} unknown; {tally['pages']} store pages read")
if tally["unknown"]:
print(f"::warning::GetItems did not answer for {tally['unknown']} games; "
f"they keep their values until the next run.")
changed = tally["filled"] + tally["replaced"] + tally["updated"]
print(f"Anti-cheat: {changed} games changed (filled {tally['filled']}, "
f"replaced {tally['replaced']}, updated {tally['updated']})")
if unnamed:
print(f"{len(unnamed)} disclose an anti-cheat without naming it: {', '.join(unnamed)}")
if args.dry_run:
print("Dry run: nothing written.")
return
if tally["unknown"] == len(targets):
print("Steam answered for no game: nothing written.")
return
save_main(games)
print(f"✓ Saved {len(games)} games")
if __name__ == "__main__":
main()
+324
View File
@@ -0,0 +1,324 @@
"""
core/anticheat.py reads the anti-cheat a developer declares on Steam and is the
one rule every writer applies it with. These pin how the store page and Steam's
JSON are read (the heading decides kernel, a Boot Protection row is a flag, no
other notice is an anti-cheat), how products are compared, and the rule: fill,
replace only when asked, never clear, never fight an override, and a second
pass changes nothing.
Run from the repository root: python -m pytest scripts/tests
"""
import glob
import json
import os
import pytest
from core import anticheat as ac
from core.anticheat import Disclosure, apply_anti_cheat
from core.overrides import OverrideGuard
T = "\n\t\t\t\t\t\t\t\t\t\t"
def _section(cls: str, heading: str, *rows: tuple[str, str]) -> str:
"""The block as Steam wrote it on 2026-10-16 (Apex Legends, CoD), tabs and all."""
names = "".join(f'{T}<div class="anticheat_name">{name}<span class="anticheat_uninstalls">{note}</span></div>'
for name, note in rows)
return (f'<div class="DRM_notice">Requires 3rd-Party Account:&nbsp;EA Account</div>'
f'{T}<div class="anticheat_section {cls}" >{T}<div>{heading}</div>{names}{T}</div>'
f'{T}<div class="DRM_notice"data-panel="{{}}" role="button" >{T}'
f'<div>Requires agreement to a 3rd-party EULA</div></div>')
KERNEL = "Uses Kernel Level Anti-Cheat"
SOFTWARE = "Uses Anti-Cheat Software"
# ──────────── The store page ────────────
def test_a_kernel_section_and_its_uninstall_span():
page = _section("DRM_notice", KERNEL, ("Easy Anti-Cheat", " - Requires manual removal after game uninstall"))
assert ac.parse_page(page) == Disclosure("Easy Anti-Cheat", True, False)
def test_kernel_comes_from_the_heading_not_the_class():
# Jujutsu Kaisen: DRM_notice, yet "Uses Anti-Cheat Software" - its boot
# protection is what earns the class.
page = _section("DRM_notice", SOFTWARE, ("Fairguard", ""), ("Boot Protection", " - Requires both Secure Boot & TPM 2.0"))
assert ac.parse_page(page) == Disclosure("Fairguard", False, True)
page = _section("anticheat_nonkernel_notice", KERNEL, ("VAC (Valve Anti-Cheat)", ""))
assert ac.parse_page(page).kernel is True
page = _section("DRM_notice", "Dùng chống gian lận cấp độ kernel", ("Easy Anti-Cheat", ""))
assert ac.parse_page(page).kernel is None # a heading it cannot read
def test_boot_protection_is_a_flag_not_a_name():
page = _section("DRM_notice", KERNEL, ("RICOCHET Anti-Cheat", ""),
("Boot Protection", " - Requires both Secure Boot & TPM 2.0"))
assert ac.parse_page(page) == Disclosure("RICOCHET Anti-Cheat", True, True)
def test_names_are_cleaned_and_an_empty_one_stays_empty():
assert ac.parse_page(_section("DRM_notice", KERNEL, ("KSS (Krafton Security Services) ", ""))).name == \
"KSS (Krafton Security Services)"
assert ac.parse_page(_section("DRM_notice", KERNEL, ("Stand &amp; Deliver AC", ""))).name == "Stand & Deliver AC"
assert ac.parse_page(_section("anticheat_nonkernel_notice", SOFTWARE, ("", ""))) == Disclosure("", False, False)
def test_other_notices_prose_and_deck_json_are_not_anti_cheat():
page = ('<div class="DRM_notice"><div>Requires 3rd-Party Account: EA Account</div></div>'
'<div class="game_area_description">Our new anti-cheat technologies, kernel level.</div>'
'<script>{"SteamDeckVerified_TestResult_UnsupportedAntiCheat_Other": 1}</script>'
'<img src="anti_cheats_1202_ru.gif">')
assert ac.parse_page(page) is None
assert ac.parse_page("") is None and ac.parse_page(None) is None
# ──────────── Steam's JSON ────────────
def _item(anticheat=None, extra=True) -> dict:
item = {"id": 1, "success": 1}
if extra:
item["extra_details"] = {} if anticheat is None else {"anticheat": anticheat}
return item
def test_store_item_disclosures():
# Measured 2026-10-16.
delta = {"kernel_mode": True, "uninstall_completely": True,
"name": "Anti-Cheat Expert (ACE)", "name_loc": "#Store_AntiCheat_Type_ace"}
assert ac.from_store_item(_item(delta)) == Disclosure("Anti-Cheat Expert (ACE)", True, False)
assert ac.from_store_item(_item({"kernel_mode": True, "uninstall_completely": True})) == \
Disclosure("", True, False) # PUBG
assert ac.from_store_item(_item({})) == Disclosure("", False, False) # Marvel Rivals
assert ac.from_store_item(_item({"name": "VAC (Valve Anti-Cheat)"})) == \
Disclosure("VAC (Valve Anti-Cheat)", False, False) # CS2
cod = {"kernel_mode": True, "boot_protection": True, "boot_protection_name": "secureboot_tpm2"}
assert ac.from_store_item(_item(cod)) == Disclosure("", True, True)
assert ac.from_store_item(_item()) is None # Warframe
assert ac.from_store_item(_item(extra=False)) is None
assert ac.from_store_item(None) is None
def test_the_page_names_what_the_json_leaves_out():
api = Disclosure("", True, True)
page = Disclosure("RICOCHET Anti-Cheat", True, False)
assert ac.combine(api, page) == Disclosure("RICOCHET Anti-Cheat", True, True)
assert ac.combine(api, None) == api
assert ac.combine(None, page) == page
named = Disclosure("Easy Anti-Cheat", True, False)
assert ac.combine(named, Disclosure("", False, False)) == named # JSON decides kernel
def test_category_vac_in_any_position():
cats = [{"id": 1, "description": "Multi-player"}, {"id": 2, "description": "Single-player"},
{"id": 8, "description": "Valve Anti-Cheat enabled"}]
assert ac.from_categories(cats) == ac.VAC
assert ac.from_categories([{"description": "Valve Anti-Cheat enabled"}]) == ac.VAC
assert ac.from_categories([{"id": 1, "description": "Multi-player"}]) is None
assert ac.from_categories(None) is None
# ──────────── Labels ────────────
@pytest.mark.parametrize("name, label", [
("Easy Anti-Cheat", "EAC"),
("VAC (Valve Anti-Cheat)", "VAC"),
("VAC (Valve Anti-Cheat); additional custom server-side anti-cheat on dedicated servers", "VAC"),
("BattlEye", "BattlEye"),
("nProtect GameGuard", "nProtect"),
("XIGNCODE3", "XIGNCODE"),
("KSS (Krafton Security Services)", "KSS"),
("NetEase Game Security", "NetEase GS"),
("RICOCHET Anti-Cheat", "Ricochet"),
("Denuvo", "Denuvo AC"),
("FACEIT Anti-Cheat", "FACEIT AC"),
# Not a known product: Steam's name, as it is.
("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE)"),
("EA Javelin Anticheat", "EA Javelin Anticheat"),
("HoYoKProtect", "HoYoKProtect"),
])
def test_labels(name, label):
assert ac.label(name) == label
FRONTIER = ("Frontier uses SCUE4 as its anti-cheat, which comes pre-built with the game. It operates "
"at the user level and scans surface memory for unauthorized injections or cheat "
"applications affecting the game. It does not install a separate anti-cheat client.")
def test_a_paragraph_where_the_name_goes_is_cut_on_a_word():
# Frontier: Battle Royale, 2026-10-16. The note keeps all of it.
cut = ac.label(FRONTIER)
assert cut == "Frontier uses SCUE4 as its anti-cheat…"
assert len(cut) <= ac.LABEL_MAX
assert ac.label(cut) == cut
assert ac.same_family(cut, FRONTIER) and ac.same_family(FRONTIER, cut)
assert not ac.same_family("Frontier uses SCUE4 as its anti-cheat…", "Frontier uses EAC")
game = _game()
apply_anti_cheat(game, Disclosure(FRONTIER, False))
assert (game["anti_cheat"], game["anti_cheat_note"]) == (cut, f"{FRONTIER} [Non-Kernel]")
def test_products_are_compared_by_family():
assert ac.same_family("EAC", "Easy Anti-Cheat")
assert ac.same_family("Easy Anti-Cheat", "EAC")
assert ac.same_family("nProtect", "nProtect GameGuard")
assert ac.same_family("Krafton Security Services", "KSS (Krafton Security Services)")
assert ac.same_family("ACE", "Anti-Cheat Expert (ACE)")
assert ac.same_family("CrackProof®", "CrackProof")
assert ac.same_family("网易游戏安全", "网易游戏安全")
assert not ac.same_family("EAC", "Anti-Cheat Expert (ACE)")
assert not ac.same_family("EAC", "BattlEye")
assert not ac.same_family("FACEIT AC", "Anti-Cheat Expert (ACE)")
def _stored_anti_cheats() -> list[str]:
root = os.path.join(os.path.dirname(__file__), "..", "..", "data")
out = set()
for path in glob.glob(os.path.join(root, "data_*.jsonl")):
with open(path, encoding="utf-8") as fh:
for line in fh:
value = json.loads(line).get("anti_cheat") if line.strip() else None
if isinstance(value, str) and value.strip() not in ("", "-"):
out.add(value)
return sorted(out)
def test_every_stored_value_and_every_label_is_its_own_family():
# What makes a second sweep a no-op: a value the sweep wrote, or one already
# there, is the same product as the name it came from.
values = _stored_anti_cheats() + ["EAC", "Anti-Cheat Expert (ACE)", "Server-side", "-x-"]
for value in values:
assert ac.same_family(value, value), value
assert ac.same_family(ac.label(value), value), value
# ──────────── The rule ────────────
def _game(**fields) -> dict:
base = {"link": "https://store.steampowered.com/app/2507950/", "name": "Delta Force",
"anti_cheat": "-", "anti_cheat_note": "", "is_kernel_ac": None}
base.update(fields)
return base
ACE = Disclosure("Anti-Cheat Expert (ACE)", True, False)
def test_an_empty_game_gets_all_three():
game = _game()
assert sorted(apply_anti_cheat(game, ACE)) == ["anti_cheat", "anti_cheat_note", "is_kernel_ac"]
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == \
("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE) [Kernel]", True)
game = _game(anti_cheat="")
apply_anti_cheat(game, Disclosure("RICOCHET Anti-Cheat", True, True))
assert (game["anti_cheat"], game["anti_cheat_note"]) == ("Ricochet", "RICOCHET Anti-Cheat [Kernel]; Boot Protection")
def test_nothing_found_or_nothing_named_changes_nothing():
for found in (None, Disclosure("", True, False)):
for game in (_game(), _game(anti_cheat="EAC", is_kernel_ac=None)):
before = dict(game)
assert apply_anti_cheat(game, found, replace=True) == []
assert game == before
def test_the_same_product_fills_and_follows_steam():
# Marvel Rivals: named, kernel unknown, no note.
game = _game(anti_cheat="NetEase Game Security")
assert sorted(apply_anti_cheat(game, Disclosure("NetEase Game Security", False))) == \
["anti_cheat_note", "is_kernel_ac"]
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == \
("NetEase Game Security", "NetEase Game Security [Non-Kernel]", False)
# Filling never moves a known flag; the sweep (replace) does.
game = _game(anti_cheat="Fairguard", anti_cheat_note="Fairguard [Kernel]", is_kernel_ac=True)
assert apply_anti_cheat(game, Disclosure("Fairguard", False, True)) == []
assert sorted(apply_anti_cheat(game, Disclosure("Fairguard", False, True), replace=True)) == \
["anti_cheat_note", "is_kernel_ac"]
assert (game["anti_cheat_note"], game["is_kernel_ac"]) == ("Fairguard [Non-Kernel]", False)
def test_a_kernel_change_rewrites_only_a_machine_suffix():
game = _game(anti_cheat="EAC", anti_cheat_note="Kernel driver, uninstall it by hand", is_kernel_ac=False)
assert apply_anti_cheat(game, Disclosure("Easy Anti-Cheat", True), replace=True) == ["is_kernel_ac"]
assert game["anti_cheat_note"] == "Kernel driver, uninstall it by hand"
def test_another_product_is_replaced_only_by_the_sweep():
# Wuthering Waves: stored EAC, Steam says ACE.
game = _game(anti_cheat="EAC", anti_cheat_note="EAC", is_kernel_ac=None)
assert apply_anti_cheat(game, ACE) == []
assert game["anti_cheat"] == "EAC"
assert sorted(apply_anti_cheat(game, ACE, replace=True)) == ["anti_cheat", "anti_cheat_note", "is_kernel_ac"]
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == \
("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE) [Kernel]", True)
def test_a_known_label_stays_as_it_was_written():
game = _game(anti_cheat="Easy Anti-Cheat", anti_cheat_note="Easy Anti-Cheat [Kernel]", is_kernel_ac=True)
assert apply_anti_cheat(game, Disclosure("Easy Anti-Cheat", True), replace=True) == []
def test_applying_twice_changes_nothing():
for found in (ACE, Disclosure("RICOCHET Anti-Cheat", True, True), ac.VAC,
Disclosure("VAC (Valve Anti-Cheat); additional custom server-side anti-cheat", False),
Disclosure("hosted game servers", False), Disclosure("Fairguard", None),
Disclosure(FRONTIER, False), Disclosure("K1LLSWITↃH", False)):
for start in (_game(), _game(anti_cheat="EAC", anti_cheat_note="EAC", is_kernel_ac=True)):
game = dict(start)
apply_anti_cheat(game, found, replace=True)
once = dict(game)
assert apply_anti_cheat(game, found, replace=True) == [], found
assert game == once
# ──────────── Overrides ────────────
def _guard(fields=None, retired=None) -> OverrideGuard:
return OverrideGuard({"2507950": {"fields": fields or {}, "retired": retired or {}}})
def test_a_field_an_override_covers_is_not_written():
guard = _guard({"is_kernel_ac": {"value": None}})
game = _game()
assert sorted(apply_anti_cheat(game, ACE, guard=guard)) == ["anti_cheat", "anti_cheat_note"]
assert game["is_kernel_ac"] is None
assert game["anti_cheat_note"] == "Anti-Cheat Expert (ACE)" # the suffix follows the locked null
def test_a_locked_label_writes_nothing_for_another_product():
guard = _guard({"anti_cheat": {"value": "BattlEye"}})
game = _game(anti_cheat="BattlEye", is_kernel_ac=None)
assert apply_anti_cheat(game, ACE, replace=True, guard=guard) == []
guard = _guard({"anti_cheat": {"value": "-"}}) # an override says: none
assert apply_anti_cheat(_game(), ACE, replace=True, guard=guard) == []
# The same product with a locked label still gets its kernel flag and note.
guard = _guard({"anti_cheat": {"value": "Anti-Cheat Expert (ACE)"}})
game = _game(anti_cheat="Anti-Cheat Expert (ACE)")
assert sorted(apply_anti_cheat(game, ACE, replace=True, guard=guard)) == ["anti_cheat_note", "is_kernel_ac"]
def test_a_locked_kernel_flag_decides_the_suffix():
guard = _guard({"is_kernel_ac": {"value": False}})
game = _game(is_kernel_ac=False)
apply_anti_cheat(game, ACE, replace=True, guard=guard)
assert (game["is_kernel_ac"], game["anti_cheat_note"]) == (False, "Anti-Cheat Expert (ACE) [Non-Kernel]")
def test_a_retired_value_is_never_written_back():
# apply_overrides() restores `was` whenever the stored value equals the
# retired one: writing it would flip-flop every day.
guard = _guard(retired={"anti_cheat": {"value": "Anti-Cheat Expert (ACE)", "was": "-"}})
assert apply_anti_cheat(_game(), ACE, replace=True, guard=guard) == []
# Another value is fine, and so is an entry whose `was` is its value.
guard = _guard(retired={"anti_cheat": {"value": "EAC", "was": "-"},
"is_kernel_ac": {"value": True, "was": True}})
assert "anti_cheat" in apply_anti_cheat(_game(), ACE, replace=True, guard=guard)
def test_an_invalid_override_entry_guards_nothing():
# apply_overrides() rejects these, so they do not own the field either.
guard = _guard({"anti_cheat": {"value": ""}, "is_kernel_ac": "true"})
assert sorted(apply_anti_cheat(_game(), ACE, guard=guard)) == ["anti_cheat", "anti_cheat_note", "is_kernel_ac"]
+98
View File
@@ -0,0 +1,98 @@
"""
fetch_full() builds every new game (ingest_new.py) and refills incomplete ones
(update_data.py, refetch_all.py). These pin how it treats the anti-cheat
fields: filled from the developer's disclosure on the page it already fetched,
else from appdetails' VAC category, never replaced, and never where an
override would undo it.
Steam is never called: the client is a fake. Run from the repository root:
python -m pytest scripts/tests
"""
from core import fetcher
from core.data_store import make_skeleton
from core.overrides import OverrideGuard
LINK = "https://store.steampowered.com/app/2507950/"
DETAILS = {"name": "Delta Force", "type": "game", "is_free": True,
"categories": [{"id": 1, "description": "Multi-player"}]}
VAC_DETAILS = dict(DETAILS, categories=[{"id": 1, "description": "Multi-player"},
{"id": 8, "description": "Valve Anti-Cheat enabled"}])
def _page(name="Anti-Cheat Expert (ACE)", heading="Uses Kernel Level Anti-Cheat") -> str:
section = (f'<div class="anticheat_section DRM_notice" ><div>{heading}</div>'
f'<div class="anticheat_name">{name}<span class="anticheat_uninstalls"></span></div></div>')
return ('<table class="game_language_options"><tr><th>x</th></tr><tr><td>English</td>'
'<td><span>&#10004;</span></td><td></td><td></td></tr></table>'
'<a class="app_tag" href="#">FPS</a>' + section)
class FakeClient:
def __init__(self, page=None, status=None, items=None):
self.page = page
self.status = status or ("ok" if page is not None else "network_error")
self.items = items
def fetch_app_details(self, appid):
return dict(DETAILS)
def fetch_reviews(self, appid):
return None
def fetch_store_page_full(self, appid):
return (self.status, self.page if self.status == "ok" else None)
def fetch_store_items(self, appids):
return self.items
def fetch_tag_names(self):
return {}
def _fetch(game=None, client=None, details=DETAILS, guard=None) -> dict:
game = game or make_skeleton(LINK)
return fetcher.fetch_full(game, client=client or FakeClient(page=_page()), fetch_players=False,
prefetched_details=dict(details), guard=guard)
def test_fetch_full_fills_from_the_page():
game = _fetch()
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == \
("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE) [Kernel]", True)
def test_a_gated_game_is_filled_from_steam_s_json():
items = {"2507950": {"id": 2507950, "success": 1, "extra_details": {
"anticheat": {"kernel_mode": True, "name": "Easy Anti-Cheat"}}}}
game = _fetch(client=FakeClient(status="age_gate", items=items))
assert (game["anti_cheat"], game["is_kernel_ac"]) == ("EAC", True)
def test_vac_comes_from_the_category_only_when_steam_declares_nothing():
game = _fetch(client=FakeClient(page=_page().split('<div class="anticheat_section')[0]), details=VAC_DETAILS)
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == \
("VAC", "VAC (Valve Anti-Cheat) [Non-Kernel]", False)
# A disclosure wins over the category.
assert _fetch(details=VAC_DETAILS)["anti_cheat"] == "Anti-Cheat Expert (ACE)"
def test_fetch_full_never_replaces_another_product():
game = make_skeleton(LINK)
game.update(anti_cheat="EAC", anti_cheat_note="EAC", is_kernel_ac=False)
_fetch(game)
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("EAC", "EAC", False)
def test_apply_details_writes_no_anti_cheat():
game = make_skeleton(LINK)
fetcher.apply_details(game, VAC_DETAILS)
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("-", "", None)
def test_the_guard_reaches_fetch_full():
# A restored game keeps its override, here "nobody has verified the kernel
# flag": filling it would be undone by save_main() on every run.
guard = OverrideGuard({"2507950": {"fields": {"is_kernel_ac": {"value": None}}}})
game = _fetch(guard=guard)
assert game["anti_cheat"] == "Anti-Cheat Expert (ACE)"
assert game["is_kernel_ac"] is None
+23 -4
View File
@@ -1,7 +1,8 @@
"""
ingest_new.py reads scripts/temp_info.jsonl, which two producers append to (the
browser extension and /admin). These pin the lines that used to crash or
slip through: a link that is not text, and a game the maintainer removed.
ingest_new.py reads scripts/temp_info.jsonl, which /admin appends to (and a
pull request may). A line is a store link and nothing else. These pin the lines
that used to crash or slip through: a link that is not text, a line carrying
fields of its own, and a game the maintainer removed.
Steam is never called: check_game_health and fetch_full are replaced.
@@ -33,7 +34,7 @@ def repo(tmp_path, monkeypatch):
return HealthResult(OK, appid, f"Game {appid}", link,
data=details if details is not None else {"type": "game", "name": f"Game {appid}"})
def fetch(game, client=None, prefetched_details=None):
def fetch(game, client=None, prefetched_details=None, **_):
game["name"] = (prefetched_details or {}).get("name", game.get("name", ""))
monkeypatch.setattr(ingest_new, "check_game_health", health)
@@ -63,6 +64,24 @@ def test_parse_entries_turns_numbers_into_links_and_drops_nothing_silently():
]
def test_parse_entries_keeps_only_the_link():
line = {"link": "730", "genre": "MOBA", "anti_cheat": "VAC", "notes": "x", "name": "Fake",
"status": "removed", "last_updated": "2030-01-01T00:00:00Z"}
assert ingest_new.parse_entries([line]) == [{"link": "730"}]
def test_a_line_s_own_fields_never_reach_the_record(repo):
# The retired browser extension sent whole records; its merge let one line
# decide any field, a manual one or the record's status alike.
_queue({"link": "730", "genre": "MOBA", "safe": "y", "notes": "CS2 babyyy",
"status": "delisted", "is_dead": True})
ingest_new.main()
game = next(g for g in data_store.load_main() if g["link"].endswith("/730/"))
assert game["name"] == "Game 730"
assert game["genre"] == "" and game["safe"] == "?" and game["notes"] == ""
assert game["status"] == "active" and game["is_dead"] is False
def test_a_numeric_link_no_longer_crashes_the_run(repo):
_queue({"link": 730}, {"link": ["not", "a", "link"]})
ingest_new.main()
+61 -18
View File
@@ -13,8 +13,8 @@ import pytest
from core import data_store
from core.normalize import (
clean_list, clean_text, decode_entities, decode_tags, normalize_record,
normalize_release_date, translate_notes,
clean_list, clean_notes, clean_text, decode_entities, decode_tags, normalize_record,
normalize_release_date,
)
@@ -126,14 +126,33 @@ def test_tags_are_decoded_and_the_duplicates_that_makes_are_dropped():
# ──────────── notes ────────────
@pytest.mark.parametrize("raw, expected", [
("Có DLC trả phí", "Has paid DLC"),
("Có DLC trả phí 💀 Dead game (no players ≥14d)", "Has paid DLC 💀 Dead game (no players ≥14d)"),
("Có DLC trả phí; Easy Anti-Cheat [Kernel]", "Has paid DLC; Easy Anti-Cheat [Kernel]"),
("Not reviewed yet", "Not reviewed yet"),
("English only", "English only"),
# Every shape the 517 notes the retired browser extension began came in.
("Has paid DLC", "Not reviewed yet"),
("[hover-add]", "Not reviewed yet"),
("Has paid DLC 💀 Dead game (no players ≥14d)", "💀 Dead game (no players ≥14d)"),
("Has paid DLC; Easy Anti-Cheat [Kernel]", "Easy Anti-Cheat [Kernel]"),
("Has paid DLC; EAC", "EAC"),
("Has paid DLC; Anti-Cheat Toolkit [Non-Kernel] ⚠ No longer free!",
"Anti-Cheat Toolkit [Non-Kernel] ⚠ No longer free!"),
])
def test_the_extensions_vietnamese_notes_read_in_english(raw, expected):
assert translate_notes(raw) == expected
def test_the_extensions_leading_phrases_are_dropped(raw, expected):
assert clean_notes(raw) == expected
@pytest.mark.parametrize("raw", [
"", "Not reviewed yet", "English only", "Has paid DLCs", "Toxic max; Has paid DLC",
"has paid dlc", " Has paid DLC", "[hover-add]er", None,
])
def test_anything_else_is_kept_exactly(raw):
# An empty note stays empty: only a note the rule emptied becomes
# "Not reviewed yet". A phrase in the middle is a person's sentence.
assert clean_notes(raw) == raw
def test_clean_notes_is_a_fixed_point():
for raw in ("Has paid DLC", "[hover-add]", "Has paid DLC; EAC", "Has paid DLC 💀 Dead game"):
once = clean_notes(raw)
assert clean_notes(once) == once
# ──────────── Records ────────────
@@ -153,7 +172,7 @@ def _messy(appid: int) -> dict:
"release_date": "28 апр. 2025 г.",
"description": "an art &quot;game&quot;",
"tags": ["Point &amp; Click"],
"notes": "Có DLC trả phí",
"notes": "Has paid DLC; EAC",
"last_updated": "2026-01-01T00:00:00Z",
}
@@ -180,7 +199,7 @@ def test_save_main_writes_every_record_normalised_and_none_as_news(workdir):
assert stored["release_date"] == "Apr 28, 2025"
assert stored["description"] == 'an art "game"'
assert stored["tags"] == ["Point & Click"]
assert stored["notes"] == "Has paid DLC"
assert stored["notes"] == "EAC"
assert stored["last_updated"] == "2026-01-01T00:00:00Z"
@@ -191,16 +210,40 @@ def test_saving_what_was_loaded_changes_no_byte(workdir):
assert _shards() == before
def test_an_override_in_vietnamese_does_not_flip_flop(workdir):
def _override(appid: int, doc: dict) -> None:
os.makedirs(os.path.join("data", "overrides"), exist_ok=True)
with open(os.path.join("data", "overrides", f"{appid}.json"), "w", encoding="utf-8") as fh:
json.dump({"schema": 1, "appid": str(appid), **doc}, fh)
def test_an_override_with_the_phrase_does_not_flip_flop(workdir):
# The override would put the phrase back after every normalisation, bump
# last_updated and commit on every run, if it were not translated too.
os.makedirs(os.path.join("data", "overrides"))
with open(os.path.join("data", "overrides", "1.json"), "w", encoding="utf-8") as fh:
json.dump({"schema": 1, "appid": "1", "fields": {
"notes": {"value": "Có DLC trả phí; EAC", "was": "Not reviewed yet"}}}, fh)
# last_updated and commit on every run, if it were not cleaned too.
_override(1, {"fields": {"notes": {"value": "Has paid DLC; EAC", "was": "Not reviewed yet"}}})
data_store.save_main([_messy(1)])
before = _shards()
[stored] = data_store.load_main()
assert stored["notes"] == "Has paid DLC; EAC"
assert stored["notes"] == "EAC"
data_store.save_main(data_store.load_main())
assert _shards() == before
def test_an_override_of_only_the_phrase_keeps_the_marker(workdir):
game = dict(_messy(1), notes="Has paid DLC 💀 Dead game (no players ≥14d)")
_override(1, {"fields": {"notes": {"value": "Has paid DLC", "was": ""}}})
data_store.save_main([game])
before = _shards()
[stored] = data_store.load_main()
assert stored["notes"] == "Not reviewed yet 💀 Dead game (no players ≥14d)"
data_store.save_main(data_store.load_main())
assert _shards() == before
def test_a_retired_entry_that_restores_nothing_is_not_news(workdir):
# was == value: putting `was` back changes nothing, and used to bump
# last_updated on every save all the same.
game = dict(_messy(1), genre="FPS")
_override(1, {"fields": {}, "retired": {"genre": {"value": "FPS", "was": "FPS"}}})
data_store.save_main([game])
[stored] = data_store.load_main()
assert stored["last_updated"] == "2026-01-01T00:00:00Z"
+21
View File
@@ -0,0 +1,21 @@
"""
refetch_all.py clears what Steam can give back and keeps what a person
decided. Run from the repository root: python -m pytest scripts/tests
"""
import refetch_all
def test_clear_fetchable_resets_a_default_to_the_skeleton_s_value():
# "-" used to become "", a value nothing else writes, which the old VAC
# detection then checked against only one category.
game = {"anti_cheat": "-", "anti_cheat_note": "", "is_kernel_ac": None, "genre": "Uncategorized"}
refetch_all.clear_fetchable(game)
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("-", "", None)
assert game["genre"] == ""
def test_clear_fetchable_keeps_a_decision():
game = {"anti_cheat": "BattlEye", "anti_cheat_note": "BattlEye [Kernel]", "is_kernel_ac": True, "genre": "FPS"}
refetch_all.clear_fetchable(game)
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"], game["genre"]) == \
("BattlEye", "BattlEye [Kernel]", True, "FPS")
+219
View File
@@ -0,0 +1,219 @@
"""
refresh_anticheat.py re-reads every game's anti-cheat disclosure from Steam
daily and lets Steam win. These pin what makes a daily writer of manual fields
safe: no disclosure and no answer never clear anything, an override is never
fought, last_updated moves only on a real change, and a second run writes the
same bytes.
Steam is never called: the client is a fake. Run from the repository root:
python -m pytest scripts/tests
"""
import json
import os
import pytest
import refresh_anticheat as rac
from core import data_store
STAMP = "2026-10-16T16:30:00Z"
OLD = "2026-01-01T00:00:00Z"
def _rec(appid: str, name: str, **fields) -> dict:
base = {"link": f"https://store.steampowered.com/app/{appid}/", "name": name,
"anti_cheat": "-", "anti_cheat_note": "", "is_kernel_ac": None,
"type_game": "online", "notes": "Not reviewed yet", "last_updated": OLD}
base.update(fields)
return base
def _page(name: str, kernel: bool) -> str:
heading = "Uses Kernel Level Anti-Cheat" if kernel else "Uses Anti-Cheat Software"
cls = "DRM_notice" if kernel else "anticheat_nonkernel_notice"
return (f'<div class="anticheat_section {cls}" >\n\t<div>{heading}</div>\n\t'
f'<div class="anticheat_name">{name}<span class="anticheat_uninstalls"></span></div>\n</div>')
class FakeClient:
"""GetItems' extra_details.anticheat by appid (None: no disclosure), store
pages by appid, and the appids whose batch failed."""
def __init__(self, items=None, pages=None, failed=()):
self.items, self.pages, self.failed = items or {}, pages or {}, set(failed)
self.calls: list[tuple] = []
def fetch_anticheat_items(self, appids, give_up_after=10):
self.calls.append(("items", tuple(appids)))
out = {}
for a in appids:
if a not in self.failed:
found = self.items.get(a)
out[a] = {"id": int(a), "success": 1,
"extra_details": {} if found is None else {"anticheat": found}}
return out, {a for a in appids if a in self.failed}
def fetch_store_page_full(self, appid):
self.calls.append(("page", appid))
html = self.pages.get(appid)
return ("ok", html) if html is not None else ("network_error", None)
@pytest.fixture
def repo(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
os.makedirs("data")
monkeypatch.setattr(rac, "now_iso", lambda: STAMP)
def run(records=None, client=None, argv=()):
if records is not None:
data_store.save_main(records, apply_human_overrides=False)
monkeypatch.setattr(rac, "get_client", lambda: client)
rac.main(list(argv))
return {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
return run
def _bytes() -> dict:
return {name: open(os.path.join("data", name), "rb").read()
for name in sorted(os.listdir("data")) if name.endswith((".jsonl", ".json"))}
def _override(appid: str, doc: dict) -> None:
os.makedirs(os.path.join("data", "overrides"), exist_ok=True)
with open(os.path.join("data", "overrides", f"{appid}.json"), "w", encoding="utf-8") as fh:
json.dump({"schema": 1, "appid": appid, **doc}, fh)
# What Steam answered on 2026-10-16.
STEAM = FakeClient(
items={
"2507950": {"kernel_mode": True, "uninstall_completely": True, "name": "Anti-Cheat Expert (ACE)"},
"1203220": None, # NARAKA: declares none
"578080": {"kernel_mode": True, "uninstall_completely": True},
"2767030": {},
"3513350": {"kernel_mode": True, "name": "Anti-Cheat Expert (ACE)"},
"4224240": {"boot_protection": True, "boot_protection_name": "secureboot_tpm2"},
"1252250": {},
},
pages={
"578080": _page("KSS (Krafton Security Services) ", True),
"2767030": _page("NetEase Game Security", False),
"4224240": _page("Fairguard", False).replace(
"</div>\n</div>", '</div>\n\t<div class="anticheat_name">Boot Protection'
'<span class="anticheat_uninstalls"> - Requires both Secure Boot &amp; TPM 2.0'
'</span></div>\n</div>'),
"1252250": _page("", False),
},
)
CATALOGUE = [
_rec("2507950", "Delta Force"),
_rec("1203220", "NARAKA: BLADEPOINT", anti_cheat="EAC", anti_cheat_note="EAC"),
_rec("578080", "PUBG: BATTLEGROUNDS", anti_cheat="Krafton Security Services",
anti_cheat_note="Krafton Security Services [Kernel]", is_kernel_ac=True),
_rec("2767030", "Marvel Rivals", anti_cheat="NetEase Game Security"),
_rec("3513350", "Wuthering Waves", anti_cheat="EAC", anti_cheat_note="Easy Anti-Cheat [Kernel]", is_kernel_ac=True),
_rec("4224240", "Jujutsu Kaisen Phantom Parade", anti_cheat="Fairguard",
anti_cheat_note="Fairguard [Kernel]", is_kernel_ac=True),
_rec("1252250", "Rebornia"),
_rec("230410", "Warframe", anti_cheat=""),
]
def test_conflicts_resolve_steam_first(repo, capsys):
games = repo([dict(g) for g in CATALOGUE], STEAM)
trio = {a: (g["anti_cheat"], g["anti_cheat_note"], g["is_kernel_ac"]) for a, g in games.items()}
assert trio["2507950"] == ("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE) [Kernel]", True)
assert trio["3513350"] == ("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE) [Kernel]", True)
assert trio["4224240"] == ("Fairguard", "Fairguard [Non-Kernel]", False)
assert trio["2767030"] == ("NetEase Game Security", "NetEase Game Security [Non-Kernel]", False)
# The same product under the name the extension wrote: nothing to do.
assert trio["578080"] == ("Krafton Security Services", "Krafton Security Services [Kernel]", True)
out = capsys.readouterr().out
assert "4 games changed (filled 1, replaced 1, updated 2)" in out
assert "1252250 Rebornia" in out
def test_no_disclosure_and_no_name_never_clear_anything(repo):
games = repo([dict(g) for g in CATALOGUE], STEAM)
assert games["1203220"]["anti_cheat"] == "EAC" # NARAKA declares none
assert games["230410"]["anti_cheat"] == "" # Warframe: not even asked about by name
assert games["1252250"]["anti_cheat"] == "-" # disclosed, named nowhere
def test_last_updated_moves_only_on_change(repo):
games = repo([dict(g) for g in CATALOGUE], STEAM)
moved = sorted(a for a, g in games.items() if g["last_updated"] == STAMP)
assert moved == ["2507950", "2767030", "3513350", "4224240"]
def test_a_second_run_changes_no_byte(repo):
repo([dict(g) for g in CATALOGUE], STEAM)
before = _bytes()
repo(client=STEAM)
assert _bytes() == before
def test_the_page_is_read_only_for_unnamed_disclosures(repo):
client = FakeClient(STEAM.items, STEAM.pages)
repo([dict(g) for g in CATALOGUE], client)
pages = sorted(a for kind, a in client.calls if kind == "page")
assert pages == ["1252250", "2767030", "4224240", "578080"]
assert [kind for kind, _ in client.calls].count("items") == 1
def test_a_failed_batch_is_unknown_and_reported(repo, capsys):
client = FakeClient(STEAM.items, STEAM.pages, failed={"2507950", "3513350"})
games = repo([dict(g) for g in CATALOGUE], client)
assert games["2507950"]["anti_cheat"] == "-"
assert games["3513350"]["anti_cheat"] == "EAC"
assert games["2767030"]["is_kernel_ac"] is False # the rest still applies
assert "::warning::GetItems did not answer for 2 games" in capsys.readouterr().out
def test_steam_down_saves_nothing_and_exits_zero(repo, capsys):
# Notes a save would clean, so a save of any kind would show in the bytes.
records = [dict(g, notes="Has paid DLC") for g in CATALOGUE]
data_store.save_main(records, apply_human_overrides=False)
with open(os.path.join("data", "data_001.jsonl"), "w", encoding="utf-8", newline="\n") as fh:
for r in records:
fh.write(json.dumps(r, ensure_ascii=False) + "\n")
before = _bytes()
repo(client=FakeClient(failed={data_store.extract_appid(g["link"]) for g in CATALOGUE}))
assert _bytes() == before
assert "Steam answered for no game: nothing written." in capsys.readouterr().out
def test_an_active_override_is_not_fought(repo):
# A human said Wuthering Waves uses EAC; Steam says ACE every day.
_override("3513350", {"fields": {"anti_cheat": {"value": "EAC"}}})
games = repo([dict(g) for g in CATALOGUE], STEAM)
assert games["3513350"]["anti_cheat"] == "EAC"
assert games["3513350"]["last_updated"] == OLD
before = _bytes()
repo(client=STEAM)
assert _bytes() == before
def test_a_retired_value_is_not_written_back(repo):
# Retired: the human's ACE gave way to the old "-". Writing ACE again would
# make apply_overrides() restore "-" on every save - a commit a day.
_override("2507950", {"fields": {}, "retired": {"anti_cheat": {"value": "Anti-Cheat Expert (ACE)", "was": "-"}}})
games = repo([dict(g) for g in CATALOGUE], STEAM)
assert games["2507950"]["anti_cheat"] == "-"
before = _bytes()
repo(client=STEAM)
assert _bytes() == before
def test_dry_run_writes_nothing_and_appid_narrows(repo, capsys):
repo([dict(g) for g in CATALOGUE], FakeClient())
before = _bytes()
client = FakeClient(STEAM.items, STEAM.pages)
repo(client=client, argv=["--dry-run", "--appid", "2507950", "3513350"])
assert _bytes() == before
[(_, asked)] = [c for c in client.calls if c[0] == "items"]
assert sorted(asked) == ["2507950", "3513350"]
out = capsys.readouterr().out
assert "2 games changed" in out and "Dry run: nothing written." in out
+13
View File
@@ -176,6 +176,19 @@ def test_description_is_compared_as_stored():
assert "description" not in rsd.apply_app_details(game, {"short_description": blurb})
def test_a_disclosed_anti_cheat_changes_no_manual_field():
# The page now carries the anti-cheat disclosure too; refresh_anticheat.py
# applies it, never this job.
page = _page() + ('<div class="anticheat_section DRM_notice" ><div>Uses Kernel Level Anti-Cheat</div>'
'<div class="anticheat_name">BattlEye<span class="anticheat_uninstalls"></span></div></div>')
game = _game(is_kernel_ac=None, anti_cheat_note="")
scraped = scrape_store_page(page)
assert scraped["anti_cheat"].name == "BattlEye"
changed = rsd.apply_store_page(game, scraped)
assert not set(changed) & MANUAL_FIELDS
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("VAC", "", None)
def test_the_refreshed_fields_never_include_human_judgement():
assert not (set(rsd.PAGE_FIELDS) | set(rsd.DETAILS_FIELDS)) & MANUAL_FIELDS
assert "notes" not in rsd.PAGE_FIELDS + rsd.DETAILS_FIELDS
+28
View File
@@ -113,6 +113,34 @@ def test_store_items_are_asked_for_in_batches_and_a_failed_batch_fails_all(monke
assert len(session.requests) == 2
def test_store_items_ask_for_the_anti_cheat_disclosure_too():
client, session = _client(_Resp(steam_client.STORE_ITEMS_URL, body={"response": {"store_items": []}}))
client.fetch_store_items(["2279510"])
sent = json.loads(session.requests[0]["params"]["input_json"])
assert sent["data_request"] == {"include_tag_count": 20, "include_supported_languages": True,
"include_extra_details": True}
def test_anticheat_items_survive_a_failed_batch(monkeypatch):
monkeypatch.setattr(steam_client, "STORE_ITEMS_BATCH", 2)
one = {"response": {"store_items": [{"id": 1, "success": 1, "extra_details": {"anticheat": {}}}]}}
client, session = _client(_Resp(steam_client.STORE_ITEMS_URL, status_code=403),
_Resp(steam_client.STORE_ITEMS_URL, body=one))
items, failed = client.fetch_anticheat_items(["3", "4", "1"])
assert list(items) == ["1"] and failed == {"3", "4"}
sent = json.loads(session.requests[1]["params"]["input_json"])
assert sent["data_request"] == {"include_extra_details": True}
def test_anticheat_items_give_up_after_failures_in_a_row(monkeypatch):
monkeypatch.setattr(steam_client, "STORE_ITEMS_BATCH", 1)
bad = [_Resp(steam_client.STORE_ITEMS_URL, status_code=403) for _ in range(2)]
client, session = _client(*bad)
items, failed = client.fetch_anticheat_items(["1", "2", "3", "4"], give_up_after=2)
assert items == {} and failed == {"1", "2", "3", "4"}
assert len(session.requests) == 2 # the rest was not asked
def test_tag_names_are_fetched_once():
body = {"response": {"tags": [{"tagid": 12095, "name": "Sexual Content"}]}}
client, session = _client(_Resp(steam_client.TAG_LIST_URL, body=body))
+10
View File
@@ -6,6 +6,7 @@ and that a DLC price - which arrives as a string - is read as money.
Run from the repository root: python -m pytest scripts/tests
"""
from core.anticheat import Disclosure
from core.scraper import scrape_store_page
from core.store_items import language_name, paid_dlc_from_items, parse_store_item
@@ -70,6 +71,15 @@ def test_the_tagids_list_is_read_when_weights_are_missing():
assert parse_store_item({"tagids": [9130, "6650"]}, TAG_NAMES)["tags"] == ["Hentai", "Nudity"]
def test_an_item_carries_its_anti_cheat():
# Horizon Walker (3279780): a login-gated game whose page cannot be read.
item = {"extra_details": {"anticheat": {"kernel_mode": True}}}
assert parse_store_item(item, TAG_NAMES)["anti_cheat"] == Disclosure("", True, False)
assert parse_store_item({"extra_details": {}}, TAG_NAMES)["anti_cheat"] is None
# The page scrape has the same key, so either can stand in for the other.
assert scrape_store_page("<html></html>")["anti_cheat"] is None
def test_a_dlc_price_is_money_even_as_a_string():
assert paid_dlc_from_items([{"best_purchase_option": {"final_price_in_cents": "999"}}]) is True
assert paid_dlc_from_items([{"best_purchase_option": {"final_price_in_cents": 1999}}]) is True
+2 -2
View File
@@ -66,7 +66,7 @@ export function seed(sqlite: SqliteD1, repo: FakeRepo, records: Rec[], now: Date
r.current_players ?? "N/A",
players(r),
JSON.stringify({ link: r.link, name: r.name, reviews: r.reviews, current_players: r.current_players }),
extra.source ?? (n % 5 === 0 ? "extension" : "discover"),
extra.source ?? (n % 5 === 0 ? "backfill:2025" : "discover"),
status,
extra.by ?? null,
extra.at ?? null,
@@ -121,7 +121,7 @@ export function seed(sqlite: SqliteD1, repo: FakeRepo, records: Rec[], now: Date
// Locked because the GAME is published: undecided rows for a game that
// already has a committed row. Neither may be approved, rejected or deferred.
const published = committed[0];
row(published, "pending", { seenAgo: 2 * HOUR, source: "extension" });
row(published, "pending", { seenAgo: 2 * HOUR, source: "backfill:2025" });
row(published, "failed", { reason: "never appeared in data/ or removed_games.jsonl - approve again to retry", by: reviewer, at: iso(80 * HOUR) });
// A rejected row that cannot be reopened: another row for the game is open.
+1 -1
View File
@@ -401,7 +401,7 @@
<div>
<h1 class="text-2xl font-semibold tracking-tight">Review queue</h1>
<p class="text-sm text-muted-foreground">
Candidates found by discovery and the browser extension. Approving asks the pipeline to publish a game.
Candidates found by discovery. Approving asks the pipeline to publish a game.
</p>
</div>
<div class="flex flex-wrap items-center gap-2">
+1 -1
View File
@@ -235,7 +235,7 @@
<h1 class="text-2xl font-semibold tracking-tight">Remove games</h1>
<p class="max-w-3xl text-sm text-muted-foreground">
A removed game leaves data/ on the next pipeline run, is listed on the site's removed-games page, and stays out:
discovery, the browser extension and every later run skip it until it is restored here.
discovery, a queued link and every later run skip it until it is restored here.
</p>
</div>
+50
View File
@@ -0,0 +1,50 @@
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { canonicalAc } from "./anti-cheat";
import { ANTI_CHEAT_ENUM } from "./enums";
/**
* The anti-cheat patterns live twice: ANTI_CHEAT_PATTERNS in
* scripts/core/constants.py labels what the pipeline writes, and anti-cheat.ts
* buckets it for the charts and filters. A pattern on one side only is a
* product the pipeline labels and the site files under "Other" (Steam's
* disclosure calls Denuvo just "Denuvo"). Nothing else held the two equal.
*/
function pythonPatterns(): [string, string[]][] {
const src = readFileSync(join(__dirname, "..", "..", "..", "scripts", "core", "constants.py"), "utf-8");
const body = src.split("ANTI_CHEAT_PATTERNS")[1].split("\n}")[0];
const out: [string, string[]][] = [];
for (const line of body.split("\n")) {
const m = /^\s*"([^"]+)":\s*\[([^\]]*)\]/.exec(line);
if (m) out.push([m[1], [...m[2].matchAll(/"([^"]*)"/g)].map((p) => p[1])]);
}
return out;
}
describe("anti-cheat patterns", () => {
const python = pythonPatterns();
it("reads the Python table", () => {
expect(python.length).toBeGreaterThanOrEqual(14);
});
it("knows every product the pipeline labels, in the same order", () => {
expect(ANTI_CHEAT_ENUM).toEqual(["-", ...python.map(([key]) => key)]);
});
it("files every Python pattern, and every label, under its own product", () => {
for (const [key, patterns] of python) {
expect(canonicalAc(key), key).toBe(key);
for (const p of patterns) expect(canonicalAc(p), `${key}: ${p}`).toBe(key);
}
});
it("files the names Steam's disclosure uses", () => {
expect(canonicalAc("Denuvo")).toBe("Denuvo AC");
expect(canonicalAc("VAC (Valve Anti-Cheat)")).toBe("VAC");
expect(canonicalAc("KSS (Krafton Security Services)")).toBe("KSS");
expect(canonicalAc("-")).toBeNull();
});
});
+1 -1
View File
@@ -25,7 +25,7 @@ const AC_PATTERNS: Record<string, string[]> = {
Ricochet: ["ricochet anti-cheat", "ricochet"],
mHyprot: ["mhyprot", "mhyprot2"],
"FACEIT AC": ["faceit anti-cheat", "faceit ac"],
"Denuvo AC": ["denuvo anti-cheat"],
"Denuvo AC": ["denuvo anti-cheat", "denuvo"],
KSS: ["kss", "krafton security"],
"NetEase GS": ["netease game security"],
Hyperion: ["hyperion", "byfron"],
-12
View File
@@ -79,18 +79,6 @@ export function migrateRecord(game: Record<string, unknown>): GameRecord {
delete out.desc;
}
for (const k of [
"free_type",
"is_free",
"is_dlc",
"is_demo",
"is_playtest",
"price",
"auto_notes",
]) {
delete out[k];
}
return out as unknown as GameRecord;
}
+3 -2
View File
@@ -3,8 +3,9 @@
* pipeline so manual edits stay compatible with what the daily refetch
* produces.
*
* ANTI_CHEAT_ENUM — keys of ANTI_CHEAT_PATTERNS in scripts/core/constants.py:45
* (extend both sides if you add a new AC).
* ANTI_CHEAT_ENUM — keys of ANTI_CHEAT_PATTERNS in scripts/core/constants.py
* (extend both sides if you add a new AC; anti-cheat.test.ts
* holds them equal).
* GENRE_ENUM — curated short list of common Steam genre buckets.
* Combobox-style: free-text new entries are allowed.
* TYPE_GAME_ENUM — fixed.
+1 -35
View File
@@ -1,6 +1,6 @@
/**
* Schema constants — port of scripts/core/constants.py.
* Keep MANUAL_FIELDS, ARRAY_FIELDS, EXTENSION_FIELDS in sync with the Python source.
* Keep MANUAL_FIELDS and the record shape in sync with the Python source.
*/
export const REPO_OWNER = "poli0981";
@@ -31,36 +31,6 @@ export const MANUAL_FIELDS = [
"genre",
] as const;
export const ARRAY_FIELDS = [
"platforms",
"languages",
"language_details",
"tags",
"developer",
"publisher",
] as const;
export const EXTENSION_FIELDS = [
"name",
"genre",
"type_game",
"has_paid_dlc",
"developer",
"publisher",
"release_date",
"description",
"header_image",
"anti_cheat",
"anti_cheat_note",
"is_kernel_ac",
"platforms",
"languages",
"language_details",
"tags",
"notes",
"safe",
] as const;
export const SKIP_GENRE_TAGS = new Set([
"Free to Play",
"Indie",
@@ -179,7 +149,3 @@ export const SKELETON_TEMPLATE: GameRecord = {
export function isManualField(key: string): key is ManualField {
return (MANUAL_FIELDS as readonly string[]).includes(key);
}
export function isArrayField(key: string): boolean {
return (ARRAY_FIELDS as readonly string[]).includes(key);
}
+6 -6
View File
@@ -164,8 +164,8 @@ export interface AppendRequest {
* Commit only if the queue file is EMPTY at the very head the commit is made
* on. Auto-approve uses it to stay out of the way of an ingest run that has
* not finished with the previous batch - checked here, against the same
* expectedHeadOid, so an extension push between its own check and this
* commit cannot slip past.
* expectedHeadOid, so a commit landing between its own check and this one
* cannot slip past.
*/
requireEmptyQueue?: boolean;
}
@@ -173,10 +173,10 @@ export interface AppendRequest {
/**
* Append one `{"link": ...}` line per appid to the queue file.
*
* APPEND, never replace. This file has two producers — the browser extension
* (poli0981/steam-f2p-extension pushes to it directly) and this Worker — and
* overwriting it silently discards whatever the other queued. That exact bug
* was found and fixed once already, in the since-removed issue-ingest
* APPEND, never replace. This Worker is not the file's only writer - Restore
* appends here too, a pull request may add a link, and the ingest clears it -
* and overwriting it silently discards whatever another one queued. That exact
* bug was found and fixed once already, in the since-removed issue-ingest
* workflow; do not reintroduce it here.
*
* It takes no actor. The commit body says ADMIN_ATTRIBUTION, and the parameter
+5 -4
View File
@@ -13,10 +13,11 @@
* and it writes one only after observing the appid in the published dataset.
*
* It also SWEEPS: a pending, deferred or failed row whose game is already in
* data/ is marked committed. That happens when the browser extension queues a
* game discovery also proposed, and when an approval aged out to 'failed' (see
* STALE_AFTER_MS) and was published afterwards anyway - before the sweep such a
* row stayed 'failed' forever for a game that was live.
* data/ is marked committed. That happens when a game reaches the catalogue by
* another route (a pull request's link, a manual edit) while discovery also
* proposed it, and when an approval aged out to 'failed' (see STALE_AFTER_MS)
* and was published afterwards anyway - before the sweep such a row stayed
* 'failed' forever for a game that was live.
*/
import type { AdminDeps } from "./deps";
import { acquireLease, releaseLease, readState, writeState } from "./locks";
+1 -1
View File
@@ -347,7 +347,7 @@ export async function handleRemoveApi(
},
{
path: TEMP_INFO_PATH,
// APPEND, never replace: the extension writes this file too.
// APPEND, never replace: approvals and pull requests write this file too.
build: (current) => {
for (const line of current.split("\n")) {
try {