fix(pipeline): read anti-cheat from Steam's own disclosure; retire the extension's paths
None of the 4,704 games added since September had an anti-cheat. The only
code that read the anti-cheat a developer declares on Steam was the browser
extension (archived; last push 2026-06-25); the pipeline matched appdetails'
categories, which only ever say "Valve Anti-Cheat enabled".
- core/anticheat.py (new, pure): the store page's div.anticheat_section
(its HEADING decides kernel, never the class; a Boot Protection row is a
flag) and GetItems' extra_details.anticheat; labels from
ANTI_CHEAT_PATTERNS, else Steam's name cut at 40 characters; same_family()
("EAC" = "Easy Anti-Cheat", "ACE" = "Anti-Cheat Expert (ACE)"); and
apply_anti_cheat(), the one rule.
- fetch_full() fills, never replaces, from the page it already fetches (or
GetItems for a gated page), else the VAC category, now found in any
position (with a stored "" only the first category was checked).
- refresh_anticheat.py (new): GetItems for every game (~160 requests), a
store page only for a disclosure without a name. Steam wins; an override
wins over Steam and is not fought (OverrideGuard); no disclosure never
clears; a failed batch is unknown. Runs at the end of the daily Refresh
Store Data job with its own commit; only=anticheat runs it alone. That job
now checks out main.
- The queue takes links only: merge_extension_data(), EXTENSION_FIELDS,
ARRAY_FIELDS and the schema.ts mirrors go. translate_notes() becomes
clean_notes(): the extension's leading "Has paid DLC" and "[hover-add]"
are dropped (517 notes, once, never news).
- refetch_all.py keeps "-" instead of resetting it to ""; a retired override
whose was equals its value no longer bumps last_updated on every save.
- "denuvo" joins Denuvo AC on both sides; anti-cheat.test.ts holds the TS
patterns and enum to constants.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9fc8be83ad
commit
3d0e22cd76
37 files changed
+1553
-296
No files matched your search
@@ -9,6 +9,13 @@ name: Refresh Store Data
|
||||
# scripts/refresh_store_data.py explains the rotation and the rules (never
|
||||
# blanks a value, never touches MANUAL_FIELDS).
|
||||
#
|
||||
# Then the anti-cheat sweep, committed separately: every game's anti-cheat as
|
||||
# its developer declares it on Steam, about 160 GetItems requests plus a store
|
||||
# page for each disclosure Steam does not name there (~70), a few minutes.
|
||||
# scripts/refresh_anticheat.py has its rules: Steam wins, an override wins
|
||||
# over Steam, and no disclosure never clears a value. `only: anticheat` runs
|
||||
# just the sweep.
|
||||
#
|
||||
# 16:00 UTC is the quiet part of this repository's day. Scheduled runs here
|
||||
# start two to five hours late, the morning data jobs (reviews, dead links,
|
||||
# purge) run for two to four hours, and the data-write group keeps only ONE
|
||||
@@ -20,11 +27,11 @@ on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
only:
|
||||
description: 'Refresh one group only; leave empty for both'
|
||||
description: 'One part only: a store group, or the anti-cheat sweep; leave empty for all'
|
||||
required: false
|
||||
default: ''
|
||||
type: choice
|
||||
options: ['', 'page', 'details']
|
||||
options: ['', 'page', 'details', 'anticheat']
|
||||
|
||||
permissions: {contents: write}
|
||||
|
||||
@@ -37,7 +44,10 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 120
|
||||
steps:
|
||||
# main, not the triggering commit: a run that waited its turn in
|
||||
# data-write must work on the data as it is now, not as it was queued.
|
||||
- uses: actions/checkout@v6
|
||||
with: {ref: main}
|
||||
- uses: actions/setup-python@v6
|
||||
with: {python-version: '3.12'}
|
||||
- run: chmod +x ./bash/refresh_store.sh && ./bash/refresh_store.sh
|
||||
|
||||
+2
-2
@@ -4,8 +4,8 @@
|
||||
# A data job can run for hours (update-reviews, check-dead-links,
|
||||
# purge-unhealthy), and `concurrency: data-write` serialises only the
|
||||
# WORKFLOWS. main still moves underneath one: a merged pull request, an /admin
|
||||
# approval or override the Worker commits, the extension's push to
|
||||
# scripts/temp_info.jsonl. A plain `git push` is then rejected ("fetch first")
|
||||
# approval, removal or override the Worker commits. A plain `git push` is then
|
||||
# rejected ("fetch first")
|
||||
# and the whole run is thrown away - purge-unhealthy lost four hours of work
|
||||
# that way on 2026-09-21.
|
||||
#
|
||||
|
||||
+12
-3
@@ -1,6 +1,15 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
pip install --quiet -r requirements.txt
|
||||
# REFRESH_ONLY: "page" or "details" from a manual run; empty = both groups.
|
||||
python scripts/refresh_store_data.py ${REFRESH_ONLY:+--only "$REFRESH_ONLY"}
|
||||
bash "$(dirname "$0")/commit_push.sh" "Refresh store data [$(date +'%Y-%m-%d')]"
|
||||
# REFRESH_ONLY from a manual run: "page" or "details" runs that group of the
|
||||
# store refresh, "anticheat" only the anti-cheat sweep; empty = everything.
|
||||
# Each part commits on its own, so a failure in the sweep never costs the
|
||||
# store refresh its push (and set -e still stops at a crash in either).
|
||||
if [ "${REFRESH_ONLY:-}" != "anticheat" ]; then
|
||||
python scripts/refresh_store_data.py ${REFRESH_ONLY:+--only "$REFRESH_ONLY"}
|
||||
bash "$(dirname "$0")/commit_push.sh" "Refresh store data [$(date +'%Y-%m-%d')]"
|
||||
fi
|
||||
if [ -z "${REFRESH_ONLY:-}" ] || [ "${REFRESH_ONLY:-}" = "anticheat" ]; then
|
||||
python scripts/refresh_anticheat.py
|
||||
bash "$(dirname "$0")/commit_push.sh" "Refresh anti-cheat [$(date +'%Y-%m-%d')]"
|
||||
fi
|
||||
@@ -0,0 +1,289 @@
|
||||
"""
|
||||
A game's anti-cheat as its developer declares it on Steam.
|
||||
|
||||
Steam asks a developer to say which anti-cheat a game uses and whether it runs
|
||||
in the kernel, and shows the answer in two places (measured 2026-10-16):
|
||||
|
||||
GetItems IStoreBrowseService/GetItems with data_request.include_extra_details:
|
||||
extra_details.anticheat = {kernel_mode, uninstall_completely,
|
||||
boot_protection, boot_protection_name, name, name_loc}. False
|
||||
booleans are left out, so {} is an anti-cheat outside the kernel;
|
||||
no `anticheat` at all is no disclosure. Steam's own list of
|
||||
products comes with a `name` (Delta Force: "Anti-Cheat Expert
|
||||
(ACE)"); a developer's own wording mostly does not (PUBG: only
|
||||
kernel_mode). It answers for the 'Adult Only' games whose store
|
||||
page needs a login.
|
||||
The page one div.anticheat_section in the Features block:
|
||||
<div class="anticheat_section DRM_notice" >
|
||||
<div>Uses Kernel Level Anti-Cheat</div>
|
||||
<div class="anticheat_name">Easy Anti-Cheat<span class="anticheat_uninstalls"> - Requires manual removal after game uninstall</span></div>
|
||||
</div>
|
||||
The HEADING says whether it is kernel level ("Uses Kernel Level
|
||||
Anti-Cheat" / "Uses Anti-Cheat Software"), never the class:
|
||||
DRM_notice also marks an anti-cheat outside the kernel that needs
|
||||
Secure Boot. A "Boot Protection" row is that requirement, not a
|
||||
product. Names can be empty, free text, or end in a space.
|
||||
|
||||
No disclosure does not mean no anti-cheat - Warframe, Overwatch 2 and NARAKA
|
||||
declare none - so nothing here ever clears a stored value.
|
||||
|
||||
apply_anti_cheat() is the one rule every writer uses: fetch_full() fills an
|
||||
empty game, refresh_anticheat.py also replaces a value Steam contradicts, and
|
||||
an override (core/overrides.py) beats both.
|
||||
"""
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from html import unescape
|
||||
from typing import Optional
|
||||
|
||||
from .constants import ANTI_CHEAT_PATTERNS
|
||||
from .data_store import ELLIPSIS, extract_appid, is_empty
|
||||
from .normalize import clean_text
|
||||
|
||||
FIELDS = ("anti_cheat", "anti_cheat_note", "is_kernel_ac")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Disclosure:
|
||||
"""What Steam says. `name` is "" when Steam does not name the product;
|
||||
`kernel` is None only for a page whose heading this module cannot read."""
|
||||
name: str = ""
|
||||
kernel: Optional[bool] = None
|
||||
boot_protection: bool = False
|
||||
|
||||
|
||||
# ──────────── The store page ────────────
|
||||
|
||||
# The heading, then the name rows that follow it, each with its own </div>.
|
||||
_RE_SECTION = re.compile(
|
||||
r'<div\s+class\s*=\s*"anticheat_section(?:\s[^"]*)?"[^>]*>\s*'
|
||||
r'<div>([^<]*)</div>'
|
||||
r'((?:\s*<div\s+class\s*=\s*"anticheat_name"[^>]*>.*?</div>)*)',
|
||||
re.DOTALL | re.IGNORECASE,
|
||||
)
|
||||
_RE_NAME_ROW = re.compile(
|
||||
r'<div\s+class\s*=\s*"anticheat_name"[^>]*>(.*?)</div>', re.DOTALL | re.IGNORECASE)
|
||||
_RE_UNINSTALLS = re.compile(
|
||||
r'<span\s+class\s*=\s*"anticheat_uninstalls"[^>]*>.*?</span>', re.DOTALL | re.IGNORECASE)
|
||||
_RE_TAG = re.compile(r"<[^>]+>")
|
||||
|
||||
_HEADINGS = {
|
||||
"uses kernel level anti-cheat": True,
|
||||
"uses anti-cheat software": False,
|
||||
}
|
||||
BOOT_PROTECTION = "boot protection"
|
||||
|
||||
|
||||
def _text(html: str) -> str:
|
||||
return clean_text(unescape(_RE_TAG.sub("", html)))
|
||||
|
||||
|
||||
def parse_page(html: Optional[str]) -> Optional[Disclosure]:
|
||||
"""The disclosure on an English store page, or None when it has none."""
|
||||
m = _RE_SECTION.search(html or "")
|
||||
if not m:
|
||||
return None
|
||||
kernel = _HEADINGS.get(_text(m.group(1)).lower())
|
||||
name, boot = "", False
|
||||
for row in _RE_NAME_ROW.findall(m.group(2)):
|
||||
text = _text(_RE_UNINSTALLS.sub("", row))
|
||||
if text.lower() == BOOT_PROTECTION:
|
||||
boot = True
|
||||
elif text and not name:
|
||||
name = text
|
||||
return Disclosure(name=name, kernel=kernel, boot_protection=boot)
|
||||
|
||||
|
||||
# ──────────── Steam's JSON ────────────
|
||||
|
||||
def from_store_item(item: Optional[dict]) -> Optional[Disclosure]:
|
||||
"""The disclosure in one GetItems item, or None when it has none."""
|
||||
details = (item or {}).get("extra_details")
|
||||
ac = details.get("anticheat") if isinstance(details, dict) else None
|
||||
if not isinstance(ac, dict):
|
||||
return None
|
||||
name = ac.get("name")
|
||||
return Disclosure(
|
||||
name=clean_text(name) if isinstance(name, str) else "",
|
||||
kernel=bool(ac.get("kernel_mode")),
|
||||
boot_protection=bool(ac.get("boot_protection")),
|
||||
)
|
||||
|
||||
|
||||
VAC = Disclosure(name="VAC (Valve Anti-Cheat)", kernel=False)
|
||||
|
||||
|
||||
def from_categories(categories) -> Optional[Disclosure]:
|
||||
"""VAC from appdetails' "Valve Anti-Cheat enabled" category (id 8), in any
|
||||
position. Named as Steam names VAC in a disclosure, so a game that later
|
||||
declares it ends with the same note. VAC never runs in the kernel."""
|
||||
for cat in categories or []:
|
||||
if not isinstance(cat, dict):
|
||||
continue
|
||||
desc = str(cat.get("description") or "").lower()
|
||||
if cat.get("id") == 8 or any(p in desc for p in ANTI_CHEAT_PATTERNS["VAC"]):
|
||||
return VAC
|
||||
return None
|
||||
|
||||
|
||||
def combine(api: Optional[Disclosure], page: Optional[Disclosure]) -> Optional[Disclosure]:
|
||||
"""GetItems decides whether there is a disclosure and whether it is kernel
|
||||
level; the page supplies the name GetItems leaves out."""
|
||||
if api is None:
|
||||
return page
|
||||
if page is None or api.name:
|
||||
return api
|
||||
return Disclosure(name=page.name, kernel=api.kernel,
|
||||
boot_protection=api.boot_protection or page.boot_protection)
|
||||
|
||||
|
||||
# ──────────── Labels ────────────
|
||||
|
||||
def _alnum(text: str) -> str:
|
||||
return "".join(ch for ch in str(text).casefold() if ch.isalnum())
|
||||
|
||||
|
||||
_RE_ACRONYM = re.compile(r"\(([A-Za-z0-9]{2,6})\)")
|
||||
|
||||
|
||||
def canonical(text: str) -> Optional[str]:
|
||||
"""The ANTI_CHEAT_PATTERNS key a name belongs to: one of its patterns, or
|
||||
the key's own name ("EAC", "KSS"). None for anything else."""
|
||||
low = str(text).lower()
|
||||
flat = _alnum(text)
|
||||
for key, patterns in ANTI_CHEAT_PATTERNS.items():
|
||||
if flat == _alnum(key) or any(p in low for p in patterns):
|
||||
return key
|
||||
return None
|
||||
|
||||
|
||||
# Longest label. A developer can type a whole paragraph where the product name
|
||||
# goes ("Frontier uses SCUE4 as its anti-cheat, which comes pre-built ...", 300
|
||||
# characters), and the label is a badge on every list; the note keeps it all.
|
||||
LABEL_MAX = 40
|
||||
|
||||
|
||||
def label(name: str) -> str:
|
||||
"""The stored `anti_cheat` for a name Steam shows: the short key for a
|
||||
known product, else Steam's own name, cut on a word at LABEL_MAX."""
|
||||
key = canonical(name)
|
||||
if key:
|
||||
return key
|
||||
text = clean_text(name)
|
||||
if len(text) <= LABEL_MAX:
|
||||
return text
|
||||
cut = text[:LABEL_MAX - 1]
|
||||
if cut.rfind(" ") >= LABEL_MAX // 2:
|
||||
cut = cut[:cut.rfind(" ")]
|
||||
return cut.rstrip(" ,;:.-") + ELLIPSIS
|
||||
|
||||
|
||||
def _aliases(text: str) -> set[str]:
|
||||
out = {_alnum(text)}
|
||||
out.update(m.lower() for m in _RE_ACRONYM.findall(str(text)))
|
||||
return out - {""}
|
||||
|
||||
|
||||
def same_family(a: str, b: str) -> bool:
|
||||
"""Whether two anti-cheat values name the same product: "Easy Anti-Cheat"
|
||||
and "EAC", "ACE" and "Anti-Cheat Expert (ACE)", "CrackProof®" and
|
||||
"CrackProof"."""
|
||||
if clean_text(str(a)).casefold() == clean_text(str(b)).casefold():
|
||||
return True
|
||||
ca, cb = canonical(a), canonical(b)
|
||||
if ca or cb:
|
||||
return ca == cb
|
||||
fa, fb = _alnum(a), _alnum(b)
|
||||
for text, short, full in ((a, fa, fb), (b, fb, fa)):
|
||||
# A label label() cut short, and the name it was cut from.
|
||||
if str(text).endswith(ELLIPSIS) and len(short) >= 12 and full.startswith(short):
|
||||
return True
|
||||
return bool(_aliases(a) & _aliases(b))
|
||||
|
||||
|
||||
# ──────────── Notes ────────────
|
||||
|
||||
KERNEL, NON_KERNEL = "[Kernel]", "[Non-Kernel]"
|
||||
|
||||
|
||||
def note_for(name: str, kernel: Optional[bool], boot_protection: bool = False) -> str:
|
||||
"""The note the extension wrote and the 71 games before this module carry:
|
||||
"Easy Anti-Cheat [Kernel]", plus "; Boot Protection" when Steam says so."""
|
||||
note = name
|
||||
if kernel is not None:
|
||||
note = f"{name} {KERNEL if kernel else NON_KERNEL}"
|
||||
return f"{note}; Boot Protection" if boot_protection else note
|
||||
|
||||
|
||||
def _with_suffix(note, kernel: bool) -> Optional[str]:
|
||||
"""The note with its kernel suffix set to `kernel`, or None when the note
|
||||
carries no suffix (a sentence a person wrote is not ours to rewrite)."""
|
||||
if not isinstance(note, str):
|
||||
return None
|
||||
for suffix in (KERNEL, NON_KERNEL):
|
||||
idx = note.find(f" {suffix}")
|
||||
if idx >= 0:
|
||||
return note[:idx] + f" {KERNEL if kernel else NON_KERNEL}" + note[idx + 1 + len(suffix):]
|
||||
return None
|
||||
|
||||
|
||||
# ──────────── The rule ────────────
|
||||
|
||||
def apply_anti_cheat(game: dict, found: Optional[Disclosure], replace: bool = False,
|
||||
guard=None) -> list[str]:
|
||||
"""Bring one record's anti_cheat, anti_cheat_note and is_kernel_ac in line
|
||||
with `found`, in place. Returns the fields whose value changed.
|
||||
|
||||
- Nothing found, or found without a name: nothing changes.
|
||||
- No anti-cheat stored yet: all three are written.
|
||||
- The same product: an empty note is filled and the kernel flag follows
|
||||
Steam (only an empty one unless `replace`); a note that carries a
|
||||
"[Kernel]"/"[Non-Kernel]" suffix gets the new one, other notes stay.
|
||||
- Another product: all three are replaced, only when `replace`.
|
||||
- `guard` (overrides.OverrideGuard) refuses writes an override would
|
||||
undo. With the label locked to another product nothing is written; a
|
||||
locked kernel flag decides the suffix of a note written here.
|
||||
"""
|
||||
if found is None or not found.name:
|
||||
return []
|
||||
appid = extract_appid(game.get("link", "")) or ""
|
||||
|
||||
def allowed(field: str, value) -> bool:
|
||||
return guard is None or guard.allows(appid, field, value)
|
||||
|
||||
changed: list[str] = []
|
||||
|
||||
def put(field: str, value) -> None:
|
||||
if game.get(field) != value and allowed(field, value):
|
||||
game[field] = value
|
||||
changed.append(field)
|
||||
|
||||
new_label = label(found.name)
|
||||
stored = game.get("anti_cheat")
|
||||
fresh = is_empty(stored)
|
||||
other = not fresh and not (same_family(stored, found.name) or same_family(stored, new_label))
|
||||
if other and not replace:
|
||||
return []
|
||||
if (fresh or other) and not allowed("anti_cheat", new_label):
|
||||
return [] # an override decided the product ("-" included)
|
||||
|
||||
kernel_before = game.get("is_kernel_ac")
|
||||
if found.kernel is not None and (fresh or other or replace or kernel_before is None):
|
||||
put("is_kernel_ac", found.kernel)
|
||||
# The note's suffix says what is STORED: Steam's flag, or the one an
|
||||
# override locked (none at all for a locked null).
|
||||
kernel_now = game.get("is_kernel_ac")
|
||||
suffix = kernel_now if isinstance(kernel_now, bool) else None
|
||||
|
||||
note = game.get("anti_cheat_note")
|
||||
if fresh or other:
|
||||
put("anti_cheat", new_label)
|
||||
if other or is_empty(note) or _with_suffix(note, True) is not None:
|
||||
put("anti_cheat_note", note_for(found.name, suffix, found.boot_protection))
|
||||
elif is_empty(note):
|
||||
put("anti_cheat_note", note_for(found.name, suffix, found.boot_protection))
|
||||
elif isinstance(suffix, bool):
|
||||
swapped = _with_suffix(note, suffix)
|
||||
if swapped is not None:
|
||||
put("anti_cheat_note", swapped)
|
||||
return changed
|
||||
@@ -5,8 +5,8 @@ data/blocklist.jsonl holds one line per blocked game. It is written by the
|
||||
Worker (/admin/remove appends lines, Restore deletes one) and by
|
||||
scripts/delete_game.py, and only READ here. Like data/overrides/, it is human
|
||||
intent that every write re-imposes: save_main() drops a blocked record before
|
||||
anything else happens to it, so no writer - the ingest of a link the browser
|
||||
extension queued again, a full refetch, a purge - can put the game back. The
|
||||
anything else happens to it, so no writer - the ingest of a link queued again,
|
||||
a full refetch, a purge - can put the game back. The
|
||||
only way back is to delete the line (Restore in /admin), which also queues the
|
||||
link for ingest_new.py in the same commit.
|
||||
|
||||
|
||||
@@ -61,7 +61,8 @@ ANTI_CHEAT_PATTERNS: dict[str, list[str]] = {
|
||||
"Ricochet": ["ricochet anti-cheat", "ricochet"],
|
||||
"mHyprot": ["mhyprot", "mhyprot2"],
|
||||
"FACEIT AC": ["faceit anti-cheat", "faceit ac"],
|
||||
"Denuvo AC": ["denuvo anti-cheat"],
|
||||
# Steam's own name for it, in a disclosure, is just "Denuvo".
|
||||
"Denuvo AC": ["denuvo anti-cheat", "denuvo"],
|
||||
"KSS": ["kss", "krafton security"],
|
||||
"NetEase GS": ["netease game security"],
|
||||
"Hyperion": ["hyperion", "byfron"],
|
||||
@@ -91,17 +92,6 @@ MACHINE_NOTE_MARKERS = (
|
||||
"💀 Delisted",
|
||||
"⚠ No longer free!",
|
||||
)
|
||||
ARRAY_FIELDS = frozenset({
|
||||
"platforms", "languages", "language_details", "tags",
|
||||
"developer", "publisher",
|
||||
})
|
||||
EXTENSION_FIELDS = frozenset({
|
||||
"name", "genre", "type_game", "has_paid_dlc",
|
||||
"developer", "publisher", "release_date", "description",
|
||||
"header_image", "anti_cheat", "anti_cheat_note", "is_kernel_ac",
|
||||
"platforms", "languages", "language_details", "tags",
|
||||
"notes", "safe",
|
||||
})
|
||||
|
||||
TOP_ONLINE_LIMIT = 100
|
||||
TOP_OFFLINE_LIMIT = 100
|
||||
@@ -5,7 +5,6 @@ Optimizations:
|
||||
- _is_empty() simplified with early returns (no set lookup)
|
||||
- save_jsonl() uses json.dumps directly (faster than jsonlines writer for large files)
|
||||
- build_index() O(n) single pass
|
||||
- merge_extension_data() avoids repeated key lookups
|
||||
"""
|
||||
import glob as _glob
|
||||
import hashlib
|
||||
@@ -18,7 +17,7 @@ from typing import Optional
|
||||
from .constants import (
|
||||
DATA_DIR, LEGACY_JSONL, TEMP_JSONL, REMOVED_JSONL,
|
||||
MAX_RECORDS_PER_FILE, SHARD_PREFIX,
|
||||
MANUAL_FIELDS, ARRAY_FIELDS, DESCRIPTION_MAX,
|
||||
DESCRIPTION_MAX,
|
||||
BLOCKLIST_MAX_DROP,
|
||||
)
|
||||
from .normalize import normalize_records
|
||||
@@ -268,11 +267,11 @@ def save_main(records: list[dict], apply_human_overrides: bool = True,
|
||||
"""Save records to sharded JSONL files (MAX_RECORDS_PER_FILE per shard).
|
||||
|
||||
Human overrides are re-imposed here, immediately before the write. This is
|
||||
the single choke point for the dataset - all 14 data-writing scripts call
|
||||
the single choke point for the dataset - every data-writing script calls
|
||||
save_main() and nothing else writes data/data_*.jsonl - so applying them
|
||||
here is what makes a correction survive refetch_all.py and
|
||||
normalize_genres.py --apply without either script knowing this layer
|
||||
exists. See core/overrides.py.
|
||||
here is what makes a correction survive refetch_all.py,
|
||||
refresh_anticheat.py and normalize_genres.py --apply without those scripts
|
||||
knowing this layer exists. See core/overrides.py.
|
||||
|
||||
The blocklist (core/blocklist.py) is applied FIRST, the same way: a game
|
||||
the maintainer removed is left out of every write, whichever script made
|
||||
@@ -283,9 +282,9 @@ def save_main(records: list[dict], apply_human_overrides: bool = True,
|
||||
un-overridden values (an audit report). Never pass it from the pipeline.
|
||||
|
||||
The representation rules of core/normalize.py run first - one release
|
||||
date shape, no invisible characters, no HTML entities, no Vietnamese
|
||||
notes - so every writer gets them, the browser extension's ingest
|
||||
included. Like the truncation, they never touch last_updated.
|
||||
date shape, no invisible characters, no HTML entities, none of the retired
|
||||
browser extension's note phrases - so every writer gets them. Like the
|
||||
truncation, they never touch last_updated.
|
||||
"""
|
||||
# Before everything else, so no work is spent on records that are not
|
||||
# written, and a blocked game's override reports as an orphan (its file
|
||||
@@ -493,7 +492,7 @@ def is_info_complete(game: dict) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
# ──────────── Extension data merge ────────────
|
||||
# ──────────── Migration ────────────
|
||||
|
||||
def _normalize_to_list(val) -> list[str]:
|
||||
if isinstance(val, list):
|
||||
@@ -503,54 +502,6 @@ def _normalize_to_list(val) -> list[str]:
|
||||
return []
|
||||
|
||||
|
||||
# Pre-compute sets for O(1) membership tests
|
||||
# Keys from extension that should NEVER be stored in data.jsonl
|
||||
_SKIP_MERGE_KEYS = frozenset({
|
||||
"link", "appid", "added_at",
|
||||
# Extension-only classification (not stored server-side)
|
||||
"free_type", "is_free", "is_dlc", "is_demo", "is_playtest",
|
||||
"price", "auto_notes",
|
||||
})
|
||||
_NORMALIZE_LIST_KEYS = frozenset({"developer", "publisher"})
|
||||
_DESC_KEYS = frozenset({"description", "desc"})
|
||||
|
||||
|
||||
def merge_extension_data(game: dict, ext: dict) -> dict:
|
||||
"""
|
||||
Merge extension data into game record.
|
||||
Optimized: single pass over ext keys, minimal dict lookups.
|
||||
"""
|
||||
for key, val in ext.items():
|
||||
if key in _SKIP_MERGE_KEYS or is_empty(val):
|
||||
continue
|
||||
|
||||
if key in _NORMALIZE_LIST_KEYS:
|
||||
val = _normalize_to_list(val)
|
||||
|
||||
if key in _DESC_KEYS:
|
||||
target = "description"
|
||||
if is_empty(game.get(target)):
|
||||
game[target] = val
|
||||
continue
|
||||
|
||||
if key in MANUAL_FIELDS:
|
||||
if is_empty(game.get(key)):
|
||||
game[key] = val
|
||||
continue
|
||||
|
||||
if key in ARRAY_FIELDS:
|
||||
game[key] = val
|
||||
continue
|
||||
|
||||
# EXTENSION_FIELDS or unknown: overwrite
|
||||
game[key] = val
|
||||
|
||||
# Strip any ephemeral fields that may have leaked through
|
||||
for k in ("free_type", "is_free", "is_dlc", "is_demo", "is_playtest", "price", "auto_notes"):
|
||||
game.pop(k, None)
|
||||
return game
|
||||
|
||||
|
||||
def migrate_record(game: dict) -> dict:
|
||||
"""Migrate v2.0 record to v2.2. Idempotent."""
|
||||
# Add missing keys
|
||||
@@ -570,9 +521,4 @@ def migrate_record(game: dict) -> dict:
|
||||
game["description"] = game["desc"]
|
||||
del game["desc"]
|
||||
|
||||
# Remove deprecated/ephemeral fields
|
||||
for k in ("free_type", "is_free", "is_dlc", "is_demo",
|
||||
"is_playtest", "price", "auto_notes"):
|
||||
game.pop(k, None)
|
||||
|
||||
return game
|
||||
+28
-32
@@ -11,13 +11,15 @@ import random
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from .anticheat import apply_anti_cheat, from_categories
|
||||
from .constants import (
|
||||
STEAM_API_KEY, SKIP_GENRE_TAGS, ANTI_CHEAT_PATTERNS,
|
||||
STEAM_API_KEY, SKIP_GENRE_TAGS,
|
||||
BATCH_SIZE, BATCH_PAUSE_MIN, BATCH_PAUSE_MAX,
|
||||
)
|
||||
from .steam_client import get_client
|
||||
from .data_store import extract_appid, now_iso, is_info_complete, is_empty
|
||||
from .normalize import decode_entities, normalize_release_date
|
||||
from .overrides import OverrideGuard
|
||||
from .peaks import raise_peak
|
||||
from .scraper import scrape_store_page
|
||||
from .store_items import paid_dlc_from_items, parse_store_item
|
||||
@@ -97,23 +99,11 @@ def apply_details(game: dict, data: dict) -> dict:
|
||||
if p.get("linux"): plats.append("Linux")
|
||||
game["platforms"] = plats
|
||||
|
||||
# Anti-cheat from categories
|
||||
ac = game.get("anti_cheat", "-")
|
||||
if is_empty(ac) or ac == "-":
|
||||
for cat in data.get("categories", []):
|
||||
desc_lower = cat.get("description", "").lower()
|
||||
for label, pats in ANTI_CHEAT_PATTERNS.items():
|
||||
if any(p in desc_lower for p in pats):
|
||||
game["anti_cheat"] = label
|
||||
if is_empty(game.get("anti_cheat_note")):
|
||||
game["anti_cheat_note"] = cat["description"]
|
||||
break
|
||||
if game["anti_cheat"] != "-":
|
||||
break
|
||||
# Anti-cheat is not read here: fetch_full() takes it from the developer's
|
||||
# disclosure on the store page, and from these categories only for VAC.
|
||||
|
||||
# Infer type_game from categories when not manually set. Manual prefills
|
||||
# win because MANUAL_FIELDS gates the merge in merge_extension_data(),
|
||||
# and ingest_new.py re-applies overrides after fetch_full.
|
||||
# Infer type_game from categories when not set. It is a manual field: only
|
||||
# an empty one is inferred, and an override still wins in save_main().
|
||||
if is_empty(game.get("type_game")):
|
||||
inferred = _infer_type_game(data.get("categories", []))
|
||||
if inferred:
|
||||
@@ -220,6 +210,7 @@ def scrape_store_fields(client, appid: str, details: Optional[dict] = None) -> O
|
||||
(core/store_items.py), and has_paid_dlc from the prices of the game's DLC.
|
||||
A page missing its language table or its tags gets only the missing part
|
||||
from the API. A failed request, or a 404, reads as nothing at all.
|
||||
`anti_cheat` is the page's disclosure, or the API's when the API was asked.
|
||||
|
||||
`details` is the game's appdetails when the caller already has them; they
|
||||
are fetched here only if a DLC check needs them.
|
||||
@@ -244,19 +235,24 @@ def scrape_store_fields(client, appid: str, details: Optional[dict] = None) -> O
|
||||
scraped["language_details"] = api["language_details"]
|
||||
if not scraped["tags"]:
|
||||
scraped["tags"] = api["tags"]
|
||||
if scraped.get("anti_cheat") is None:
|
||||
scraped["anti_cheat"] = api.get("anti_cheat")
|
||||
return scraped
|
||||
|
||||
|
||||
# ──────────── fetch_full ────────────
|
||||
|
||||
def fetch_full(game: dict, client=None, fetch_players=True,
|
||||
scrape=True, prefetched_details=None) -> dict:
|
||||
scrape=True, prefetched_details=None, guard=None) -> dict:
|
||||
"""
|
||||
Fetch all data for one game.
|
||||
|
||||
Args:
|
||||
prefetched_details: If health_checker already fetched appdetails,
|
||||
pass it here to avoid a second API call.
|
||||
guard: an overrides.OverrideGuard. The anti-cheat fields are written
|
||||
only where no override would undo them (an override may hold
|
||||
is_kernel_ac at null, which reads as empty).
|
||||
"""
|
||||
c = client or get_client()
|
||||
appid = extract_appid(game.get("link", ""))
|
||||
@@ -281,24 +277,23 @@ def fetch_full(game: dict, client=None, fetch_players=True,
|
||||
if count is not None:
|
||||
apply_players(game, count)
|
||||
|
||||
# 4) Store page (languages + tags + DLC pricing) – single request, or
|
||||
# Steam's JSON API when the page is gated (scrape_store_fields).
|
||||
# Always scrape if has_paid_dlc hasn't been set by extension,
|
||||
# or if any of languages/tags/language_details are missing.
|
||||
needs_scrape = (
|
||||
is_empty(game.get("languages")) or
|
||||
is_empty(game.get("tags")) or
|
||||
is_empty(game.get("language_details")) or
|
||||
not game.get("_dlc_checked") # internal flag: DLC not yet verified from HTML
|
||||
)
|
||||
if scrape and needs_scrape:
|
||||
# 4) Store page (languages + tags + DLC pricing + anti-cheat disclosure) -
|
||||
# a single request, or Steam's JSON API when the page is gated
|
||||
# (scrape_store_fields).
|
||||
disclosure = None
|
||||
if scrape:
|
||||
scraped = scrape_store_fields(c, appid, details=data)
|
||||
if scraped:
|
||||
apply_scraped(game, scraped)
|
||||
game["_dlc_checked"] = True
|
||||
disclosure = scraped.get("anti_cheat")
|
||||
|
||||
# 5) Anti-cheat, fill only (refresh_anticheat.py is what replaces): what the
|
||||
# developer declares on Steam, else appdetails' VAC category.
|
||||
apply_anti_cheat(game, disclosure, guard=guard)
|
||||
if is_empty(game.get("anti_cheat")):
|
||||
apply_anti_cheat(game, from_categories((data or {}).get("categories")), guard=guard)
|
||||
|
||||
# Housekeeping
|
||||
game.pop("_dlc_checked", None) # internal flag, don't persist
|
||||
if not game.get("type_game"):
|
||||
game["type_game"] = "offline"
|
||||
if not game.get("notes", "").strip():
|
||||
@@ -337,7 +332,8 @@ def update_all_full(games, force=False):
|
||||
if not to_fetch:
|
||||
print("All up to date!")
|
||||
return
|
||||
process_batch(to_fetch, lambda g: fetch_full(g, client=client), "Fetching")
|
||||
guard = OverrideGuard.load()
|
||||
process_batch(to_fetch, lambda g: fetch_full(g, client=client, guard=guard), "Fetching")
|
||||
|
||||
|
||||
def update_reviews_only(games):
|
||||
|
||||
+20
-15
@@ -213,24 +213,29 @@ def decode_tags(tags):
|
||||
|
||||
# ──────────── notes ────────────
|
||||
|
||||
# The browser extension (poli0981/steam-f2p-extension) records "this game has
|
||||
# paid DLC" as Vietnamese text in `notes`. The catalogue is English - nine
|
||||
# records already said "Has paid DLC" - and has_paid_dlc carries the fact.
|
||||
_NOTE_TRANSLATIONS = (
|
||||
("Có DLC trả phí", "Has paid DLC"),
|
||||
)
|
||||
# The retired browser extension (poli0981/steam-f2p-extension) began `notes`
|
||||
# with two phrases a reader gains nothing from: "Has paid DLC" (written in
|
||||
# Vietnamese, then translated in v4.2.1) - a fact has_paid_dlc carries and the
|
||||
# weekly store refresh keeps true, where the note went stale: 10 contradicted
|
||||
# the flag and 791 games with paid DLC never had it - and "[hover-add]", how
|
||||
# the game was added. Measured 2026-10-16, both only ever came first.
|
||||
_RE_NOTE_ARTEFACT = re.compile(r"^(?:Has paid DLC|\[hover-add\])(?=$|;|\s)\s*;?\s*")
|
||||
NOT_REVIEWED = "Not reviewed yet"
|
||||
|
||||
|
||||
def translate_notes(value):
|
||||
"""notes with the extension's Vietnamese phrases in English; the rest of the
|
||||
note ("; Easy Anti-Cheat [Kernel]", "💀 Dead game ...") stays as it is."""
|
||||
def clean_notes(value):
|
||||
"""notes without the extension's leading phrases. The rest of the note
|
||||
("Easy Anti-Cheat [Kernel]", "💀 Dead game ...") stays as it is; a note
|
||||
that was nothing else reads "Not reviewed yet", as fetch_full() writes it
|
||||
for a game nobody has looked at. An empty note stays empty."""
|
||||
if not isinstance(value, str):
|
||||
return value
|
||||
nfc = unicodedata.normalize("NFC", value)
|
||||
out = nfc
|
||||
for vi, en in _NOTE_TRANSLATIONS:
|
||||
out = out.replace(vi, en)
|
||||
return out if out != nfc else value
|
||||
out = value
|
||||
while (m := _RE_NOTE_ARTEFACT.match(out)) is not None:
|
||||
out = out[m.end():]
|
||||
if out == value:
|
||||
return value
|
||||
return out if out.strip() else NOT_REVIEWED
|
||||
|
||||
|
||||
# ──────────── Records ────────────
|
||||
@@ -265,7 +270,7 @@ def normalize_record(game: dict) -> list[str]:
|
||||
hit.append("text")
|
||||
if "release_date" in game and _set(game, "release_date", normalize_release_date(game["release_date"])):
|
||||
hit.append("release_date")
|
||||
if "notes" in game and _set(game, "notes", translate_notes(game["notes"])):
|
||||
if "notes" in game and _set(game, "notes", clean_notes(game["notes"])):
|
||||
hit.append("notes")
|
||||
return hit
|
||||
|
||||
|
||||
+55
-14
@@ -4,22 +4,25 @@ Human edits as standing instructions, not one-time mutations.
|
||||
The problem this solves
|
||||
-----------------------
|
||||
MANUAL_FIELDS (anti_cheat, anti_cheat_note, is_kernel_ac, notes, type_game,
|
||||
safe, genre) are protected only by "fill if empty": merge_extension_data()
|
||||
skips them when the existing value is non-empty. Nothing anywhere records which
|
||||
value a human chose and which a scraper guessed, so:
|
||||
safe, genre) are protected only by "fill if empty": apply_details() and
|
||||
fetch_full() skip them when the existing value is non-empty. Nothing anywhere
|
||||
records which value a human chose and which a scraper guessed, so:
|
||||
|
||||
* `normalize_genres.py --apply` rewrites `genre` for every game and cannot
|
||||
tell a correction from scraper output;
|
||||
* `refresh_anticheat.py` replaces the three anti-cheat fields with what
|
||||
Steam's disclosure says;
|
||||
* `refetch_all.py` re-fetches everything;
|
||||
|
||||
and a hand-made correction is reverted with no error and no trace.
|
||||
|
||||
An override is therefore NOT applied once. It lives permanently in Git as
|
||||
`data/overrides/<appid>.json`, and `apply_overrides()` is called from inside
|
||||
`save_main()` — the single choke point through which all 14 data-writing
|
||||
scripts pass, and the only code that writes `data/data_*.jsonl`. Whatever a
|
||||
script did to a record, the human's decision is re-imposed as the last act
|
||||
before bytes hit the disk. No script needs to know this layer exists.
|
||||
`save_main()` — the single choke point through which every data-writing script
|
||||
passes, and the only code that writes `data/data_*.jsonl`. Whatever a script
|
||||
did to a record, the human's decision is re-imposed as the last act before
|
||||
bytes hit the disk. No script needs to know this layer exists; one that writes
|
||||
manual fields daily asks OverrideGuard first, so it does not fight it.
|
||||
|
||||
Retiring an override
|
||||
--------------------
|
||||
@@ -36,7 +39,7 @@ import os
|
||||
from typing import Optional
|
||||
|
||||
from .constants import MANUAL_FIELDS, MACHINE_NOTE_MARKERS, OVERRIDES_DIR
|
||||
from .normalize import translate_notes
|
||||
from .normalize import clean_notes
|
||||
|
||||
OVERRIDE_SCHEMA = 1
|
||||
|
||||
@@ -167,6 +170,42 @@ def validate_value(field: str, value) -> Optional[str]:
|
||||
return None
|
||||
|
||||
|
||||
class OverrideGuard:
|
||||
"""Which writes to a manual field an override would undo.
|
||||
|
||||
A script that writes a manual field into a record an override governs gains
|
||||
nothing - save_main() puts the human value back - and loses a lot: the
|
||||
record's last_updated moves on every run, and so do the shard bytes. Two
|
||||
cases, decided exactly as apply_overrides() decides them:
|
||||
|
||||
- an ACTIVE entry (an object whose value passes validate_value()) owns
|
||||
its field, whatever is written there;
|
||||
- a RETIRED entry restores `was` whenever the stored value equals its
|
||||
`value`, so writing that value would flip-flop every day.
|
||||
"""
|
||||
|
||||
def __init__(self, docs: dict[str, dict]):
|
||||
self._active: dict[str, set[str]] = {}
|
||||
self._retired: dict[str, dict[str, tuple]] = {}
|
||||
for appid, doc in docs.items():
|
||||
for field, entry in (doc.get("fields") or {}).items():
|
||||
if isinstance(entry, dict) and validate_value(field, entry.get("value")) is None:
|
||||
self._active.setdefault(appid, set()).add(field)
|
||||
for field, entry in (doc.get("retired") or {}).items():
|
||||
if isinstance(entry, dict) and field in MANUAL_FIELDS:
|
||||
self._retired.setdefault(appid, {})[field] = (entry.get("value"), entry.get("was"))
|
||||
|
||||
@classmethod
|
||||
def load(cls) -> "OverrideGuard":
|
||||
return cls(load_overrides())
|
||||
|
||||
def allows(self, appid: str, field: str, value) -> bool:
|
||||
if field in self._active.get(str(appid), ()):
|
||||
return False
|
||||
retired = self._retired.get(str(appid), {}).get(field)
|
||||
return not (retired and retired[0] == value and retired[1] != value)
|
||||
|
||||
|
||||
def apply_overrides(games: list[dict], quiet: bool = False) -> ApplyStats:
|
||||
"""Re-impose every human decision onto `games`, in place.
|
||||
|
||||
@@ -212,11 +251,11 @@ def apply_overrides(games: list[dict], quiet: bool = False) -> ApplyStats:
|
||||
continue
|
||||
|
||||
if field == "notes":
|
||||
# Translated as save_main() translates the stored notes: an
|
||||
# override still saying "Có DLC trả phí" would otherwise put it
|
||||
# back after every normalisation, bump last_updated and commit
|
||||
# on every run.
|
||||
value = preserve_machine_notes(game.get("notes", ""), translate_notes(value))
|
||||
# Cleaned as save_main() cleans the stored notes: an override
|
||||
# still starting "Has paid DLC" would otherwise put it back
|
||||
# after every normalisation, bump last_updated and commit on
|
||||
# every run.
|
||||
value = preserve_machine_notes(game.get("notes", ""), clean_notes(value))
|
||||
|
||||
stats.enforced += 1
|
||||
if game.get(field) != value:
|
||||
@@ -230,7 +269,9 @@ def apply_overrides(games: list[dict], quiet: bool = False) -> ApplyStats:
|
||||
# Restore only while the stored value is still the one this
|
||||
# override put there. Once restored the two differ, so every later
|
||||
# pass is a no-op and the entry can stay in the file as history.
|
||||
if game.get(field) == entry.get("value"):
|
||||
# An entry whose `was` IS its value restores nothing, and must not
|
||||
# count as a change: it would bump last_updated on every save.
|
||||
if game.get(field) == entry.get("value") and entry.get("was") != entry.get("value"):
|
||||
game[field] = entry.get("was")
|
||||
stats.restored += 1
|
||||
touched = True
|
||||
|
||||
@@ -10,6 +10,8 @@ Text is HTML-unescaped before it is stored: the tag regex used to keep
|
||||
import re
|
||||
from html import unescape as _unescape
|
||||
|
||||
from .anticheat import parse_page
|
||||
|
||||
# ──── Pre-compiled patterns ────
|
||||
|
||||
_RE_LANG_TABLE = re.compile(
|
||||
@@ -174,11 +176,13 @@ def parse_tags(html: str) -> list[str]:
|
||||
# ──── Combined ────
|
||||
|
||||
def scrape_store_page(html: str) -> dict:
|
||||
"""Parse languages + DLC pricing + tags from one HTML string."""
|
||||
"""Parse languages + DLC pricing + tags + the anti-cheat disclosure
|
||||
(core/anticheat.py; None when the page has none) from one HTML string."""
|
||||
lang = parse_language_table(html)
|
||||
return {
|
||||
"languages": lang["languages"],
|
||||
"language_details": lang["language_details"],
|
||||
"has_paid_dlc": parse_has_paid_dlc(html),
|
||||
"tags": parse_tags(html),
|
||||
"anti_cheat": parse_page(html),
|
||||
}
|
||||
@@ -216,38 +216,76 @@ class SteamClient:
|
||||
_, html = self.fetch_store_page_full(appid)
|
||||
return html
|
||||
|
||||
def _store_items_batch(self, ids: list[str], data_request: dict) -> Optional[list[dict]]:
|
||||
"""One GetItems request: the items Steam knows (success 1), or None if
|
||||
the request failed. An id Steam does not know comes back as success 15
|
||||
with appid 0."""
|
||||
request = {
|
||||
"ids": [{"appid": int(a)} for a in ids],
|
||||
"context": {"language": "english", "country_code": "US", "steam_realm": 1},
|
||||
"data_request": data_request,
|
||||
}
|
||||
resp = self._get(
|
||||
STORE_ITEMS_URL,
|
||||
params={"input_json": json.dumps(request)},
|
||||
throttle_fn=self._throttle_api,
|
||||
)
|
||||
if not resp or resp.status_code != 200:
|
||||
return None
|
||||
try:
|
||||
items = resp.json()["response"].get("store_items") or []
|
||||
except (ValueError, KeyError, AttributeError):
|
||||
return None
|
||||
return [i for i in items if isinstance(i, dict) and i.get("success") == 1 and i.get("id")]
|
||||
|
||||
def fetch_store_items(self, appids) -> Optional[dict[str, dict]]:
|
||||
"""IStoreBrowseService/GetItems: {appid: item} for the ids Steam knows.
|
||||
|
||||
Keyless, and it answers for games whose store page needs a login. An id
|
||||
Steam does not know comes back as success 15 with appid 0, so items are
|
||||
keyed by the id that was asked for and kept only with success 1. None
|
||||
if any request failed, so a partial answer is never read as a whole one.
|
||||
Keyless, and it answers for games whose store page needs a login. Items
|
||||
are keyed by the id that was asked for. Tags, languages and the
|
||||
anti-cheat disclosure (extra_details, core/anticheat.py) are asked for.
|
||||
None if any request failed, so a partial answer is never read as a
|
||||
whole one.
|
||||
"""
|
||||
ids = [str(a) for a in appids if str(a).isdigit()]
|
||||
out: dict[str, dict] = {}
|
||||
request = {"include_tag_count": 20, "include_supported_languages": True,
|
||||
"include_extra_details": True}
|
||||
for start in range(0, len(ids), STORE_ITEMS_BATCH):
|
||||
request = {
|
||||
"ids": [{"appid": int(a)} for a in ids[start:start + STORE_ITEMS_BATCH]],
|
||||
"context": {"language": "english", "country_code": "US", "steam_realm": 1},
|
||||
"data_request": {"include_tag_count": 20, "include_supported_languages": True},
|
||||
}
|
||||
resp = self._get(
|
||||
STORE_ITEMS_URL,
|
||||
params={"input_json": json.dumps(request)},
|
||||
throttle_fn=self._throttle_api,
|
||||
)
|
||||
if not resp or resp.status_code != 200:
|
||||
return None
|
||||
try:
|
||||
items = resp.json()["response"].get("store_items") or []
|
||||
except (ValueError, KeyError, AttributeError):
|
||||
items = self._store_items_batch(ids[start:start + STORE_ITEMS_BATCH], request)
|
||||
if items is None:
|
||||
return None
|
||||
for item in items:
|
||||
if isinstance(item, dict) and item.get("success") == 1 and item.get("id"):
|
||||
out[str(item["id"])] = item
|
||||
out[str(item["id"])] = item
|
||||
return out
|
||||
|
||||
def fetch_anticheat_items(self, appids, give_up_after: int = 10) -> tuple[dict[str, dict], set[str]]:
|
||||
"""GetItems asking only for extra_details - each game's anti-cheat
|
||||
disclosure - for a whole catalogue: ({appid: item}, ids not answered).
|
||||
|
||||
Unlike fetch_store_items() a failed batch costs only its own ids, which
|
||||
the caller must treat as unknown, never as "no disclosure". After
|
||||
`give_up_after` failed batches in a row the rest is not asked at all.
|
||||
"""
|
||||
ids = [str(a) for a in appids if str(a).isdigit()]
|
||||
out: dict[str, dict] = {}
|
||||
failed: set[str] = set()
|
||||
in_a_row = 0
|
||||
for start in range(0, len(ids), STORE_ITEMS_BATCH):
|
||||
chunk = ids[start:start + STORE_ITEMS_BATCH]
|
||||
if in_a_row >= give_up_after:
|
||||
failed.update(chunk)
|
||||
continue
|
||||
items = self._store_items_batch(chunk, {"include_extra_details": True})
|
||||
if items is None:
|
||||
failed.update(chunk)
|
||||
in_a_row += 1
|
||||
continue
|
||||
in_a_row = 0
|
||||
for item in items:
|
||||
out[str(item["id"])] = item
|
||||
return out, failed
|
||||
|
||||
def fetch_tag_names(self) -> dict[int, str]:
|
||||
"""{tagid: English name} from IStoreService/GetTagList, fetched once per
|
||||
client. A failure is not cached, and returns {} (no tag is named)."""
|
||||
|
||||
@@ -20,6 +20,8 @@ Everything here is pure; steam_client.py makes the requests.
|
||||
"""
|
||||
from typing import Iterable, Optional
|
||||
|
||||
from .anticheat import from_store_item
|
||||
|
||||
# Steam's ELanguage ids, named the way the store page names them. Derived on
|
||||
# 2026-10-10 by pairing GetItems' rows with stored HTML scrapes of the same games
|
||||
# (row order is the same; one game lists all 102 languages). 29 never appeared.
|
||||
@@ -83,6 +85,7 @@ def parse_store_item(item: dict, tag_names: dict[int, str]) -> dict:
|
||||
what tells apply_scraped()/apply_store_page() to leave the stored value
|
||||
alone. Rows follow the page's rules - a language with none of the three
|
||||
columns ticked is not listed, a tag at most once (case-insensitively).
|
||||
anti_cheat is the developer's disclosure (core/anticheat.py), None for none.
|
||||
"""
|
||||
languages, details = [], []
|
||||
for row in item.get("supported_languages") or []:
|
||||
@@ -113,6 +116,7 @@ def parse_store_item(item: dict, tag_names: dict[int, str]) -> dict:
|
||||
"language_details": details,
|
||||
"tags": tags,
|
||||
"has_paid_dlc": None,
|
||||
"anti_cheat": from_store_item(item),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -5,8 +5,8 @@ Remove games from the catalogue - the command-line twin of /admin/remove.
|
||||
Removing a game is not a one-time deletion. It adds a line to
|
||||
data/blocklist.jsonl, a standing instruction that save_main() re-imposes on
|
||||
every write (core/blocklist.py): the game leaves data/, is logged once in
|
||||
scripts/removed_games.jsonl, and nothing - the browser extension, a refetch,
|
||||
the discovery sweep - can bring it back. Restoring is done in /admin (Remove ->
|
||||
scripts/removed_games.jsonl, and nothing - a queued link, a refetch, the
|
||||
discovery sweep - can bring it back. Restoring is done in /admin (Remove ->
|
||||
Removed -> Restore), because it also has to clear the review queue's record.
|
||||
|
||||
python scripts/delete_game.py 730 --reason not_a_game
|
||||
|
||||
+20
-21
@@ -9,14 +9,15 @@ import sys, os
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from core.blocklist import blocked_appids
|
||||
from core.constants import REMOVED_JSONL, MANUAL_FIELDS
|
||||
from core.constants import REMOVED_JSONL
|
||||
from core.data_store import (
|
||||
load_main, save_main, load_temp, clear_temp,
|
||||
load_jsonl, save_jsonl, dedup_removed,
|
||||
normalize_link, extract_appid, build_index,
|
||||
make_skeleton, merge_extension_data, now_iso, is_empty,
|
||||
make_skeleton, now_iso,
|
||||
)
|
||||
from core.fetcher import fetch_full
|
||||
from core.overrides import OverrideGuard
|
||||
from core.steam_client import get_client
|
||||
from core.health_checker import check_game_health, NETWORK_ERROR
|
||||
|
||||
@@ -34,12 +35,16 @@ def _link_text(value) -> str:
|
||||
|
||||
|
||||
def parse_entries(records):
|
||||
"""The queued links, as [{"link": text}]. A line is a store link and
|
||||
nothing else: /admin writes {"link": url}, and any other key on a line -
|
||||
the retired browser extension sent a whole record - is ignored, so a
|
||||
queued line can never decide a field. The game comes from Steam."""
|
||||
entries = []
|
||||
for rec in records:
|
||||
if isinstance(rec, (str, int)) and not isinstance(rec, bool):
|
||||
entries.append({"link": _link_text(rec)})
|
||||
elif isinstance(rec, dict) and rec.get("link") not in (None, ""):
|
||||
entries.append({**rec, "link": _link_text(rec["link"])})
|
||||
entries.append({"link": _link_text(rec["link"])})
|
||||
return entries
|
||||
|
||||
|
||||
@@ -60,6 +65,9 @@ def main():
|
||||
added = skipped_dup = skipped_bad = skipped_net = skipped_blocked = 0
|
||||
rejected = []
|
||||
client = get_client()
|
||||
# A restored game keeps its overrides: fetch_full() must not write a
|
||||
# manual field one of them would undo.
|
||||
guard = OverrideGuard.load()
|
||||
|
||||
for entry in entries:
|
||||
link = normalize_link(entry.get("link", ""))
|
||||
@@ -82,7 +90,7 @@ def main():
|
||||
|
||||
# Health check (also fetches appdetails → cached in result.data)
|
||||
print(f" 🔍 {appid}...", end=" ")
|
||||
health = check_game_health(link, entry.get("name", ""), client)
|
||||
health = check_game_health(link, "", client)
|
||||
|
||||
if health.status == NETWORK_ERROR:
|
||||
print("⚠ network – skip")
|
||||
@@ -96,10 +104,10 @@ def main():
|
||||
skipped_bad += 1
|
||||
continue
|
||||
# The catalogue is games. Discovery only proposes appdetails type
|
||||
# "game", but this queue is also fed by the extension and - with
|
||||
# auto-approve on - by candidates nobody looked at, so a free DLC, demo
|
||||
# or tool is refused HERE, where Steam's own answer is in hand. Not in
|
||||
# health_checker: Purge uses it, and would start deleting records.
|
||||
# "game", but this queue also takes a link a pull request adds and -
|
||||
# with auto-approve on - candidates nobody looked at, so a free DLC,
|
||||
# demo or tool is refused HERE, where Steam's own answer is in hand.
|
||||
# Not in health_checker: Purge uses it, and would start deleting records.
|
||||
app_type = str((health.data or {}).get("type") or "").strip().lower()
|
||||
if app_type and app_type != "game":
|
||||
print(f"❌ not a game (Steam type: {app_type})")
|
||||
@@ -110,21 +118,12 @@ def main():
|
||||
continue
|
||||
print("✅")
|
||||
|
||||
# Build record
|
||||
# Build the record from Steam (reusing the health check's appdetails,
|
||||
# so no second call). Human values arrive through overrides, which
|
||||
# save_main() re-imposes.
|
||||
game = make_skeleton(link)
|
||||
merge_extension_data(game, entry)
|
||||
|
||||
# Save manual overrides
|
||||
overrides = {k: game[k] for k in MANUAL_FIELDS
|
||||
if not is_empty(game.get(k))}
|
||||
|
||||
# Fetch (reuse health check data → no second appdetails call)
|
||||
print(f" ➕ Fetching ({appid})...")
|
||||
fetch_full(game, client=client, prefetched_details=health.data)
|
||||
|
||||
# Re-apply overrides
|
||||
for k, v in overrides.items():
|
||||
game[k] = v
|
||||
fetch_full(game, client=client, prefetched_details=health.data, guard=guard)
|
||||
|
||||
games.append(game)
|
||||
index[appid] = len(games) - 1
|
||||
|
||||
+8
-19
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Force re-fetch ALL games. MANUAL ONLY.
|
||||
Backup → migrate → clear fetchable + ephemeral → re-fetch → cleanup → save.
|
||||
Backup → migrate → clear fetchable → re-fetch → save.
|
||||
"""
|
||||
import sys, os, shutil
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
@@ -9,6 +9,7 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from core.constants import DATA_DIR, STEAM_API_KEY
|
||||
from core.data_store import load_main, save_main, migrate_record, now_iso
|
||||
from core.fetcher import fetch_full, process_batch
|
||||
from core.overrides import OverrideGuard
|
||||
from core.steam_client import get_client
|
||||
|
||||
CLEARABLE = {
|
||||
@@ -18,6 +19,9 @@ CLEARABLE = {
|
||||
"drm_notes", "has_paid_dlc",
|
||||
"platforms", "languages", "language_details", "tags",
|
||||
}
|
||||
# Manual fields holding a scraper default rather than a decision: reset to the
|
||||
# first value (the skeleton's), so the re-fetch fills them again. anti_cheat
|
||||
# stays "-": it used to become "", which nothing else ever writes.
|
||||
CONDITIONAL = {
|
||||
"genre": ("", "N/A", "Uncategorized"),
|
||||
"anti_cheat": ("-",),
|
||||
@@ -26,10 +30,6 @@ CONDITIONAL = {
|
||||
}
|
||||
_LIST_FIELDS = {"developer", "publisher", "platforms", "languages", "language_details", "tags"}
|
||||
|
||||
# Extension-only ephemeral fields that should never be in data.jsonl
|
||||
_EPHEMERAL = ("free_type", "is_free", "is_dlc", "is_demo",
|
||||
"is_playtest", "price", "auto_notes", "desc")
|
||||
|
||||
|
||||
def clear_fetchable(game):
|
||||
"""Reset API-fetchable fields to empty. Preserves manual fields."""
|
||||
@@ -42,22 +42,12 @@ def clear_fetchable(game):
|
||||
game[f] = ""
|
||||
for f, defaults in CONDITIONAL.items():
|
||||
if game.get(f) in defaults:
|
||||
game[f] = defaults[0] if (isinstance(defaults[0], bool) or defaults[0] is None) else ""
|
||||
game[f] = defaults[0]
|
||||
game["reviews"] = "N/A"
|
||||
game["current_players"] = "N/A"
|
||||
game["peak_today"] = "N/A"
|
||||
game["metacritic"] = "N/A"
|
||||
|
||||
# Remove ephemeral fields from old data
|
||||
for k in _EPHEMERAL:
|
||||
game.pop(k, None)
|
||||
|
||||
|
||||
def cleanup_record(game):
|
||||
"""Final cleanup before save – remove any ephemeral fields that leaked."""
|
||||
for k in _EPHEMERAL:
|
||||
game.pop(k, None)
|
||||
|
||||
|
||||
def main():
|
||||
games = load_main()
|
||||
@@ -83,12 +73,11 @@ def main():
|
||||
print("⚠ No STEAM_API_KEY – players skipped")
|
||||
|
||||
client = get_client()
|
||||
process_batch(games, lambda g: fetch_full(g, client=client), "Re-fetching")
|
||||
guard = OverrideGuard.load()
|
||||
process_batch(games, lambda g: fetch_full(g, client=client, guard=guard), "Re-fetching")
|
||||
|
||||
# Final cleanup + timestamp
|
||||
stamp = now_iso()
|
||||
for g in games:
|
||||
cleanup_record(g)
|
||||
g["last_updated"] = stamp
|
||||
save_main(games)
|
||||
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Refresh every game's anti-cheat from what its developer declares on Steam. Daily.
|
||||
|
||||
core/anticheat.py explains where Steam shows the disclosure and the rule that
|
||||
applies it. Each run:
|
||||
|
||||
1. asks GetItems (IStoreBrowseService, keyless) for the disclosure of EVERY
|
||||
game, 50 at a time: about 160 requests, a minute and a half;
|
||||
2. reads the store page of a disclosed game only when GetItems leaves out its
|
||||
name - a developer's own wording, which Steam keeps to the page (PUBG's
|
||||
"KSS (Krafton Security Services)");
|
||||
3. applies the rule with Steam winning: an empty game is filled, and a value
|
||||
Steam contradicts (another product, the other kernel flag) is replaced.
|
||||
|
||||
The rules:
|
||||
- No disclosure never clears anything: Warframe, Overwatch 2 and NARAKA use
|
||||
an anti-cheat and declare none. A disclosure that names nothing, not even
|
||||
on the page, changes nothing either, and is listed.
|
||||
- An override wins (core/overrides.py) and is not fought: a field it covers
|
||||
is not written, and neither is a retired entry's value, which
|
||||
apply_overrides() would undo on every run.
|
||||
- A batch GetItems did not answer leaves its games alone: unknown, not "no
|
||||
disclosure". After ten failed batches in a row the rest is not asked and
|
||||
the run ends with a warning, not a failure; with no answer at all nothing
|
||||
is written.
|
||||
- last_updated moves only for a game whose fields changed, so a run that
|
||||
learns nothing writes byte-identical shards. Everything goes through
|
||||
save_main().
|
||||
|
||||
It runs at the end of the daily Refresh Store Data job (bash/refresh_store.sh)
|
||||
and is committed on its own; that workflow's `only: anticheat` runs just this.
|
||||
|
||||
Usage (from the repository root):
|
||||
python scripts/refresh_anticheat.py every game
|
||||
python scripts/refresh_anticheat.py --appid 2507950 578080
|
||||
python scripts/refresh_anticheat.py --limit 50 --dry-run
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from collections import Counter
|
||||
from typing import Optional
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from core.anticheat import FIELDS, apply_anti_cheat, combine, from_store_item, parse_page
|
||||
from core.data_store import extract_appid, is_empty, load_main, migrate_record, now_iso, save_main
|
||||
from core.normalize import normalize_records
|
||||
from core.overrides import OverrideGuard
|
||||
from core.steam_client import get_client
|
||||
|
||||
GIVE_UP_AFTER = 10 # failed GetItems batches in a row
|
||||
|
||||
|
||||
def _kernel(value) -> str:
|
||||
return {True: "kernel", False: "not kernel"}.get(value, "kernel unknown")
|
||||
|
||||
|
||||
def sweep(targets: list[dict], client, guard, log=print) -> tuple[Counter, list[str]]:
|
||||
"""Apply Steam's disclosures to `targets` in place. Returns a tally -
|
||||
disclosed, none, unknown, pages, filled, replaced, updated, unnamed - and
|
||||
the unnamed games, as "appid name"."""
|
||||
by_id: dict[str, dict] = {}
|
||||
for game in targets:
|
||||
appid = extract_appid(game.get("link", ""))
|
||||
if appid:
|
||||
by_id[appid] = game
|
||||
items, failed = client.fetch_anticheat_items(list(by_id), give_up_after=GIVE_UP_AFTER)
|
||||
tally: Counter = Counter(unknown=len(failed))
|
||||
unnamed: list[str] = []
|
||||
|
||||
for appid, game in by_id.items():
|
||||
if appid in failed:
|
||||
continue
|
||||
api = from_store_item(items.get(appid))
|
||||
if api is None:
|
||||
tally["none"] += 1
|
||||
continue
|
||||
tally["disclosed"] += 1
|
||||
found = api
|
||||
if not api.name:
|
||||
# GetItems keeps a developer's own wording to the page.
|
||||
tally["pages"] += 1
|
||||
status, page = client.fetch_store_page_full(appid)
|
||||
found = combine(api, parse_page(page) if status == "ok" else None)
|
||||
if not found.name:
|
||||
tally["unnamed"] += 1
|
||||
unnamed.append(f"{appid} {game.get('name', '')[:40]}")
|
||||
continue
|
||||
|
||||
before = {f: game.get(f) for f in FIELDS}
|
||||
changed = apply_anti_cheat(game, found, replace=True, guard=guard)
|
||||
if not changed:
|
||||
continue
|
||||
game["last_updated"] = now_iso()
|
||||
if is_empty(before["anti_cheat"]):
|
||||
kind = "filled"
|
||||
elif "anti_cheat" in changed:
|
||||
kind = "replaced"
|
||||
else:
|
||||
kind = "updated"
|
||||
tally[kind] += 1
|
||||
log(f" {appid} {game.get('name', '')[:40]}: {before['anti_cheat'] or '-'} "
|
||||
f"({_kernel(before['is_kernel_ac'])}) -> {game['anti_cheat']} "
|
||||
f"({_kernel(game.get('is_kernel_ac'))}) [{', '.join(changed)}]")
|
||||
return tally, unnamed
|
||||
|
||||
|
||||
def main(argv: Optional[list[str]] = None) -> None:
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0].strip())
|
||||
ap.add_argument("--appid", nargs="+", help="only these games")
|
||||
ap.add_argument("--limit", type=int, help="at most this many games (for trying it out)")
|
||||
ap.add_argument("--dry-run", action="store_true", help="ask Steam and report, write nothing")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
games = load_main()
|
||||
for g in games:
|
||||
migrate_record(g)
|
||||
# Compared as stored, like refresh_store_data.py.
|
||||
normalize_records(games)
|
||||
|
||||
targets = [g for g in games if extract_appid(g.get("link", ""))]
|
||||
if args.appid:
|
||||
wanted = {str(a) for a in args.appid}
|
||||
targets = [g for g in targets if extract_appid(g["link"]) in wanted]
|
||||
if args.limit is not None:
|
||||
targets = targets[: args.limit]
|
||||
print(f"{len(games)} games; asking Steam about the anti-cheat of {len(targets)}")
|
||||
if not targets:
|
||||
return
|
||||
|
||||
tally, unnamed = sweep(targets, get_client(), OverrideGuard.load())
|
||||
print(f"\nSteam: {tally['disclosed']} disclose an anti-cheat, {tally['none']} declare none, "
|
||||
f"{tally['unknown']} unknown; {tally['pages']} store pages read")
|
||||
if tally["unknown"]:
|
||||
print(f"::warning::GetItems did not answer for {tally['unknown']} games; "
|
||||
f"they keep their values until the next run.")
|
||||
changed = tally["filled"] + tally["replaced"] + tally["updated"]
|
||||
print(f"Anti-cheat: {changed} games changed (filled {tally['filled']}, "
|
||||
f"replaced {tally['replaced']}, updated {tally['updated']})")
|
||||
if unnamed:
|
||||
print(f"{len(unnamed)} disclose an anti-cheat without naming it: {', '.join(unnamed)}")
|
||||
|
||||
if args.dry_run:
|
||||
print("Dry run: nothing written.")
|
||||
return
|
||||
if tally["unknown"] == len(targets):
|
||||
print("Steam answered for no game: nothing written.")
|
||||
return
|
||||
save_main(games)
|
||||
print(f"✓ Saved {len(games)} games")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,324 @@
|
||||
"""
|
||||
core/anticheat.py reads the anti-cheat a developer declares on Steam and is the
|
||||
one rule every writer applies it with. These pin how the store page and Steam's
|
||||
JSON are read (the heading decides kernel, a Boot Protection row is a flag, no
|
||||
other notice is an anti-cheat), how products are compared, and the rule: fill,
|
||||
replace only when asked, never clear, never fight an override, and a second
|
||||
pass changes nothing.
|
||||
|
||||
Run from the repository root: python -m pytest scripts/tests
|
||||
"""
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from core import anticheat as ac
|
||||
from core.anticheat import Disclosure, apply_anti_cheat
|
||||
from core.overrides import OverrideGuard
|
||||
|
||||
T = "\n\t\t\t\t\t\t\t\t\t\t"
|
||||
|
||||
|
||||
def _section(cls: str, heading: str, *rows: tuple[str, str]) -> str:
|
||||
"""The block as Steam wrote it on 2026-10-16 (Apex Legends, CoD), tabs and all."""
|
||||
names = "".join(f'{T}<div class="anticheat_name">{name}<span class="anticheat_uninstalls">{note}</span></div>'
|
||||
for name, note in rows)
|
||||
return (f'<div class="DRM_notice">Requires 3rd-Party Account: EA Account</div>'
|
||||
f'{T}<div class="anticheat_section {cls}" >{T}<div>{heading}</div>{names}{T}</div>'
|
||||
f'{T}<div class="DRM_notice"data-panel="{{}}" role="button" >{T}'
|
||||
f'<div>Requires agreement to a 3rd-party EULA</div></div>')
|
||||
|
||||
|
||||
KERNEL = "Uses Kernel Level Anti-Cheat"
|
||||
SOFTWARE = "Uses Anti-Cheat Software"
|
||||
|
||||
|
||||
# ──────────── The store page ────────────
|
||||
|
||||
def test_a_kernel_section_and_its_uninstall_span():
|
||||
page = _section("DRM_notice", KERNEL, ("Easy Anti-Cheat", " - Requires manual removal after game uninstall"))
|
||||
assert ac.parse_page(page) == Disclosure("Easy Anti-Cheat", True, False)
|
||||
|
||||
|
||||
def test_kernel_comes_from_the_heading_not_the_class():
|
||||
# Jujutsu Kaisen: DRM_notice, yet "Uses Anti-Cheat Software" - its boot
|
||||
# protection is what earns the class.
|
||||
page = _section("DRM_notice", SOFTWARE, ("Fairguard", ""), ("Boot Protection", " - Requires both Secure Boot & TPM 2.0"))
|
||||
assert ac.parse_page(page) == Disclosure("Fairguard", False, True)
|
||||
page = _section("anticheat_nonkernel_notice", KERNEL, ("VAC (Valve Anti-Cheat)", ""))
|
||||
assert ac.parse_page(page).kernel is True
|
||||
page = _section("DRM_notice", "Dùng chống gian lận cấp độ kernel", ("Easy Anti-Cheat", ""))
|
||||
assert ac.parse_page(page).kernel is None # a heading it cannot read
|
||||
|
||||
|
||||
def test_boot_protection_is_a_flag_not_a_name():
|
||||
page = _section("DRM_notice", KERNEL, ("RICOCHET Anti-Cheat", ""),
|
||||
("Boot Protection", " - Requires both Secure Boot & TPM 2.0"))
|
||||
assert ac.parse_page(page) == Disclosure("RICOCHET Anti-Cheat", True, True)
|
||||
|
||||
|
||||
def test_names_are_cleaned_and_an_empty_one_stays_empty():
|
||||
assert ac.parse_page(_section("DRM_notice", KERNEL, ("KSS (Krafton Security Services) ", ""))).name == \
|
||||
"KSS (Krafton Security Services)"
|
||||
assert ac.parse_page(_section("DRM_notice", KERNEL, ("Stand & Deliver AC", ""))).name == "Stand & Deliver AC"
|
||||
assert ac.parse_page(_section("anticheat_nonkernel_notice", SOFTWARE, ("", ""))) == Disclosure("", False, False)
|
||||
|
||||
|
||||
def test_other_notices_prose_and_deck_json_are_not_anti_cheat():
|
||||
page = ('<div class="DRM_notice"><div>Requires 3rd-Party Account: EA Account</div></div>'
|
||||
'<div class="game_area_description">Our new anti-cheat technologies, kernel level.</div>'
|
||||
'<script>{"SteamDeckVerified_TestResult_UnsupportedAntiCheat_Other": 1}</script>'
|
||||
'<img src="anti_cheats_1202_ru.gif">')
|
||||
assert ac.parse_page(page) is None
|
||||
assert ac.parse_page("") is None and ac.parse_page(None) is None
|
||||
|
||||
|
||||
# ──────────── Steam's JSON ────────────
|
||||
|
||||
def _item(anticheat=None, extra=True) -> dict:
|
||||
item = {"id": 1, "success": 1}
|
||||
if extra:
|
||||
item["extra_details"] = {} if anticheat is None else {"anticheat": anticheat}
|
||||
return item
|
||||
|
||||
|
||||
def test_store_item_disclosures():
|
||||
# Measured 2026-10-16.
|
||||
delta = {"kernel_mode": True, "uninstall_completely": True,
|
||||
"name": "Anti-Cheat Expert (ACE)", "name_loc": "#Store_AntiCheat_Type_ace"}
|
||||
assert ac.from_store_item(_item(delta)) == Disclosure("Anti-Cheat Expert (ACE)", True, False)
|
||||
assert ac.from_store_item(_item({"kernel_mode": True, "uninstall_completely": True})) == \
|
||||
Disclosure("", True, False) # PUBG
|
||||
assert ac.from_store_item(_item({})) == Disclosure("", False, False) # Marvel Rivals
|
||||
assert ac.from_store_item(_item({"name": "VAC (Valve Anti-Cheat)"})) == \
|
||||
Disclosure("VAC (Valve Anti-Cheat)", False, False) # CS2
|
||||
cod = {"kernel_mode": True, "boot_protection": True, "boot_protection_name": "secureboot_tpm2"}
|
||||
assert ac.from_store_item(_item(cod)) == Disclosure("", True, True)
|
||||
assert ac.from_store_item(_item()) is None # Warframe
|
||||
assert ac.from_store_item(_item(extra=False)) is None
|
||||
assert ac.from_store_item(None) is None
|
||||
|
||||
|
||||
def test_the_page_names_what_the_json_leaves_out():
|
||||
api = Disclosure("", True, True)
|
||||
page = Disclosure("RICOCHET Anti-Cheat", True, False)
|
||||
assert ac.combine(api, page) == Disclosure("RICOCHET Anti-Cheat", True, True)
|
||||
assert ac.combine(api, None) == api
|
||||
assert ac.combine(None, page) == page
|
||||
named = Disclosure("Easy Anti-Cheat", True, False)
|
||||
assert ac.combine(named, Disclosure("", False, False)) == named # JSON decides kernel
|
||||
|
||||
|
||||
def test_category_vac_in_any_position():
|
||||
cats = [{"id": 1, "description": "Multi-player"}, {"id": 2, "description": "Single-player"},
|
||||
{"id": 8, "description": "Valve Anti-Cheat enabled"}]
|
||||
assert ac.from_categories(cats) == ac.VAC
|
||||
assert ac.from_categories([{"description": "Valve Anti-Cheat enabled"}]) == ac.VAC
|
||||
assert ac.from_categories([{"id": 1, "description": "Multi-player"}]) is None
|
||||
assert ac.from_categories(None) is None
|
||||
|
||||
|
||||
# ──────────── Labels ────────────
|
||||
|
||||
@pytest.mark.parametrize("name, label", [
|
||||
("Easy Anti-Cheat", "EAC"),
|
||||
("VAC (Valve Anti-Cheat)", "VAC"),
|
||||
("VAC (Valve Anti-Cheat); additional custom server-side anti-cheat on dedicated servers", "VAC"),
|
||||
("BattlEye", "BattlEye"),
|
||||
("nProtect GameGuard", "nProtect"),
|
||||
("XIGNCODE3", "XIGNCODE"),
|
||||
("KSS (Krafton Security Services)", "KSS"),
|
||||
("NetEase Game Security", "NetEase GS"),
|
||||
("RICOCHET Anti-Cheat", "Ricochet"),
|
||||
("Denuvo", "Denuvo AC"),
|
||||
("FACEIT Anti-Cheat", "FACEIT AC"),
|
||||
# Not a known product: Steam's name, as it is.
|
||||
("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE)"),
|
||||
("EA Javelin Anticheat", "EA Javelin Anticheat"),
|
||||
("HoYoKProtect", "HoYoKProtect"),
|
||||
])
|
||||
def test_labels(name, label):
|
||||
assert ac.label(name) == label
|
||||
|
||||
|
||||
FRONTIER = ("Frontier uses SCUE4 as its anti-cheat, which comes pre-built with the game. It operates "
|
||||
"at the user level and scans surface memory for unauthorized injections or cheat "
|
||||
"applications affecting the game. It does not install a separate anti-cheat client.")
|
||||
|
||||
|
||||
def test_a_paragraph_where_the_name_goes_is_cut_on_a_word():
|
||||
# Frontier: Battle Royale, 2026-10-16. The note keeps all of it.
|
||||
cut = ac.label(FRONTIER)
|
||||
assert cut == "Frontier uses SCUE4 as its anti-cheat…"
|
||||
assert len(cut) <= ac.LABEL_MAX
|
||||
assert ac.label(cut) == cut
|
||||
assert ac.same_family(cut, FRONTIER) and ac.same_family(FRONTIER, cut)
|
||||
assert not ac.same_family("Frontier uses SCUE4 as its anti-cheat…", "Frontier uses EAC")
|
||||
game = _game()
|
||||
apply_anti_cheat(game, Disclosure(FRONTIER, False))
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"]) == (cut, f"{FRONTIER} [Non-Kernel]")
|
||||
|
||||
|
||||
def test_products_are_compared_by_family():
|
||||
assert ac.same_family("EAC", "Easy Anti-Cheat")
|
||||
assert ac.same_family("Easy Anti-Cheat", "EAC")
|
||||
assert ac.same_family("nProtect", "nProtect GameGuard")
|
||||
assert ac.same_family("Krafton Security Services", "KSS (Krafton Security Services)")
|
||||
assert ac.same_family("ACE", "Anti-Cheat Expert (ACE)")
|
||||
assert ac.same_family("CrackProof®", "CrackProof")
|
||||
assert ac.same_family("网易游戏安全", "网易游戏安全")
|
||||
assert not ac.same_family("EAC", "Anti-Cheat Expert (ACE)")
|
||||
assert not ac.same_family("EAC", "BattlEye")
|
||||
assert not ac.same_family("FACEIT AC", "Anti-Cheat Expert (ACE)")
|
||||
|
||||
|
||||
def _stored_anti_cheats() -> list[str]:
|
||||
root = os.path.join(os.path.dirname(__file__), "..", "..", "data")
|
||||
out = set()
|
||||
for path in glob.glob(os.path.join(root, "data_*.jsonl")):
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
value = json.loads(line).get("anti_cheat") if line.strip() else None
|
||||
if isinstance(value, str) and value.strip() not in ("", "-"):
|
||||
out.add(value)
|
||||
return sorted(out)
|
||||
|
||||
|
||||
def test_every_stored_value_and_every_label_is_its_own_family():
|
||||
# What makes a second sweep a no-op: a value the sweep wrote, or one already
|
||||
# there, is the same product as the name it came from.
|
||||
values = _stored_anti_cheats() + ["EAC", "Anti-Cheat Expert (ACE)", "Server-side", "-x-"]
|
||||
for value in values:
|
||||
assert ac.same_family(value, value), value
|
||||
assert ac.same_family(ac.label(value), value), value
|
||||
|
||||
|
||||
# ──────────── The rule ────────────
|
||||
|
||||
def _game(**fields) -> dict:
|
||||
base = {"link": "https://store.steampowered.com/app/2507950/", "name": "Delta Force",
|
||||
"anti_cheat": "-", "anti_cheat_note": "", "is_kernel_ac": None}
|
||||
base.update(fields)
|
||||
return base
|
||||
|
||||
|
||||
ACE = Disclosure("Anti-Cheat Expert (ACE)", True, False)
|
||||
|
||||
|
||||
def test_an_empty_game_gets_all_three():
|
||||
game = _game()
|
||||
assert sorted(apply_anti_cheat(game, ACE)) == ["anti_cheat", "anti_cheat_note", "is_kernel_ac"]
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == \
|
||||
("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE) [Kernel]", True)
|
||||
game = _game(anti_cheat="")
|
||||
apply_anti_cheat(game, Disclosure("RICOCHET Anti-Cheat", True, True))
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"]) == ("Ricochet", "RICOCHET Anti-Cheat [Kernel]; Boot Protection")
|
||||
|
||||
|
||||
def test_nothing_found_or_nothing_named_changes_nothing():
|
||||
for found in (None, Disclosure("", True, False)):
|
||||
for game in (_game(), _game(anti_cheat="EAC", is_kernel_ac=None)):
|
||||
before = dict(game)
|
||||
assert apply_anti_cheat(game, found, replace=True) == []
|
||||
assert game == before
|
||||
|
||||
|
||||
def test_the_same_product_fills_and_follows_steam():
|
||||
# Marvel Rivals: named, kernel unknown, no note.
|
||||
game = _game(anti_cheat="NetEase Game Security")
|
||||
assert sorted(apply_anti_cheat(game, Disclosure("NetEase Game Security", False))) == \
|
||||
["anti_cheat_note", "is_kernel_ac"]
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == \
|
||||
("NetEase Game Security", "NetEase Game Security [Non-Kernel]", False)
|
||||
# Filling never moves a known flag; the sweep (replace) does.
|
||||
game = _game(anti_cheat="Fairguard", anti_cheat_note="Fairguard [Kernel]", is_kernel_ac=True)
|
||||
assert apply_anti_cheat(game, Disclosure("Fairguard", False, True)) == []
|
||||
assert sorted(apply_anti_cheat(game, Disclosure("Fairguard", False, True), replace=True)) == \
|
||||
["anti_cheat_note", "is_kernel_ac"]
|
||||
assert (game["anti_cheat_note"], game["is_kernel_ac"]) == ("Fairguard [Non-Kernel]", False)
|
||||
|
||||
|
||||
def test_a_kernel_change_rewrites_only_a_machine_suffix():
|
||||
game = _game(anti_cheat="EAC", anti_cheat_note="Kernel driver, uninstall it by hand", is_kernel_ac=False)
|
||||
assert apply_anti_cheat(game, Disclosure("Easy Anti-Cheat", True), replace=True) == ["is_kernel_ac"]
|
||||
assert game["anti_cheat_note"] == "Kernel driver, uninstall it by hand"
|
||||
|
||||
|
||||
def test_another_product_is_replaced_only_by_the_sweep():
|
||||
# Wuthering Waves: stored EAC, Steam says ACE.
|
||||
game = _game(anti_cheat="EAC", anti_cheat_note="EAC", is_kernel_ac=None)
|
||||
assert apply_anti_cheat(game, ACE) == []
|
||||
assert game["anti_cheat"] == "EAC"
|
||||
assert sorted(apply_anti_cheat(game, ACE, replace=True)) == ["anti_cheat", "anti_cheat_note", "is_kernel_ac"]
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == \
|
||||
("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE) [Kernel]", True)
|
||||
|
||||
|
||||
def test_a_known_label_stays_as_it_was_written():
|
||||
game = _game(anti_cheat="Easy Anti-Cheat", anti_cheat_note="Easy Anti-Cheat [Kernel]", is_kernel_ac=True)
|
||||
assert apply_anti_cheat(game, Disclosure("Easy Anti-Cheat", True), replace=True) == []
|
||||
|
||||
|
||||
def test_applying_twice_changes_nothing():
|
||||
for found in (ACE, Disclosure("RICOCHET Anti-Cheat", True, True), ac.VAC,
|
||||
Disclosure("VAC (Valve Anti-Cheat); additional custom server-side anti-cheat", False),
|
||||
Disclosure("hosted game servers", False), Disclosure("Fairguard", None),
|
||||
Disclosure(FRONTIER, False), Disclosure("K1LLSWITↃH", False)):
|
||||
for start in (_game(), _game(anti_cheat="EAC", anti_cheat_note="EAC", is_kernel_ac=True)):
|
||||
game = dict(start)
|
||||
apply_anti_cheat(game, found, replace=True)
|
||||
once = dict(game)
|
||||
assert apply_anti_cheat(game, found, replace=True) == [], found
|
||||
assert game == once
|
||||
|
||||
|
||||
# ──────────── Overrides ────────────
|
||||
|
||||
def _guard(fields=None, retired=None) -> OverrideGuard:
|
||||
return OverrideGuard({"2507950": {"fields": fields or {}, "retired": retired or {}}})
|
||||
|
||||
|
||||
def test_a_field_an_override_covers_is_not_written():
|
||||
guard = _guard({"is_kernel_ac": {"value": None}})
|
||||
game = _game()
|
||||
assert sorted(apply_anti_cheat(game, ACE, guard=guard)) == ["anti_cheat", "anti_cheat_note"]
|
||||
assert game["is_kernel_ac"] is None
|
||||
assert game["anti_cheat_note"] == "Anti-Cheat Expert (ACE)" # the suffix follows the locked null
|
||||
|
||||
|
||||
def test_a_locked_label_writes_nothing_for_another_product():
|
||||
guard = _guard({"anti_cheat": {"value": "BattlEye"}})
|
||||
game = _game(anti_cheat="BattlEye", is_kernel_ac=None)
|
||||
assert apply_anti_cheat(game, ACE, replace=True, guard=guard) == []
|
||||
guard = _guard({"anti_cheat": {"value": "-"}}) # an override says: none
|
||||
assert apply_anti_cheat(_game(), ACE, replace=True, guard=guard) == []
|
||||
# The same product with a locked label still gets its kernel flag and note.
|
||||
guard = _guard({"anti_cheat": {"value": "Anti-Cheat Expert (ACE)"}})
|
||||
game = _game(anti_cheat="Anti-Cheat Expert (ACE)")
|
||||
assert sorted(apply_anti_cheat(game, ACE, replace=True, guard=guard)) == ["anti_cheat_note", "is_kernel_ac"]
|
||||
|
||||
|
||||
def test_a_locked_kernel_flag_decides_the_suffix():
|
||||
guard = _guard({"is_kernel_ac": {"value": False}})
|
||||
game = _game(is_kernel_ac=False)
|
||||
apply_anti_cheat(game, ACE, replace=True, guard=guard)
|
||||
assert (game["is_kernel_ac"], game["anti_cheat_note"]) == (False, "Anti-Cheat Expert (ACE) [Non-Kernel]")
|
||||
|
||||
|
||||
def test_a_retired_value_is_never_written_back():
|
||||
# apply_overrides() restores `was` whenever the stored value equals the
|
||||
# retired one: writing it would flip-flop every day.
|
||||
guard = _guard(retired={"anti_cheat": {"value": "Anti-Cheat Expert (ACE)", "was": "-"}})
|
||||
assert apply_anti_cheat(_game(), ACE, replace=True, guard=guard) == []
|
||||
# Another value is fine, and so is an entry whose `was` is its value.
|
||||
guard = _guard(retired={"anti_cheat": {"value": "EAC", "was": "-"},
|
||||
"is_kernel_ac": {"value": True, "was": True}})
|
||||
assert "anti_cheat" in apply_anti_cheat(_game(), ACE, replace=True, guard=guard)
|
||||
|
||||
|
||||
def test_an_invalid_override_entry_guards_nothing():
|
||||
# apply_overrides() rejects these, so they do not own the field either.
|
||||
guard = _guard({"anti_cheat": {"value": ""}, "is_kernel_ac": "true"})
|
||||
assert sorted(apply_anti_cheat(_game(), ACE, guard=guard)) == ["anti_cheat", "anti_cheat_note", "is_kernel_ac"]
|
||||
@@ -0,0 +1,98 @@
|
||||
"""
|
||||
fetch_full() builds every new game (ingest_new.py) and refills incomplete ones
|
||||
(update_data.py, refetch_all.py). These pin how it treats the anti-cheat
|
||||
fields: filled from the developer's disclosure on the page it already fetched,
|
||||
else from appdetails' VAC category, never replaced, and never where an
|
||||
override would undo it.
|
||||
|
||||
Steam is never called: the client is a fake. Run from the repository root:
|
||||
python -m pytest scripts/tests
|
||||
"""
|
||||
from core import fetcher
|
||||
from core.data_store import make_skeleton
|
||||
from core.overrides import OverrideGuard
|
||||
|
||||
LINK = "https://store.steampowered.com/app/2507950/"
|
||||
DETAILS = {"name": "Delta Force", "type": "game", "is_free": True,
|
||||
"categories": [{"id": 1, "description": "Multi-player"}]}
|
||||
VAC_DETAILS = dict(DETAILS, categories=[{"id": 1, "description": "Multi-player"},
|
||||
{"id": 8, "description": "Valve Anti-Cheat enabled"}])
|
||||
|
||||
|
||||
def _page(name="Anti-Cheat Expert (ACE)", heading="Uses Kernel Level Anti-Cheat") -> str:
|
||||
section = (f'<div class="anticheat_section DRM_notice" ><div>{heading}</div>'
|
||||
f'<div class="anticheat_name">{name}<span class="anticheat_uninstalls"></span></div></div>')
|
||||
return ('<table class="game_language_options"><tr><th>x</th></tr><tr><td>English</td>'
|
||||
'<td><span>✔</span></td><td></td><td></td></tr></table>'
|
||||
'<a class="app_tag" href="#">FPS</a>' + section)
|
||||
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, page=None, status=None, items=None):
|
||||
self.page = page
|
||||
self.status = status or ("ok" if page is not None else "network_error")
|
||||
self.items = items
|
||||
|
||||
def fetch_app_details(self, appid):
|
||||
return dict(DETAILS)
|
||||
|
||||
def fetch_reviews(self, appid):
|
||||
return None
|
||||
|
||||
def fetch_store_page_full(self, appid):
|
||||
return (self.status, self.page if self.status == "ok" else None)
|
||||
|
||||
def fetch_store_items(self, appids):
|
||||
return self.items
|
||||
|
||||
def fetch_tag_names(self):
|
||||
return {}
|
||||
|
||||
|
||||
def _fetch(game=None, client=None, details=DETAILS, guard=None) -> dict:
|
||||
game = game or make_skeleton(LINK)
|
||||
return fetcher.fetch_full(game, client=client or FakeClient(page=_page()), fetch_players=False,
|
||||
prefetched_details=dict(details), guard=guard)
|
||||
|
||||
|
||||
def test_fetch_full_fills_from_the_page():
|
||||
game = _fetch()
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == \
|
||||
("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE) [Kernel]", True)
|
||||
|
||||
|
||||
def test_a_gated_game_is_filled_from_steam_s_json():
|
||||
items = {"2507950": {"id": 2507950, "success": 1, "extra_details": {
|
||||
"anticheat": {"kernel_mode": True, "name": "Easy Anti-Cheat"}}}}
|
||||
game = _fetch(client=FakeClient(status="age_gate", items=items))
|
||||
assert (game["anti_cheat"], game["is_kernel_ac"]) == ("EAC", True)
|
||||
|
||||
|
||||
def test_vac_comes_from_the_category_only_when_steam_declares_nothing():
|
||||
game = _fetch(client=FakeClient(page=_page().split('<div class="anticheat_section')[0]), details=VAC_DETAILS)
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == \
|
||||
("VAC", "VAC (Valve Anti-Cheat) [Non-Kernel]", False)
|
||||
# A disclosure wins over the category.
|
||||
assert _fetch(details=VAC_DETAILS)["anti_cheat"] == "Anti-Cheat Expert (ACE)"
|
||||
|
||||
|
||||
def test_fetch_full_never_replaces_another_product():
|
||||
game = make_skeleton(LINK)
|
||||
game.update(anti_cheat="EAC", anti_cheat_note="EAC", is_kernel_ac=False)
|
||||
_fetch(game)
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("EAC", "EAC", False)
|
||||
|
||||
|
||||
def test_apply_details_writes_no_anti_cheat():
|
||||
game = make_skeleton(LINK)
|
||||
fetcher.apply_details(game, VAC_DETAILS)
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("-", "", None)
|
||||
|
||||
|
||||
def test_the_guard_reaches_fetch_full():
|
||||
# A restored game keeps its override, here "nobody has verified the kernel
|
||||
# flag": filling it would be undone by save_main() on every run.
|
||||
guard = OverrideGuard({"2507950": {"fields": {"is_kernel_ac": {"value": None}}}})
|
||||
game = _fetch(guard=guard)
|
||||
assert game["anti_cheat"] == "Anti-Cheat Expert (ACE)"
|
||||
assert game["is_kernel_ac"] is None
|
||||
@@ -1,7 +1,8 @@
|
||||
"""
|
||||
ingest_new.py reads scripts/temp_info.jsonl, which two producers append to (the
|
||||
browser extension and /admin). These pin the lines that used to crash or
|
||||
slip through: a link that is not text, and a game the maintainer removed.
|
||||
ingest_new.py reads scripts/temp_info.jsonl, which /admin appends to (and a
|
||||
pull request may). A line is a store link and nothing else. These pin the lines
|
||||
that used to crash or slip through: a link that is not text, a line carrying
|
||||
fields of its own, and a game the maintainer removed.
|
||||
|
||||
Steam is never called: check_game_health and fetch_full are replaced.
|
||||
|
||||
@@ -33,7 +34,7 @@ def repo(tmp_path, monkeypatch):
|
||||
return HealthResult(OK, appid, f"Game {appid}", link,
|
||||
data=details if details is not None else {"type": "game", "name": f"Game {appid}"})
|
||||
|
||||
def fetch(game, client=None, prefetched_details=None):
|
||||
def fetch(game, client=None, prefetched_details=None, **_):
|
||||
game["name"] = (prefetched_details or {}).get("name", game.get("name", ""))
|
||||
|
||||
monkeypatch.setattr(ingest_new, "check_game_health", health)
|
||||
@@ -63,6 +64,24 @@ def test_parse_entries_turns_numbers_into_links_and_drops_nothing_silently():
|
||||
]
|
||||
|
||||
|
||||
def test_parse_entries_keeps_only_the_link():
|
||||
line = {"link": "730", "genre": "MOBA", "anti_cheat": "VAC", "notes": "x", "name": "Fake",
|
||||
"status": "removed", "last_updated": "2030-01-01T00:00:00Z"}
|
||||
assert ingest_new.parse_entries([line]) == [{"link": "730"}]
|
||||
|
||||
|
||||
def test_a_line_s_own_fields_never_reach_the_record(repo):
|
||||
# The retired browser extension sent whole records; its merge let one line
|
||||
# decide any field, a manual one or the record's status alike.
|
||||
_queue({"link": "730", "genre": "MOBA", "safe": "y", "notes": "CS2 babyyy",
|
||||
"status": "delisted", "is_dead": True})
|
||||
ingest_new.main()
|
||||
game = next(g for g in data_store.load_main() if g["link"].endswith("/730/"))
|
||||
assert game["name"] == "Game 730"
|
||||
assert game["genre"] == "" and game["safe"] == "?" and game["notes"] == ""
|
||||
assert game["status"] == "active" and game["is_dead"] is False
|
||||
|
||||
|
||||
def test_a_numeric_link_no_longer_crashes_the_run(repo):
|
||||
_queue({"link": 730}, {"link": ["not", "a", "link"]})
|
||||
ingest_new.main()
|
||||
|
||||
@@ -13,8 +13,8 @@ import pytest
|
||||
|
||||
from core import data_store
|
||||
from core.normalize import (
|
||||
clean_list, clean_text, decode_entities, decode_tags, normalize_record,
|
||||
normalize_release_date, translate_notes,
|
||||
clean_list, clean_notes, clean_text, decode_entities, decode_tags, normalize_record,
|
||||
normalize_release_date,
|
||||
)
|
||||
|
||||
|
||||
@@ -126,14 +126,33 @@ def test_tags_are_decoded_and_the_duplicates_that_makes_are_dropped():
|
||||
# ──────────── notes ────────────
|
||||
|
||||
@pytest.mark.parametrize("raw, expected", [
|
||||
("Có DLC trả phí", "Has paid DLC"),
|
||||
("Có DLC trả phí 💀 Dead game (no players ≥14d)", "Has paid DLC 💀 Dead game (no players ≥14d)"),
|
||||
("Có DLC trả phí; Easy Anti-Cheat [Kernel]", "Has paid DLC; Easy Anti-Cheat [Kernel]"),
|
||||
("Not reviewed yet", "Not reviewed yet"),
|
||||
("English only", "English only"),
|
||||
# Every shape the 517 notes the retired browser extension began came in.
|
||||
("Has paid DLC", "Not reviewed yet"),
|
||||
("[hover-add]", "Not reviewed yet"),
|
||||
("Has paid DLC 💀 Dead game (no players ≥14d)", "💀 Dead game (no players ≥14d)"),
|
||||
("Has paid DLC; Easy Anti-Cheat [Kernel]", "Easy Anti-Cheat [Kernel]"),
|
||||
("Has paid DLC; EAC", "EAC"),
|
||||
("Has paid DLC; Anti-Cheat Toolkit [Non-Kernel] ⚠ No longer free!",
|
||||
"Anti-Cheat Toolkit [Non-Kernel] ⚠ No longer free!"),
|
||||
])
|
||||
def test_the_extensions_vietnamese_notes_read_in_english(raw, expected):
|
||||
assert translate_notes(raw) == expected
|
||||
def test_the_extensions_leading_phrases_are_dropped(raw, expected):
|
||||
assert clean_notes(raw) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw", [
|
||||
"", "Not reviewed yet", "English only", "Has paid DLCs", "Toxic max; Has paid DLC",
|
||||
"has paid dlc", " Has paid DLC", "[hover-add]er", None,
|
||||
])
|
||||
def test_anything_else_is_kept_exactly(raw):
|
||||
# An empty note stays empty: only a note the rule emptied becomes
|
||||
# "Not reviewed yet". A phrase in the middle is a person's sentence.
|
||||
assert clean_notes(raw) == raw
|
||||
|
||||
|
||||
def test_clean_notes_is_a_fixed_point():
|
||||
for raw in ("Has paid DLC", "[hover-add]", "Has paid DLC; EAC", "Has paid DLC 💀 Dead game"):
|
||||
once = clean_notes(raw)
|
||||
assert clean_notes(once) == once
|
||||
|
||||
|
||||
# ──────────── Records ────────────
|
||||
@@ -153,7 +172,7 @@ def _messy(appid: int) -> dict:
|
||||
"release_date": "28 апр. 2025 г.",
|
||||
"description": "an art "game"",
|
||||
"tags": ["Point & Click"],
|
||||
"notes": "Có DLC trả phí",
|
||||
"notes": "Has paid DLC; EAC",
|
||||
"last_updated": "2026-01-01T00:00:00Z",
|
||||
}
|
||||
|
||||
@@ -180,7 +199,7 @@ def test_save_main_writes_every_record_normalised_and_none_as_news(workdir):
|
||||
assert stored["release_date"] == "Apr 28, 2025"
|
||||
assert stored["description"] == 'an art "game"'
|
||||
assert stored["tags"] == ["Point & Click"]
|
||||
assert stored["notes"] == "Has paid DLC"
|
||||
assert stored["notes"] == "EAC"
|
||||
assert stored["last_updated"] == "2026-01-01T00:00:00Z"
|
||||
|
||||
|
||||
@@ -191,16 +210,40 @@ def test_saving_what_was_loaded_changes_no_byte(workdir):
|
||||
assert _shards() == before
|
||||
|
||||
|
||||
def test_an_override_in_vietnamese_does_not_flip_flop(workdir):
|
||||
def _override(appid: int, doc: dict) -> None:
|
||||
os.makedirs(os.path.join("data", "overrides"), exist_ok=True)
|
||||
with open(os.path.join("data", "overrides", f"{appid}.json"), "w", encoding="utf-8") as fh:
|
||||
json.dump({"schema": 1, "appid": str(appid), **doc}, fh)
|
||||
|
||||
|
||||
def test_an_override_with_the_phrase_does_not_flip_flop(workdir):
|
||||
# The override would put the phrase back after every normalisation, bump
|
||||
# last_updated and commit on every run, if it were not translated too.
|
||||
os.makedirs(os.path.join("data", "overrides"))
|
||||
with open(os.path.join("data", "overrides", "1.json"), "w", encoding="utf-8") as fh:
|
||||
json.dump({"schema": 1, "appid": "1", "fields": {
|
||||
"notes": {"value": "Có DLC trả phí; EAC", "was": "Not reviewed yet"}}}, fh)
|
||||
# last_updated and commit on every run, if it were not cleaned too.
|
||||
_override(1, {"fields": {"notes": {"value": "Has paid DLC; EAC", "was": "Not reviewed yet"}}})
|
||||
data_store.save_main([_messy(1)])
|
||||
before = _shards()
|
||||
[stored] = data_store.load_main()
|
||||
assert stored["notes"] == "Has paid DLC; EAC"
|
||||
assert stored["notes"] == "EAC"
|
||||
data_store.save_main(data_store.load_main())
|
||||
assert _shards() == before
|
||||
|
||||
|
||||
def test_an_override_of_only_the_phrase_keeps_the_marker(workdir):
|
||||
game = dict(_messy(1), notes="Has paid DLC 💀 Dead game (no players ≥14d)")
|
||||
_override(1, {"fields": {"notes": {"value": "Has paid DLC", "was": ""}}})
|
||||
data_store.save_main([game])
|
||||
before = _shards()
|
||||
[stored] = data_store.load_main()
|
||||
assert stored["notes"] == "Not reviewed yet 💀 Dead game (no players ≥14d)"
|
||||
data_store.save_main(data_store.load_main())
|
||||
assert _shards() == before
|
||||
|
||||
|
||||
def test_a_retired_entry_that_restores_nothing_is_not_news(workdir):
|
||||
# was == value: putting `was` back changes nothing, and used to bump
|
||||
# last_updated on every save all the same.
|
||||
game = dict(_messy(1), genre="FPS")
|
||||
_override(1, {"fields": {}, "retired": {"genre": {"value": "FPS", "was": "FPS"}}})
|
||||
data_store.save_main([game])
|
||||
[stored] = data_store.load_main()
|
||||
assert stored["last_updated"] == "2026-01-01T00:00:00Z"
|
||||
@@ -0,0 +1,21 @@
|
||||
"""
|
||||
refetch_all.py clears what Steam can give back and keeps what a person
|
||||
decided. Run from the repository root: python -m pytest scripts/tests
|
||||
"""
|
||||
import refetch_all
|
||||
|
||||
|
||||
def test_clear_fetchable_resets_a_default_to_the_skeleton_s_value():
|
||||
# "-" used to become "", a value nothing else writes, which the old VAC
|
||||
# detection then checked against only one category.
|
||||
game = {"anti_cheat": "-", "anti_cheat_note": "", "is_kernel_ac": None, "genre": "Uncategorized"}
|
||||
refetch_all.clear_fetchable(game)
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("-", "", None)
|
||||
assert game["genre"] == ""
|
||||
|
||||
|
||||
def test_clear_fetchable_keeps_a_decision():
|
||||
game = {"anti_cheat": "BattlEye", "anti_cheat_note": "BattlEye [Kernel]", "is_kernel_ac": True, "genre": "FPS"}
|
||||
refetch_all.clear_fetchable(game)
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"], game["genre"]) == \
|
||||
("BattlEye", "BattlEye [Kernel]", True, "FPS")
|
||||
@@ -0,0 +1,219 @@
|
||||
"""
|
||||
refresh_anticheat.py re-reads every game's anti-cheat disclosure from Steam
|
||||
daily and lets Steam win. These pin what makes a daily writer of manual fields
|
||||
safe: no disclosure and no answer never clear anything, an override is never
|
||||
fought, last_updated moves only on a real change, and a second run writes the
|
||||
same bytes.
|
||||
|
||||
Steam is never called: the client is a fake. Run from the repository root:
|
||||
python -m pytest scripts/tests
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
import refresh_anticheat as rac
|
||||
from core import data_store
|
||||
|
||||
STAMP = "2026-10-16T16:30:00Z"
|
||||
OLD = "2026-01-01T00:00:00Z"
|
||||
|
||||
|
||||
def _rec(appid: str, name: str, **fields) -> dict:
|
||||
base = {"link": f"https://store.steampowered.com/app/{appid}/", "name": name,
|
||||
"anti_cheat": "-", "anti_cheat_note": "", "is_kernel_ac": None,
|
||||
"type_game": "online", "notes": "Not reviewed yet", "last_updated": OLD}
|
||||
base.update(fields)
|
||||
return base
|
||||
|
||||
|
||||
def _page(name: str, kernel: bool) -> str:
|
||||
heading = "Uses Kernel Level Anti-Cheat" if kernel else "Uses Anti-Cheat Software"
|
||||
cls = "DRM_notice" if kernel else "anticheat_nonkernel_notice"
|
||||
return (f'<div class="anticheat_section {cls}" >\n\t<div>{heading}</div>\n\t'
|
||||
f'<div class="anticheat_name">{name}<span class="anticheat_uninstalls"></span></div>\n</div>')
|
||||
|
||||
|
||||
class FakeClient:
|
||||
"""GetItems' extra_details.anticheat by appid (None: no disclosure), store
|
||||
pages by appid, and the appids whose batch failed."""
|
||||
|
||||
def __init__(self, items=None, pages=None, failed=()):
|
||||
self.items, self.pages, self.failed = items or {}, pages or {}, set(failed)
|
||||
self.calls: list[tuple] = []
|
||||
|
||||
def fetch_anticheat_items(self, appids, give_up_after=10):
|
||||
self.calls.append(("items", tuple(appids)))
|
||||
out = {}
|
||||
for a in appids:
|
||||
if a not in self.failed:
|
||||
found = self.items.get(a)
|
||||
out[a] = {"id": int(a), "success": 1,
|
||||
"extra_details": {} if found is None else {"anticheat": found}}
|
||||
return out, {a for a in appids if a in self.failed}
|
||||
|
||||
def fetch_store_page_full(self, appid):
|
||||
self.calls.append(("page", appid))
|
||||
html = self.pages.get(appid)
|
||||
return ("ok", html) if html is not None else ("network_error", None)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def repo(tmp_path, monkeypatch):
|
||||
monkeypatch.chdir(tmp_path)
|
||||
os.makedirs("data")
|
||||
monkeypatch.setattr(rac, "now_iso", lambda: STAMP)
|
||||
|
||||
def run(records=None, client=None, argv=()):
|
||||
if records is not None:
|
||||
data_store.save_main(records, apply_human_overrides=False)
|
||||
monkeypatch.setattr(rac, "get_client", lambda: client)
|
||||
rac.main(list(argv))
|
||||
return {data_store.extract_appid(g["link"]): g for g in data_store.load_main()}
|
||||
return run
|
||||
|
||||
|
||||
def _bytes() -> dict:
|
||||
return {name: open(os.path.join("data", name), "rb").read()
|
||||
for name in sorted(os.listdir("data")) if name.endswith((".jsonl", ".json"))}
|
||||
|
||||
|
||||
def _override(appid: str, doc: dict) -> None:
|
||||
os.makedirs(os.path.join("data", "overrides"), exist_ok=True)
|
||||
with open(os.path.join("data", "overrides", f"{appid}.json"), "w", encoding="utf-8") as fh:
|
||||
json.dump({"schema": 1, "appid": appid, **doc}, fh)
|
||||
|
||||
|
||||
# What Steam answered on 2026-10-16.
|
||||
STEAM = FakeClient(
|
||||
items={
|
||||
"2507950": {"kernel_mode": True, "uninstall_completely": True, "name": "Anti-Cheat Expert (ACE)"},
|
||||
"1203220": None, # NARAKA: declares none
|
||||
"578080": {"kernel_mode": True, "uninstall_completely": True},
|
||||
"2767030": {},
|
||||
"3513350": {"kernel_mode": True, "name": "Anti-Cheat Expert (ACE)"},
|
||||
"4224240": {"boot_protection": True, "boot_protection_name": "secureboot_tpm2"},
|
||||
"1252250": {},
|
||||
},
|
||||
pages={
|
||||
"578080": _page("KSS (Krafton Security Services) ", True),
|
||||
"2767030": _page("NetEase Game Security", False),
|
||||
"4224240": _page("Fairguard", False).replace(
|
||||
"</div>\n</div>", '</div>\n\t<div class="anticheat_name">Boot Protection'
|
||||
'<span class="anticheat_uninstalls"> - Requires both Secure Boot & TPM 2.0'
|
||||
'</span></div>\n</div>'),
|
||||
"1252250": _page("", False),
|
||||
},
|
||||
)
|
||||
|
||||
CATALOGUE = [
|
||||
_rec("2507950", "Delta Force"),
|
||||
_rec("1203220", "NARAKA: BLADEPOINT", anti_cheat="EAC", anti_cheat_note="EAC"),
|
||||
_rec("578080", "PUBG: BATTLEGROUNDS", anti_cheat="Krafton Security Services",
|
||||
anti_cheat_note="Krafton Security Services [Kernel]", is_kernel_ac=True),
|
||||
_rec("2767030", "Marvel Rivals", anti_cheat="NetEase Game Security"),
|
||||
_rec("3513350", "Wuthering Waves", anti_cheat="EAC", anti_cheat_note="Easy Anti-Cheat [Kernel]", is_kernel_ac=True),
|
||||
_rec("4224240", "Jujutsu Kaisen Phantom Parade", anti_cheat="Fairguard",
|
||||
anti_cheat_note="Fairguard [Kernel]", is_kernel_ac=True),
|
||||
_rec("1252250", "Rebornia"),
|
||||
_rec("230410", "Warframe", anti_cheat=""),
|
||||
]
|
||||
|
||||
|
||||
def test_conflicts_resolve_steam_first(repo, capsys):
|
||||
games = repo([dict(g) for g in CATALOGUE], STEAM)
|
||||
trio = {a: (g["anti_cheat"], g["anti_cheat_note"], g["is_kernel_ac"]) for a, g in games.items()}
|
||||
assert trio["2507950"] == ("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE) [Kernel]", True)
|
||||
assert trio["3513350"] == ("Anti-Cheat Expert (ACE)", "Anti-Cheat Expert (ACE) [Kernel]", True)
|
||||
assert trio["4224240"] == ("Fairguard", "Fairguard [Non-Kernel]", False)
|
||||
assert trio["2767030"] == ("NetEase Game Security", "NetEase Game Security [Non-Kernel]", False)
|
||||
# The same product under the name the extension wrote: nothing to do.
|
||||
assert trio["578080"] == ("Krafton Security Services", "Krafton Security Services [Kernel]", True)
|
||||
out = capsys.readouterr().out
|
||||
assert "4 games changed (filled 1, replaced 1, updated 2)" in out
|
||||
assert "1252250 Rebornia" in out
|
||||
|
||||
|
||||
def test_no_disclosure_and_no_name_never_clear_anything(repo):
|
||||
games = repo([dict(g) for g in CATALOGUE], STEAM)
|
||||
assert games["1203220"]["anti_cheat"] == "EAC" # NARAKA declares none
|
||||
assert games["230410"]["anti_cheat"] == "" # Warframe: not even asked about by name
|
||||
assert games["1252250"]["anti_cheat"] == "-" # disclosed, named nowhere
|
||||
|
||||
|
||||
def test_last_updated_moves_only_on_change(repo):
|
||||
games = repo([dict(g) for g in CATALOGUE], STEAM)
|
||||
moved = sorted(a for a, g in games.items() if g["last_updated"] == STAMP)
|
||||
assert moved == ["2507950", "2767030", "3513350", "4224240"]
|
||||
|
||||
|
||||
def test_a_second_run_changes_no_byte(repo):
|
||||
repo([dict(g) for g in CATALOGUE], STEAM)
|
||||
before = _bytes()
|
||||
repo(client=STEAM)
|
||||
assert _bytes() == before
|
||||
|
||||
|
||||
def test_the_page_is_read_only_for_unnamed_disclosures(repo):
|
||||
client = FakeClient(STEAM.items, STEAM.pages)
|
||||
repo([dict(g) for g in CATALOGUE], client)
|
||||
pages = sorted(a for kind, a in client.calls if kind == "page")
|
||||
assert pages == ["1252250", "2767030", "4224240", "578080"]
|
||||
assert [kind for kind, _ in client.calls].count("items") == 1
|
||||
|
||||
|
||||
def test_a_failed_batch_is_unknown_and_reported(repo, capsys):
|
||||
client = FakeClient(STEAM.items, STEAM.pages, failed={"2507950", "3513350"})
|
||||
games = repo([dict(g) for g in CATALOGUE], client)
|
||||
assert games["2507950"]["anti_cheat"] == "-"
|
||||
assert games["3513350"]["anti_cheat"] == "EAC"
|
||||
assert games["2767030"]["is_kernel_ac"] is False # the rest still applies
|
||||
assert "::warning::GetItems did not answer for 2 games" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_steam_down_saves_nothing_and_exits_zero(repo, capsys):
|
||||
# Notes a save would clean, so a save of any kind would show in the bytes.
|
||||
records = [dict(g, notes="Has paid DLC") for g in CATALOGUE]
|
||||
data_store.save_main(records, apply_human_overrides=False)
|
||||
with open(os.path.join("data", "data_001.jsonl"), "w", encoding="utf-8", newline="\n") as fh:
|
||||
for r in records:
|
||||
fh.write(json.dumps(r, ensure_ascii=False) + "\n")
|
||||
before = _bytes()
|
||||
repo(client=FakeClient(failed={data_store.extract_appid(g["link"]) for g in CATALOGUE}))
|
||||
assert _bytes() == before
|
||||
assert "Steam answered for no game: nothing written." in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_an_active_override_is_not_fought(repo):
|
||||
# A human said Wuthering Waves uses EAC; Steam says ACE every day.
|
||||
_override("3513350", {"fields": {"anti_cheat": {"value": "EAC"}}})
|
||||
games = repo([dict(g) for g in CATALOGUE], STEAM)
|
||||
assert games["3513350"]["anti_cheat"] == "EAC"
|
||||
assert games["3513350"]["last_updated"] == OLD
|
||||
before = _bytes()
|
||||
repo(client=STEAM)
|
||||
assert _bytes() == before
|
||||
|
||||
|
||||
def test_a_retired_value_is_not_written_back(repo):
|
||||
# Retired: the human's ACE gave way to the old "-". Writing ACE again would
|
||||
# make apply_overrides() restore "-" on every save - a commit a day.
|
||||
_override("2507950", {"fields": {}, "retired": {"anti_cheat": {"value": "Anti-Cheat Expert (ACE)", "was": "-"}}})
|
||||
games = repo([dict(g) for g in CATALOGUE], STEAM)
|
||||
assert games["2507950"]["anti_cheat"] == "-"
|
||||
before = _bytes()
|
||||
repo(client=STEAM)
|
||||
assert _bytes() == before
|
||||
|
||||
|
||||
def test_dry_run_writes_nothing_and_appid_narrows(repo, capsys):
|
||||
repo([dict(g) for g in CATALOGUE], FakeClient())
|
||||
before = _bytes()
|
||||
client = FakeClient(STEAM.items, STEAM.pages)
|
||||
repo(client=client, argv=["--dry-run", "--appid", "2507950", "3513350"])
|
||||
assert _bytes() == before
|
||||
[(_, asked)] = [c for c in client.calls if c[0] == "items"]
|
||||
assert sorted(asked) == ["2507950", "3513350"]
|
||||
out = capsys.readouterr().out
|
||||
assert "2 games changed" in out and "Dry run: nothing written." in out
|
||||
@@ -176,6 +176,19 @@ def test_description_is_compared_as_stored():
|
||||
assert "description" not in rsd.apply_app_details(game, {"short_description": blurb})
|
||||
|
||||
|
||||
def test_a_disclosed_anti_cheat_changes_no_manual_field():
|
||||
# The page now carries the anti-cheat disclosure too; refresh_anticheat.py
|
||||
# applies it, never this job.
|
||||
page = _page() + ('<div class="anticheat_section DRM_notice" ><div>Uses Kernel Level Anti-Cheat</div>'
|
||||
'<div class="anticheat_name">BattlEye<span class="anticheat_uninstalls"></span></div></div>')
|
||||
game = _game(is_kernel_ac=None, anti_cheat_note="")
|
||||
scraped = scrape_store_page(page)
|
||||
assert scraped["anti_cheat"].name == "BattlEye"
|
||||
changed = rsd.apply_store_page(game, scraped)
|
||||
assert not set(changed) & MANUAL_FIELDS
|
||||
assert (game["anti_cheat"], game["anti_cheat_note"], game["is_kernel_ac"]) == ("VAC", "", None)
|
||||
|
||||
|
||||
def test_the_refreshed_fields_never_include_human_judgement():
|
||||
assert not (set(rsd.PAGE_FIELDS) | set(rsd.DETAILS_FIELDS)) & MANUAL_FIELDS
|
||||
assert "notes" not in rsd.PAGE_FIELDS + rsd.DETAILS_FIELDS
|
||||
|
||||
@@ -113,6 +113,34 @@ def test_store_items_are_asked_for_in_batches_and_a_failed_batch_fails_all(monke
|
||||
assert len(session.requests) == 2
|
||||
|
||||
|
||||
def test_store_items_ask_for_the_anti_cheat_disclosure_too():
|
||||
client, session = _client(_Resp(steam_client.STORE_ITEMS_URL, body={"response": {"store_items": []}}))
|
||||
client.fetch_store_items(["2279510"])
|
||||
sent = json.loads(session.requests[0]["params"]["input_json"])
|
||||
assert sent["data_request"] == {"include_tag_count": 20, "include_supported_languages": True,
|
||||
"include_extra_details": True}
|
||||
|
||||
|
||||
def test_anticheat_items_survive_a_failed_batch(monkeypatch):
|
||||
monkeypatch.setattr(steam_client, "STORE_ITEMS_BATCH", 2)
|
||||
one = {"response": {"store_items": [{"id": 1, "success": 1, "extra_details": {"anticheat": {}}}]}}
|
||||
client, session = _client(_Resp(steam_client.STORE_ITEMS_URL, status_code=403),
|
||||
_Resp(steam_client.STORE_ITEMS_URL, body=one))
|
||||
items, failed = client.fetch_anticheat_items(["3", "4", "1"])
|
||||
assert list(items) == ["1"] and failed == {"3", "4"}
|
||||
sent = json.loads(session.requests[1]["params"]["input_json"])
|
||||
assert sent["data_request"] == {"include_extra_details": True}
|
||||
|
||||
|
||||
def test_anticheat_items_give_up_after_failures_in_a_row(monkeypatch):
|
||||
monkeypatch.setattr(steam_client, "STORE_ITEMS_BATCH", 1)
|
||||
bad = [_Resp(steam_client.STORE_ITEMS_URL, status_code=403) for _ in range(2)]
|
||||
client, session = _client(*bad)
|
||||
items, failed = client.fetch_anticheat_items(["1", "2", "3", "4"], give_up_after=2)
|
||||
assert items == {} and failed == {"1", "2", "3", "4"}
|
||||
assert len(session.requests) == 2 # the rest was not asked
|
||||
|
||||
|
||||
def test_tag_names_are_fetched_once():
|
||||
body = {"response": {"tags": [{"tagid": 12095, "name": "Sexual Content"}]}}
|
||||
client, session = _client(_Resp(steam_client.TAG_LIST_URL, body=body))
|
||||
|
||||
@@ -6,6 +6,7 @@ and that a DLC price - which arrives as a string - is read as money.
|
||||
|
||||
Run from the repository root: python -m pytest scripts/tests
|
||||
"""
|
||||
from core.anticheat import Disclosure
|
||||
from core.scraper import scrape_store_page
|
||||
from core.store_items import language_name, paid_dlc_from_items, parse_store_item
|
||||
|
||||
@@ -70,6 +71,15 @@ def test_the_tagids_list_is_read_when_weights_are_missing():
|
||||
assert parse_store_item({"tagids": [9130, "6650"]}, TAG_NAMES)["tags"] == ["Hentai", "Nudity"]
|
||||
|
||||
|
||||
def test_an_item_carries_its_anti_cheat():
|
||||
# Horizon Walker (3279780): a login-gated game whose page cannot be read.
|
||||
item = {"extra_details": {"anticheat": {"kernel_mode": True}}}
|
||||
assert parse_store_item(item, TAG_NAMES)["anti_cheat"] == Disclosure("", True, False)
|
||||
assert parse_store_item({"extra_details": {}}, TAG_NAMES)["anti_cheat"] is None
|
||||
# The page scrape has the same key, so either can stand in for the other.
|
||||
assert scrape_store_page("<html></html>")["anti_cheat"] is None
|
||||
|
||||
|
||||
def test_a_dlc_price_is_money_even_as_a_string():
|
||||
assert paid_dlc_from_items([{"best_purchase_option": {"final_price_in_cents": "999"}}]) is True
|
||||
assert paid_dlc_from_items([{"best_purchase_option": {"final_price_in_cents": 1999}}]) is True
|
||||
|
||||
@@ -66,7 +66,7 @@ export function seed(sqlite: SqliteD1, repo: FakeRepo, records: Rec[], now: Date
|
||||
r.current_players ?? "N/A",
|
||||
players(r),
|
||||
JSON.stringify({ link: r.link, name: r.name, reviews: r.reviews, current_players: r.current_players }),
|
||||
extra.source ?? (n % 5 === 0 ? "extension" : "discover"),
|
||||
extra.source ?? (n % 5 === 0 ? "backfill:2025" : "discover"),
|
||||
status,
|
||||
extra.by ?? null,
|
||||
extra.at ?? null,
|
||||
@@ -121,7 +121,7 @@ export function seed(sqlite: SqliteD1, repo: FakeRepo, records: Rec[], now: Date
|
||||
// Locked because the GAME is published: undecided rows for a game that
|
||||
// already has a committed row. Neither may be approved, rejected or deferred.
|
||||
const published = committed[0];
|
||||
row(published, "pending", { seenAgo: 2 * HOUR, source: "extension" });
|
||||
row(published, "pending", { seenAgo: 2 * HOUR, source: "backfill:2025" });
|
||||
row(published, "failed", { reason: "never appeared in data/ or removed_games.jsonl - approve again to retry", by: reviewer, at: iso(80 * HOUR) });
|
||||
|
||||
// A rejected row that cannot be reopened: another row for the game is open.
|
||||
|
||||
@@ -401,7 +401,7 @@
|
||||
<div>
|
||||
<h1 class="text-2xl font-semibold tracking-tight">Review queue</h1>
|
||||
<p class="text-sm text-muted-foreground">
|
||||
Candidates found by discovery and the browser extension. Approving asks the pipeline to publish a game.
|
||||
Candidates found by discovery. Approving asks the pipeline to publish a game.
|
||||
</p>
|
||||
</div>
|
||||
<div class="flex flex-wrap items-center gap-2">
|
||||
|
||||
@@ -235,7 +235,7 @@
|
||||
<h1 class="text-2xl font-semibold tracking-tight">Remove games</h1>
|
||||
<p class="max-w-3xl text-sm text-muted-foreground">
|
||||
A removed game leaves data/ on the next pipeline run, is listed on the site's removed-games page, and stays out:
|
||||
discovery, the browser extension and every later run skip it until it is restored here.
|
||||
discovery, a queued link and every later run skip it until it is restored here.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { canonicalAc } from "./anti-cheat";
|
||||
import { ANTI_CHEAT_ENUM } from "./enums";
|
||||
|
||||
/**
|
||||
* The anti-cheat patterns live twice: ANTI_CHEAT_PATTERNS in
|
||||
* scripts/core/constants.py labels what the pipeline writes, and anti-cheat.ts
|
||||
* buckets it for the charts and filters. A pattern on one side only is a
|
||||
* product the pipeline labels and the site files under "Other" (Steam's
|
||||
* disclosure calls Denuvo just "Denuvo"). Nothing else held the two equal.
|
||||
*/
|
||||
|
||||
function pythonPatterns(): [string, string[]][] {
|
||||
const src = readFileSync(join(__dirname, "..", "..", "..", "scripts", "core", "constants.py"), "utf-8");
|
||||
const body = src.split("ANTI_CHEAT_PATTERNS")[1].split("\n}")[0];
|
||||
const out: [string, string[]][] = [];
|
||||
for (const line of body.split("\n")) {
|
||||
const m = /^\s*"([^"]+)":\s*\[([^\]]*)\]/.exec(line);
|
||||
if (m) out.push([m[1], [...m[2].matchAll(/"([^"]*)"/g)].map((p) => p[1])]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
describe("anti-cheat patterns", () => {
|
||||
const python = pythonPatterns();
|
||||
|
||||
it("reads the Python table", () => {
|
||||
expect(python.length).toBeGreaterThanOrEqual(14);
|
||||
});
|
||||
|
||||
it("knows every product the pipeline labels, in the same order", () => {
|
||||
expect(ANTI_CHEAT_ENUM).toEqual(["-", ...python.map(([key]) => key)]);
|
||||
});
|
||||
|
||||
it("files every Python pattern, and every label, under its own product", () => {
|
||||
for (const [key, patterns] of python) {
|
||||
expect(canonicalAc(key), key).toBe(key);
|
||||
for (const p of patterns) expect(canonicalAc(p), `${key}: ${p}`).toBe(key);
|
||||
}
|
||||
});
|
||||
|
||||
it("files the names Steam's disclosure uses", () => {
|
||||
expect(canonicalAc("Denuvo")).toBe("Denuvo AC");
|
||||
expect(canonicalAc("VAC (Valve Anti-Cheat)")).toBe("VAC");
|
||||
expect(canonicalAc("KSS (Krafton Security Services)")).toBe("KSS");
|
||||
expect(canonicalAc("-")).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -25,7 +25,7 @@ const AC_PATTERNS: Record<string, string[]> = {
|
||||
Ricochet: ["ricochet anti-cheat", "ricochet"],
|
||||
mHyprot: ["mhyprot", "mhyprot2"],
|
||||
"FACEIT AC": ["faceit anti-cheat", "faceit ac"],
|
||||
"Denuvo AC": ["denuvo anti-cheat"],
|
||||
"Denuvo AC": ["denuvo anti-cheat", "denuvo"],
|
||||
KSS: ["kss", "krafton security"],
|
||||
"NetEase GS": ["netease game security"],
|
||||
Hyperion: ["hyperion", "byfron"],
|
||||
|
||||
@@ -79,18 +79,6 @@ export function migrateRecord(game: Record<string, unknown>): GameRecord {
|
||||
delete out.desc;
|
||||
}
|
||||
|
||||
for (const k of [
|
||||
"free_type",
|
||||
"is_free",
|
||||
"is_dlc",
|
||||
"is_demo",
|
||||
"is_playtest",
|
||||
"price",
|
||||
"auto_notes",
|
||||
]) {
|
||||
delete out[k];
|
||||
}
|
||||
|
||||
return out as unknown as GameRecord;
|
||||
}
|
||||
|
||||
|
||||
@@ -3,8 +3,9 @@
|
||||
* pipeline so manual edits stay compatible with what the daily refetch
|
||||
* produces.
|
||||
*
|
||||
* ANTI_CHEAT_ENUM — keys of ANTI_CHEAT_PATTERNS in scripts/core/constants.py:45
|
||||
* (extend both sides if you add a new AC).
|
||||
* ANTI_CHEAT_ENUM — keys of ANTI_CHEAT_PATTERNS in scripts/core/constants.py
|
||||
* (extend both sides if you add a new AC; anti-cheat.test.ts
|
||||
* holds them equal).
|
||||
* GENRE_ENUM — curated short list of common Steam genre buckets.
|
||||
* Combobox-style: free-text new entries are allowed.
|
||||
* TYPE_GAME_ENUM — fixed.
|
||||
|
||||
+1
-35
@@ -1,6 +1,6 @@
|
||||
/**
|
||||
* Schema constants — port of scripts/core/constants.py.
|
||||
* Keep MANUAL_FIELDS, ARRAY_FIELDS, EXTENSION_FIELDS in sync with the Python source.
|
||||
* Keep MANUAL_FIELDS and the record shape in sync with the Python source.
|
||||
*/
|
||||
|
||||
export const REPO_OWNER = "poli0981";
|
||||
@@ -31,36 +31,6 @@ export const MANUAL_FIELDS = [
|
||||
"genre",
|
||||
] as const;
|
||||
|
||||
export const ARRAY_FIELDS = [
|
||||
"platforms",
|
||||
"languages",
|
||||
"language_details",
|
||||
"tags",
|
||||
"developer",
|
||||
"publisher",
|
||||
] as const;
|
||||
|
||||
export const EXTENSION_FIELDS = [
|
||||
"name",
|
||||
"genre",
|
||||
"type_game",
|
||||
"has_paid_dlc",
|
||||
"developer",
|
||||
"publisher",
|
||||
"release_date",
|
||||
"description",
|
||||
"header_image",
|
||||
"anti_cheat",
|
||||
"anti_cheat_note",
|
||||
"is_kernel_ac",
|
||||
"platforms",
|
||||
"languages",
|
||||
"language_details",
|
||||
"tags",
|
||||
"notes",
|
||||
"safe",
|
||||
] as const;
|
||||
|
||||
export const SKIP_GENRE_TAGS = new Set([
|
||||
"Free to Play",
|
||||
"Indie",
|
||||
@@ -179,7 +149,3 @@ export const SKELETON_TEMPLATE: GameRecord = {
|
||||
export function isManualField(key: string): key is ManualField {
|
||||
return (MANUAL_FIELDS as readonly string[]).includes(key);
|
||||
}
|
||||
|
||||
export function isArrayField(key: string): boolean {
|
||||
return (ARRAY_FIELDS as readonly string[]).includes(key);
|
||||
}
|
||||
@@ -164,8 +164,8 @@ export interface AppendRequest {
|
||||
* Commit only if the queue file is EMPTY at the very head the commit is made
|
||||
* on. Auto-approve uses it to stay out of the way of an ingest run that has
|
||||
* not finished with the previous batch - checked here, against the same
|
||||
* expectedHeadOid, so an extension push between its own check and this
|
||||
* commit cannot slip past.
|
||||
* expectedHeadOid, so a commit landing between its own check and this one
|
||||
* cannot slip past.
|
||||
*/
|
||||
requireEmptyQueue?: boolean;
|
||||
}
|
||||
@@ -173,10 +173,10 @@ export interface AppendRequest {
|
||||
/**
|
||||
* Append one `{"link": ...}` line per appid to the queue file.
|
||||
*
|
||||
* APPEND, never replace. This file has two producers — the browser extension
|
||||
* (poli0981/steam-f2p-extension pushes to it directly) and this Worker — and
|
||||
* overwriting it silently discards whatever the other queued. That exact bug
|
||||
* was found and fixed once already, in the since-removed issue-ingest
|
||||
* APPEND, never replace. This Worker is not the file's only writer - Restore
|
||||
* appends here too, a pull request may add a link, and the ingest clears it -
|
||||
* and overwriting it silently discards whatever another one queued. That exact
|
||||
* bug was found and fixed once already, in the since-removed issue-ingest
|
||||
* workflow; do not reintroduce it here.
|
||||
*
|
||||
* It takes no actor. The commit body says ADMIN_ATTRIBUTION, and the parameter
|
||||
|
||||
@@ -13,10 +13,11 @@
|
||||
* and it writes one only after observing the appid in the published dataset.
|
||||
*
|
||||
* It also SWEEPS: a pending, deferred or failed row whose game is already in
|
||||
* data/ is marked committed. That happens when the browser extension queues a
|
||||
* game discovery also proposed, and when an approval aged out to 'failed' (see
|
||||
* STALE_AFTER_MS) and was published afterwards anyway - before the sweep such a
|
||||
* row stayed 'failed' forever for a game that was live.
|
||||
* data/ is marked committed. That happens when a game reaches the catalogue by
|
||||
* another route (a pull request's link, a manual edit) while discovery also
|
||||
* proposed it, and when an approval aged out to 'failed' (see STALE_AFTER_MS)
|
||||
* and was published afterwards anyway - before the sweep such a row stayed
|
||||
* 'failed' forever for a game that was live.
|
||||
*/
|
||||
import type { AdminDeps } from "./deps";
|
||||
import { acquireLease, releaseLease, readState, writeState } from "./locks";
|
||||
|
||||
@@ -347,7 +347,7 @@ export async function handleRemoveApi(
|
||||
},
|
||||
{
|
||||
path: TEMP_INFO_PATH,
|
||||
// APPEND, never replace: the extension writes this file too.
|
||||
// APPEND, never replace: approvals and pull requests write this file too.
|
||||
build: (current) => {
|
||||
for (const line of current.split("\n")) {
|
||||
try {
|
||||
|
||||
Reference in new issue
Block a user