* feat(pipeline): rotating refresh + patch/apply pipeline, drop lists/
Replace check_paid / check_alive / update_reviews / update_status: each
crawled all 2,600+ pages sequentially, overran its job timeout, saved
only at the end and has lost every run since late June (unreported:
timeouts end as "cancelled").
- scripts/refresh.py: daily slice of ~1/7 of the catalog (pending
strikes first, then least recently checked), one GET per game for
alive / paid / rating / status (+ release_date, thumbnail, updated_at
backfill). Removal needs the same 404/410 or paid strike on two
different UTC days. Paced requests, capped Retry-After back-off, stop
after 3x 429, checkpoints and an in-script deadline. --full replaces
force_update.py (rotating batches, editable fields preserved).
- Scanners emit a URL-keyed patch (patch.py); apply_patch.py applies it
to the latest main and runs validate.py; bash/commit_push.sh fetches,
re-applies and retries pushes, so concurrent writers never clobber
each other.
- update_info.py: canonical URLs (canonical.py), skip games in the
deleted log, keep transient failures queued (max 3 runs, tracked in
scripts/state/), stamp added_at, never drop links queued meanwhile.
- data_store.py: games stay in their chunk, re-slice only past
thresholds, write only changed bytes; index/count_history change only
with data (no more no-op commits).
- validate.py (schema, enums, canonical/unique URLs, index consistency)
+ normalize one invalid safe_virus ("y" -> "Yes").
- tests/: 85 network-free pytest cases, HTML fixtures (from sample/),
shared URL vectors and golden chunk/serialization fixtures for the
future Worker port.
- Workflows: refresh.yml (scan -> apply -> notify), update.yml (runs on
queue change), force_update.yml (refresh.yml full mode),
notify-failure.yml (Discord on failure/cancel/rate limit),
python-ci.yml, webapp-ci.yml, deploy-status.yml; draft-first release
with one create-release job and tauri-action v1, no caches in release
builds; SHA-pinned actions, permissions: {} defaults, reusable
workflows pinned with explicit secrets, injection fixes; dependabot.yml.
- Remove lists/, generate_md, generate_table, log_deleted,
deleted_games.txt, the Telegram bot-ingest path and old bash wrappers;
update README (EN/VI), CONTRIBUTING, templates, CLAUDE.md, CHANGELOG
and mark the retired Telegram path in the policies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(pipeline): review findings on the refresh / ingest rewrite
Removals
- Mass-change guard: a run whose removals or new strikes exceed
max(10, 10%) of the games it checked withholds all its removals and
alerts (mass_change). Withheld games are marked withheld_at and stay
withheld in later (even single-URL or rate-limited) runs until a healthy
check clears them or a run is dispatched with allow_mass_removal.
- Checkpoints never carry removals, so a scan that dies before its guard
cannot delete games; apply_patch re-checks the per-run limit and caps
any patch at max(25, 5%) of the catalog (ALLOW_MASS_REMOVAL=1 overrides).
- Strikes mature by time, not calendar day: a second sighting must come
>= 20 h after the first (strike_at), so two checks around midnight UTC no
longer remove a game.
- Overlapping scans can't undo each other: refresh_state merges keep the
newest check and the newest full stamp in either apply order, and a
removal is skipped when a newer check has landed since.
- A corrupt count_history.json fails the apply instead of being replaced
by a one-row history.
Ingest
- A second spelling of a queued game no longer drops the first copy
before it reaches a final outcome.
- A dispatched url is processed first and kept in temp_link.json
(queue_add) until it finishes, so a retry, deadline or rate-limit stop
keeps it.
- Failures count as an attempt only >= 6 h apart and a 429 counts as none,
so a burst of queued runs can't burn a link's three attempts.
- Non-string queue entries are cleaned on every apply instead of crashing
it; {"url": ...} objects are unwrapped.
CI
- queue: max on the ingest, refresh-apply and force-update concurrency
groups: the default queue replaced (and silently dropped) pending runs.
- Seed scripts/state/*.json and mkdir -p before git add (the pathspec
failed on an empty patch until the directory existed on main).
- Cancelling a refresh / force-update / desktop release really stops it
(!cancelled() instead of always()).
- deploy-status.yml becomes an hourly check of main's latest finished
"Workers Builds: free-games-itchio-list" run (bash/cf_build_status.sh):
GitHub never starts check_run workflows for GITHUB_TOKEN data commits.
- The reusable release notifiers still take their script from
poli0981/.github@main; the caller comments now say so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(pipeline): close the remaining mass-removal and monitor gaps
From the second review round of the pipeline fixes:
- A strike spike now marks every strike of that run withheld_at (the
checkpoint of a run that dies mid-spike too), so a later short,
rate-limited or single-URL run can't remove them without an
allow_mass_removal dispatch.
- Withheld games take at most half of a run's budget (all of it for the
allow_mass_removal run that confirms them), and a withheld full-mode
game moves its `full` stamp, so neither rotation stalls on them.
- refresh.yml runs one scan -> apply pipeline at a time (workflow-level
concurrency refresh-pipeline, queue: max), so an older removal can't be
applied after a newer healthy check of the same game.
- deploy-status.yml alerts once per failed Cloudflare build, keyed on its
check-run id and marked only after the Discord post, instead of by age
(which could drop an alert after a missed run, or repeat it); the
monitor walks main's first-parent history so preview builds of merged
branch commits are never mistaken for the site's build.
- CLAUDE.md: merge PRs with squash or a merge commit, never by fast-forward.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
179 lines
6.0 KiB
Python
179 lines
6.0 KiB
Python
"""
|
|
data_store.py — Multi-file game data storage.
|
|
|
|
Splits game entries across numbered JSON files in data_game/:
|
|
data_game/game_info_001.json (up to 500 entries)
|
|
data_game/game_info_002.json (next 500)
|
|
...
|
|
|
|
All other scripts import load_all_games / save_all_games
|
|
instead of reading the chunk files directly.
|
|
|
|
Write policy (keeps diffs small and avoids no-op commits / rebuilds):
|
|
- A game stays in the chunk it already lives in. Removals are in place,
|
|
updates are in place, new games are appended to the last chunk.
|
|
- Chunks are re-sliced only when one overflows CHUNK_SIZE or a chunk that
|
|
is not the last one drops below MIN_FILL. Chunk names stay contiguous.
|
|
- Files are written only when their bytes change (json_io.save_json).
|
|
- index.json `last_updated` changes only when some chunk changed, and
|
|
count_history.json gets a row only when the total changes.
|
|
|
|
The Worker (webapp/worker/catalog.ts) ports this exact policy; both are
|
|
checked against tests/fixtures/golden/.
|
|
"""
|
|
|
|
import glob
|
|
import os
|
|
from datetime import UTC, datetime
|
|
|
|
from json_io import load_json, save_json
|
|
|
|
DATA_DIR = "data_game"
|
|
FILE_PREFIX = "game_info_"
|
|
FILE_SUFFIX = ".json"
|
|
CHUNK_SIZE = 500
|
|
MIN_FILL = 250
|
|
|
|
|
|
def _index_file() -> str:
|
|
return os.path.join(DATA_DIR, "index.json")
|
|
|
|
|
|
def _count_history_file() -> str:
|
|
return os.path.join(DATA_DIR, "count_history.json")
|
|
|
|
|
|
def _chunk_path(index: int) -> str:
|
|
"""Return the file path for a 1-based chunk index."""
|
|
return os.path.join(DATA_DIR, f"{FILE_PREFIX}{index:03d}{FILE_SUFFIX}")
|
|
|
|
|
|
def _list_chunk_files() -> list[str]:
|
|
"""Return sorted list of existing chunk file paths."""
|
|
pattern = os.path.join(DATA_DIR, f"{FILE_PREFIX}*{FILE_SUFFIX}")
|
|
return sorted(glob.glob(pattern))
|
|
|
|
|
|
def _utc_now() -> datetime:
|
|
return datetime.now(UTC)
|
|
|
|
|
|
def load_chunks() -> list[list[dict]]:
|
|
"""Load every chunk file, preserving chunk boundaries."""
|
|
return [load_json(path) for path in _list_chunk_files()]
|
|
|
|
|
|
def load_all_games() -> list[dict]:
|
|
"""Load every chunk file and return a single flat list."""
|
|
return [game for chunk in load_chunks() for game in chunk]
|
|
|
|
|
|
def plan_chunks(old_chunks: list[list[dict]], games: list[dict]) -> list[list[dict]]:
|
|
"""Place `games` into chunks, keeping each game in its current chunk.
|
|
|
|
Pure function (no I/O) so the placement policy can be unit-tested and
|
|
mirrored exactly by the Worker.
|
|
"""
|
|
by_url: dict[str, dict] = {}
|
|
for game in games:
|
|
by_url.setdefault(game["url"], game)
|
|
|
|
placed: set[str] = set()
|
|
chunks: list[list[dict]] = []
|
|
for old in old_chunks:
|
|
chunk = []
|
|
for game in old:
|
|
url = game.get("url")
|
|
if url in by_url and url not in placed:
|
|
chunk.append(by_url[url])
|
|
placed.add(url)
|
|
chunks.append(chunk)
|
|
|
|
new_games = [g for url, g in by_url.items() if url not in placed]
|
|
if new_games:
|
|
if not chunks:
|
|
chunks.append([])
|
|
chunks[-1].extend(new_games)
|
|
|
|
chunks = [c for c in chunks if c]
|
|
needs_reslice = any(len(c) > CHUNK_SIZE for c in chunks) or any(
|
|
len(c) < MIN_FILL for c in chunks[:-1]
|
|
)
|
|
if needs_reslice:
|
|
flat = [g for c in chunks for g in c]
|
|
chunks = [flat[i : i + CHUNK_SIZE] for i in range(0, len(flat), CHUNK_SIZE)]
|
|
return chunks
|
|
|
|
|
|
def save_all_games(games: list[dict]) -> bool:
|
|
"""Persist `games` to data_game/ with minimal file churn.
|
|
|
|
Returns True when any data file changed.
|
|
"""
|
|
os.makedirs(DATA_DIR, exist_ok=True)
|
|
old_files = _list_chunk_files()
|
|
chunks = plan_chunks([load_json(p) for p in old_files], games)
|
|
|
|
changed = False
|
|
new_files: list[str] = []
|
|
for i, chunk in enumerate(chunks, start=1):
|
|
path = _chunk_path(i)
|
|
new_files.append(path)
|
|
changed |= save_json(path, chunk)
|
|
for stale in set(old_files) - set(new_files):
|
|
os.remove(stale)
|
|
changed = True
|
|
|
|
changed |= _write_index([(p, len(c)) for p, c in zip(new_files, chunks, strict=True)], changed)
|
|
return changed
|
|
|
|
|
|
def _write_index(chunks: list[tuple[str, int]], data_changed: bool) -> bool:
|
|
"""Write data_game/index.json; `last_updated` moves only when data changed."""
|
|
total = sum(count for _, count in chunks)
|
|
previous = load_json(_index_file(), default={})
|
|
files = [{"name": os.path.basename(path), "count": count} for path, count in chunks]
|
|
layout_changed = previous.get("files") != files or previous.get("total_games") != total
|
|
last_updated = previous.get("last_updated")
|
|
if data_changed or layout_changed or not last_updated:
|
|
last_updated = _utc_now().strftime("%Y-%m-%dT%H:%M:%SZ")
|
|
index = {
|
|
"total_games": total,
|
|
"max_per_file": CHUNK_SIZE,
|
|
"last_updated": last_updated,
|
|
"files": files,
|
|
}
|
|
changed = save_json(_index_file(), index)
|
|
changed |= _append_count_history(total)
|
|
return changed
|
|
|
|
|
|
def _append_count_history(total: int) -> bool:
|
|
"""Record today's total in data_game/count_history.json when it changed.
|
|
|
|
Date-keyed upsert: each UTC day holds one row, last write wins. A day whose
|
|
total equals the latest recorded total adds no row (keeps the series and
|
|
the git history free of no-op points). A corrupt file raises (like a
|
|
corrupt index or chunk) instead of being replaced by a one-row history;
|
|
restore it from git history.
|
|
"""
|
|
today = _utc_now().strftime("%Y-%m-%d")
|
|
history: list[dict] = load_json(_count_history_file())
|
|
if not isinstance(history, list):
|
|
raise ValueError(f"{_count_history_file()} must be a JSON array")
|
|
history.sort(key=lambda r: r.get("date", ""))
|
|
for row in history:
|
|
if row.get("date") == today:
|
|
row["total"] = total
|
|
break
|
|
else:
|
|
if history and history[-1].get("total") == total:
|
|
return False
|
|
history.append({"date": today, "total": total})
|
|
return save_json(_count_history_file(), history)
|
|
|
|
|
|
def get_all_urls(games: list[dict]) -> set[str]:
|
|
"""Return a set of all game URLs (for duplicate checking)."""
|
|
return {g["url"] for g in games}
|