Files
poli0981andClaude Opus 5 e020be12fa feat(pipeline): rotating refresh + patch/apply pipeline, drop lists/ (v4 Phase 1) (#82)
* feat(pipeline): rotating refresh + patch/apply pipeline, drop lists/

Replace check_paid / check_alive / update_reviews / update_status: each
crawled all 2,600+ pages sequentially, overran its job timeout, saved
only at the end and has lost every run since late June (unreported:
timeouts end as "cancelled").

- scripts/refresh.py: daily slice of ~1/7 of the catalog (pending
  strikes first, then least recently checked), one GET per game for
  alive / paid / rating / status (+ release_date, thumbnail, updated_at
  backfill). Removal needs the same 404/410 or paid strike on two
  different UTC days. Paced requests, capped Retry-After back-off, stop
  after 3x 429, checkpoints and an in-script deadline. --full replaces
  force_update.py (rotating batches, editable fields preserved).
- Scanners emit a URL-keyed patch (patch.py); apply_patch.py applies it
  to the latest main and runs validate.py; bash/commit_push.sh fetches,
  re-applies and retries pushes, so concurrent writers never clobber
  each other.
- update_info.py: canonical URLs (canonical.py), skip games in the
  deleted log, keep transient failures queued (max 3 runs, tracked in
  scripts/state/), stamp added_at, never drop links queued meanwhile.
- data_store.py: games stay in their chunk, re-slice only past
  thresholds, write only changed bytes; index/count_history change only
  with data (no more no-op commits).
- validate.py (schema, enums, canonical/unique URLs, index consistency)
  + normalize one invalid safe_virus ("y" -> "Yes").
- tests/: 85 network-free pytest cases, HTML fixtures (from sample/),
  shared URL vectors and golden chunk/serialization fixtures for the
  future Worker port.
- Workflows: refresh.yml (scan -> apply -> notify), update.yml (runs on
  queue change), force_update.yml (refresh.yml full mode),
  notify-failure.yml (Discord on failure/cancel/rate limit),
  python-ci.yml, webapp-ci.yml, deploy-status.yml; draft-first release
  with one create-release job and tauri-action v1, no caches in release
  builds; SHA-pinned actions, permissions: {} defaults, reusable
  workflows pinned with explicit secrets, injection fixes; dependabot.yml.
- Remove lists/, generate_md, generate_table, log_deleted,
  deleted_games.txt, the Telegram bot-ingest path and old bash wrappers;
  update README (EN/VI), CONTRIBUTING, templates, CLAUDE.md, CHANGELOG
  and mark the retired Telegram path in the policies.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(pipeline): review findings on the refresh / ingest rewrite

Removals
- Mass-change guard: a run whose removals or new strikes exceed
  max(10, 10%) of the games it checked withholds all its removals and
  alerts (mass_change). Withheld games are marked withheld_at and stay
  withheld in later (even single-URL or rate-limited) runs until a healthy
  check clears them or a run is dispatched with allow_mass_removal.
- Checkpoints never carry removals, so a scan that dies before its guard
  cannot delete games; apply_patch re-checks the per-run limit and caps
  any patch at max(25, 5%) of the catalog (ALLOW_MASS_REMOVAL=1 overrides).
- Strikes mature by time, not calendar day: a second sighting must come
  >= 20 h after the first (strike_at), so two checks around midnight UTC no
  longer remove a game.
- Overlapping scans can't undo each other: refresh_state merges keep the
  newest check and the newest full stamp in either apply order, and a
  removal is skipped when a newer check has landed since.
- A corrupt count_history.json fails the apply instead of being replaced
  by a one-row history.

Ingest
- A second spelling of a queued game no longer drops the first copy
  before it reaches a final outcome.
- A dispatched url is processed first and kept in temp_link.json
  (queue_add) until it finishes, so a retry, deadline or rate-limit stop
  keeps it.
- Failures count as an attempt only >= 6 h apart and a 429 counts as none,
  so a burst of queued runs can't burn a link's three attempts.
- Non-string queue entries are cleaned on every apply instead of crashing
  it; {"url": ...} objects are unwrapped.

CI
- queue: max on the ingest, refresh-apply and force-update concurrency
  groups: the default queue replaced (and silently dropped) pending runs.
- Seed scripts/state/*.json and mkdir -p before git add (the pathspec
  failed on an empty patch until the directory existed on main).
- Cancelling a refresh / force-update / desktop release really stops it
  (!cancelled() instead of always()).
- deploy-status.yml becomes an hourly check of main's latest finished
  "Workers Builds: free-games-itchio-list" run (bash/cf_build_status.sh):
  GitHub never starts check_run workflows for GITHUB_TOKEN data commits.
- The reusable release notifiers still take their script from
  poli0981/.github@main; the caller comments now say so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(pipeline): close the remaining mass-removal and monitor gaps

From the second review round of the pipeline fixes:
- A strike spike now marks every strike of that run withheld_at (the
  checkpoint of a run that dies mid-spike too), so a later short,
  rate-limited or single-URL run can't remove them without an
  allow_mass_removal dispatch.
- Withheld games take at most half of a run's budget (all of it for the
  allow_mass_removal run that confirms them), and a withheld full-mode
  game moves its `full` stamp, so neither rotation stalls on them.
- refresh.yml runs one scan -> apply pipeline at a time (workflow-level
  concurrency refresh-pipeline, queue: max), so an older removal can't be
  applied after a newer healthy check of the same game.
- deploy-status.yml alerts once per failed Cloudflare build, keyed on its
  check-run id and marked only after the Discord post, instead of by age
  (which could drop an alert after a missed run, or repeat it); the
  monitor walks main's first-parent history so preview builds of merged
  branch commits are never mistaken for the site's build.
- CLAUDE.md: merge PRs with squash or a merge commit, never by fast-forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 17:52:57 +07:00

89 lines
3.6 KiB
Python

import json
import os
from datetime import UTC
import data_store
import pytest
from conftest import FIXTURES, make_game, read_json, seed_catalog
from json_io import dumps
CASES = json.loads((FIXTURES / "golden" / "plan_chunks.json").read_text(encoding="utf-8"))
@pytest.mark.parametrize("case", CASES, ids=[c["name"] for c in CASES])
def test_plan_chunks_golden(case, monkeypatch):
monkeypatch.setattr(data_store, "CHUNK_SIZE", case["chunk_size"])
monkeypatch.setattr(data_store, "MIN_FILL", case["min_fill"])
old = [[{"url": u} for u in chunk] for chunk in case["old"]]
games = [{"url": u} for u in case["games"]]
got = data_store.plan_chunks(old, games)
assert [[g["url"] for g in chunk] for chunk in got] == case["expected"]
def test_serialization_matches_golden():
data = read_json(FIXTURES / "golden" / "serialize_input.json")
expected = (FIXTURES / "golden" / "serialize_expected.txt").read_bytes().decode("utf-8")
assert dumps(data) == expected
def test_save_writes_chunks_index_and_history(repo, monkeypatch):
monkeypatch.setattr(data_store, "CHUNK_SIZE", 2)
games = [make_game(s) for s in ("a", "b", "c")]
assert data_store.save_all_games(games) is True
index = read_json("data_game/index.json")
assert index["total_games"] == 3
assert [f["name"] for f in index["files"]] == ["game_info_001.json", "game_info_002.json"]
assert index["max_per_file"] == 2
assert read_json("data_game/count_history.json")[-1]["total"] == 3
def test_noop_save_changes_nothing(repo):
games = [make_game(s) for s in ("a", "b")]
seed_catalog(games)
before = {p: os.stat(p).st_mtime_ns for p in os.listdir(".") if p}
index_before = read_json("data_game/index.json")
assert data_store.save_all_games(data_store.load_all_games()) is False
assert read_json("data_game/index.json") == index_before
assert before # sanity
def test_update_bumps_last_updated_but_not_history(repo, monkeypatch):
seed_catalog([make_game("a"), make_game("b")])
history_before = read_json("data_game/count_history.json")
index_before = read_json("data_game/index.json")
from datetime import datetime
later = datetime(2099, 1, 2, 3, 4, 5, tzinfo=UTC)
monkeypatch.setattr(data_store, "_utc_now", lambda: later)
games = data_store.load_all_games()
games[0]["rating"] = "5.0"
assert data_store.save_all_games(games) is True
index = read_json("data_game/index.json")
assert index["last_updated"] == "2099-01-02T03:04:05Z" != index_before["last_updated"]
# Same total → no new count_history row.
assert read_json("data_game/count_history.json") == history_before
def test_removing_last_game_of_a_chunk_drops_the_file(repo, monkeypatch):
monkeypatch.setattr(data_store, "CHUNK_SIZE", 2)
monkeypatch.setattr(data_store, "MIN_FILL", 1)
seed_catalog([make_game(s) for s in ("a", "b", "c")])
games = [g for g in data_store.load_all_games() if g["url"].endswith("/c")]
data_store.save_all_games([g for g in data_store.load_all_games() if g not in games])
assert sorted(os.listdir("data_game")) == [
"count_history.json",
"game_info_001.json",
"index.json",
]
def test_corrupt_count_history_fails_instead_of_being_truncated(repo):
seed_catalog([make_game("a")])
broken = '\ufeff[{"date": "2026-01-01", "total": 1}'
(repo / "data_game" / "count_history.json").write_text(broken, encoding="utf-8")
with pytest.raises(ValueError):
data_store.save_all_games([make_game("a"), make_game("b")])
assert (repo / "data_game" / "count_history.json").read_text(encoding="utf-8") == broken