Files
poli0981andClaude Opus 5 e020be12fa feat(pipeline): rotating refresh + patch/apply pipeline, drop lists/ (v4 Phase 1) (#82)
* feat(pipeline): rotating refresh + patch/apply pipeline, drop lists/

Replace check_paid / check_alive / update_reviews / update_status: each
crawled all 2,600+ pages sequentially, overran its job timeout, saved
only at the end and has lost every run since late June (unreported:
timeouts end as "cancelled").

- scripts/refresh.py: daily slice of ~1/7 of the catalog (pending
  strikes first, then least recently checked), one GET per game for
  alive / paid / rating / status (+ release_date, thumbnail, updated_at
  backfill). Removal needs the same 404/410 or paid strike on two
  different UTC days. Paced requests, capped Retry-After back-off, stop
  after 3x 429, checkpoints and an in-script deadline. --full replaces
  force_update.py (rotating batches, editable fields preserved).
- Scanners emit a URL-keyed patch (patch.py); apply_patch.py applies it
  to the latest main and runs validate.py; bash/commit_push.sh fetches,
  re-applies and retries pushes, so concurrent writers never clobber
  each other.
- update_info.py: canonical URLs (canonical.py), skip games in the
  deleted log, keep transient failures queued (max 3 runs, tracked in
  scripts/state/), stamp added_at, never drop links queued meanwhile.
- data_store.py: games stay in their chunk, re-slice only past
  thresholds, write only changed bytes; index/count_history change only
  with data (no more no-op commits).
- validate.py (schema, enums, canonical/unique URLs, index consistency)
  + normalize one invalid safe_virus ("y" -> "Yes").
- tests/: 85 network-free pytest cases, HTML fixtures (from sample/),
  shared URL vectors and golden chunk/serialization fixtures for the
  future Worker port.
- Workflows: refresh.yml (scan -> apply -> notify), update.yml (runs on
  queue change), force_update.yml (refresh.yml full mode),
  notify-failure.yml (Discord on failure/cancel/rate limit),
  python-ci.yml, webapp-ci.yml, deploy-status.yml; draft-first release
  with one create-release job and tauri-action v1, no caches in release
  builds; SHA-pinned actions, permissions: {} defaults, reusable
  workflows pinned with explicit secrets, injection fixes; dependabot.yml.
- Remove lists/, generate_md, generate_table, log_deleted,
  deleted_games.txt, the Telegram bot-ingest path and old bash wrappers;
  update README (EN/VI), CONTRIBUTING, templates, CLAUDE.md, CHANGELOG
  and mark the retired Telegram path in the policies.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(pipeline): review findings on the refresh / ingest rewrite

Removals
- Mass-change guard: a run whose removals or new strikes exceed
  max(10, 10%) of the games it checked withholds all its removals and
  alerts (mass_change). Withheld games are marked withheld_at and stay
  withheld in later (even single-URL or rate-limited) runs until a healthy
  check clears them or a run is dispatched with allow_mass_removal.
- Checkpoints never carry removals, so a scan that dies before its guard
  cannot delete games; apply_patch re-checks the per-run limit and caps
  any patch at max(25, 5%) of the catalog (ALLOW_MASS_REMOVAL=1 overrides).
- Strikes mature by time, not calendar day: a second sighting must come
  >= 20 h after the first (strike_at), so two checks around midnight UTC no
  longer remove a game.
- Overlapping scans can't undo each other: refresh_state merges keep the
  newest check and the newest full stamp in either apply order, and a
  removal is skipped when a newer check has landed since.
- A corrupt count_history.json fails the apply instead of being replaced
  by a one-row history.

Ingest
- A second spelling of a queued game no longer drops the first copy
  before it reaches a final outcome.
- A dispatched url is processed first and kept in temp_link.json
  (queue_add) until it finishes, so a retry, deadline or rate-limit stop
  keeps it.
- Failures count as an attempt only >= 6 h apart and a 429 counts as none,
  so a burst of queued runs can't burn a link's three attempts.
- Non-string queue entries are cleaned on every apply instead of crashing
  it; {"url": ...} objects are unwrapped.

CI
- queue: max on the ingest, refresh-apply and force-update concurrency
  groups: the default queue replaced (and silently dropped) pending runs.
- Seed scripts/state/*.json and mkdir -p before git add (the pathspec
  failed on an empty patch until the directory existed on main).
- Cancelling a refresh / force-update / desktop release really stops it
  (!cancelled() instead of always()).
- deploy-status.yml becomes an hourly check of main's latest finished
  "Workers Builds: free-games-itchio-list" run (bash/cf_build_status.sh):
  GitHub never starts check_run workflows for GITHUB_TOKEN data commits.
- The reusable release notifiers still take their script from
  poli0981/.github@main; the caller comments now say so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(pipeline): close the remaining mass-removal and monitor gaps

From the second review round of the pipeline fixes:
- A strike spike now marks every strike of that run withheld_at (the
  checkpoint of a run that dies mid-spike too), so a later short,
  rate-limited or single-URL run can't remove them without an
  allow_mass_removal dispatch.
- Withheld games take at most half of a run's budget (all of it for the
  allow_mass_removal run that confirms them), and a withheld full-mode
  game moves its `full` stamp, so neither rotation stalls on them.
- refresh.yml runs one scan -> apply pipeline at a time (workflow-level
  concurrency refresh-pipeline, queue: max), so an older removal can't be
  applied after a newer healthy check of the same game.
- deploy-status.yml alerts once per failed Cloudflare build, keyed on its
  check-run id and marked only after the Discord post, instead of by age
  (which could drop an alert after a missed run, or repeat it); the
  monitor walks main's first-parent history so preview builds of merged
  branch commits are never mistaken for the site's build.
- CLAUDE.md: merge PRs with squash or a merge commit, never by fast-forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 17:52:57 +07:00

82 lines
2.3 KiB
Python

"""Shared pytest setup: import path, no network, and a throwaway repo layout."""
import json
import os
import socket
import sys
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parents[1]
FIXTURES = ROOT / "tests" / "fixtures"
sys.path.insert(0, str(ROOT / "scripts"))
@pytest.fixture(autouse=True)
def no_network(monkeypatch):
"""Tests must never touch itch.io (or anything else)."""
def guard(*_args, **_kwargs):
raise RuntimeError("network access is disabled in tests")
monkeypatch.setattr(socket.socket, "connect", guard)
monkeypatch.setattr(socket, "create_connection", guard)
@pytest.fixture
def repo(tmp_path, monkeypatch):
"""Empty repo layout (data_game/, scripts/state/) as the working directory."""
(tmp_path / "data_game").mkdir()
(tmp_path / "scripts" / "state").mkdir(parents=True)
(tmp_path / "scripts" / "deleted_games.json").write_text("[]", encoding="utf-8")
(tmp_path / "scripts" / "temp_link.json").write_text("[]", encoding="utf-8")
monkeypatch.chdir(tmp_path)
return tmp_path
def make_game(slug: str, **overrides) -> dict:
"""A schema-complete record for https://dev.itch.io/<slug>."""
game = {
"url": f"https://dev.itch.io/{slug}",
"name": slug.title(),
"dev": "Dev",
"description": "N/A",
"genre": "Puzzle",
"status": "Released",
"publisher": "N/A",
"release_date": "N/A",
"rating": "4.0",
"rating_count": "10",
"average_session": "N/A",
"nsfw": "No",
"thumbnail": "https://img.itch.zone/aW1n/original/x.png",
"tags": [],
"platforms": ["Windows"],
"languages": [],
"inputs": [],
"made_with": [],
"safe_virus": "?",
"notes": "",
}
game.update(overrides)
return game
def load_fixture_soup(name: str):
from bs4 import BeautifulSoup
return BeautifulSoup((FIXTURES / name).read_bytes(), "html.parser")
def read_json(path) -> object:
return json.loads(Path(path).read_text(encoding="utf-8"))
def seed_catalog(games: list[dict]) -> None:
"""Write `games` into data_game/ through the real storage code."""
from data_store import save_all_games
save_all_games(games)
assert os.path.exists("data_game/index.json")