poli0981andClaude Opus 5 e020be12fa feat(pipeline): rotating refresh + patch/apply pipeline, drop lists/ (v4 Phase 1) (#82)
* feat(pipeline): rotating refresh + patch/apply pipeline, drop lists/

Replace check_paid / check_alive / update_reviews / update_status: each
crawled all 2,600+ pages sequentially, overran its job timeout, saved
only at the end and has lost every run since late June (unreported:
timeouts end as "cancelled").

- scripts/refresh.py: daily slice of ~1/7 of the catalog (pending
  strikes first, then least recently checked), one GET per game for
  alive / paid / rating / status (+ release_date, thumbnail, updated_at
  backfill). Removal needs the same 404/410 or paid strike on two
  different UTC days. Paced requests, capped Retry-After back-off, stop
  after 3x 429, checkpoints and an in-script deadline. --full replaces
  force_update.py (rotating batches, editable fields preserved).
- Scanners emit a URL-keyed patch (patch.py); apply_patch.py applies it
  to the latest main and runs validate.py; bash/commit_push.sh fetches,
  re-applies and retries pushes, so concurrent writers never clobber
  each other.
- update_info.py: canonical URLs (canonical.py), skip games in the
  deleted log, keep transient failures queued (max 3 runs, tracked in
  scripts/state/), stamp added_at, never drop links queued meanwhile.
- data_store.py: games stay in their chunk, re-slice only past
  thresholds, write only changed bytes; index/count_history change only
  with data (no more no-op commits).
- validate.py (schema, enums, canonical/unique URLs, index consistency)
  + normalize one invalid safe_virus ("y" -> "Yes").
- tests/: 85 network-free pytest cases, HTML fixtures (from sample/),
  shared URL vectors and golden chunk/serialization fixtures for the
  future Worker port.
- Workflows: refresh.yml (scan -> apply -> notify), update.yml (runs on
  queue change), force_update.yml (refresh.yml full mode),
  notify-failure.yml (Discord on failure/cancel/rate limit),
  python-ci.yml, webapp-ci.yml, deploy-status.yml; draft-first release
  with one create-release job and tauri-action v1, no caches in release
  builds; SHA-pinned actions, permissions: {} defaults, reusable
  workflows pinned with explicit secrets, injection fixes; dependabot.yml.
- Remove lists/, generate_md, generate_table, log_deleted,
  deleted_games.txt, the Telegram bot-ingest path and old bash wrappers;
  update README (EN/VI), CONTRIBUTING, templates, CLAUDE.md, CHANGELOG
  and mark the retired Telegram path in the policies.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(pipeline): review findings on the refresh / ingest rewrite

Removals
- Mass-change guard: a run whose removals or new strikes exceed
  max(10, 10%) of the games it checked withholds all its removals and
  alerts (mass_change). Withheld games are marked withheld_at and stay
  withheld in later (even single-URL or rate-limited) runs until a healthy
  check clears them or a run is dispatched with allow_mass_removal.
- Checkpoints never carry removals, so a scan that dies before its guard
  cannot delete games; apply_patch re-checks the per-run limit and caps
  any patch at max(25, 5%) of the catalog (ALLOW_MASS_REMOVAL=1 overrides).
- Strikes mature by time, not calendar day: a second sighting must come
  >= 20 h after the first (strike_at), so two checks around midnight UTC no
  longer remove a game.
- Overlapping scans can't undo each other: refresh_state merges keep the
  newest check and the newest full stamp in either apply order, and a
  removal is skipped when a newer check has landed since.
- A corrupt count_history.json fails the apply instead of being replaced
  by a one-row history.

Ingest
- A second spelling of a queued game no longer drops the first copy
  before it reaches a final outcome.
- A dispatched url is processed first and kept in temp_link.json
  (queue_add) until it finishes, so a retry, deadline or rate-limit stop
  keeps it.
- Failures count as an attempt only >= 6 h apart and a 429 counts as none,
  so a burst of queued runs can't burn a link's three attempts.
- Non-string queue entries are cleaned on every apply instead of crashing
  it; {"url": ...} objects are unwrapped.

CI
- queue: max on the ingest, refresh-apply and force-update concurrency
  groups: the default queue replaced (and silently dropped) pending runs.
- Seed scripts/state/*.json and mkdir -p before git add (the pathspec
  failed on an empty patch until the directory existed on main).
- Cancelling a refresh / force-update / desktop release really stops it
  (!cancelled() instead of always()).
- deploy-status.yml becomes an hourly check of main's latest finished
  "Workers Builds: free-games-itchio-list" run (bash/cf_build_status.sh):
  GitHub never starts check_run workflows for GITHUB_TOKEN data commits.
- The reusable release notifiers still take their script from
  poli0981/.github@main; the caller comments now say so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(pipeline): close the remaining mass-removal and monitor gaps

From the second review round of the pipeline fixes:
- A strike spike now marks every strike of that run withheld_at (the
  checkpoint of a run that dies mid-spike too), so a later short,
  rate-limited or single-URL run can't remove them without an
  allow_mass_removal dispatch.
- Withheld games take at most half of a run's budget (all of it for the
  allow_mass_removal run that confirms them), and a withheld full-mode
  game moves its `full` stamp, so neither rotation stalls on them.
- refresh.yml runs one scan -> apply pipeline at a time (workflow-level
  concurrency refresh-pipeline, queue: max), so an older removal can't be
  applied after a newer healthy check of the same game.
- deploy-status.yml alerts once per failed Cloudflare build, keyed on its
  check-run id and marked only after the Discord post, instead of by age
  (which could drop an alert after a missed run, or repeat it); the
  monitor walks main's first-parent history so preview builds of merged
  branch commits are never mistaken for the site's build.
- CLAUDE.md: merge PRs with squash or a merge commit, never by fast-forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 17:52:57 +07:00
2025-12-28 16:37:25 +07:00
2025-12-20 11:57:11 +07:00

Free Itch.io Games List

Version Stars Forks Last Updated License: MIT Tiếng Việt

🇻🇳 Tiếng Việt: see README.vi.md. Vietnamese translations of the policy docs live in docs/i18n/vi/. The English versions in this repo remain controlling for legal interpretation.

A curated, auto-updating catalog of free games on itch.io. Games are scraped, validated, and published as a browsable, filterable website — https://freeitchgames.win — refreshed daily via GitHub Actions.

Table of contents

Browse the catalog

The catalog lives on the website: https://freeitchgames.win — search, filter by genre / platform / status, sort, and open any game on itch.io. (The per-genre markdown tables under lists/ were retired in favour of the site.) The raw data is plain JSON in data_game/.

Webapp (browse + edit + analytics)

A React + TypeScript SPA in webapp/ provides a browsable UI on top of the same JSON catalog: virtualized DataTable for all 2,600+ games, faceted filters (genre / status / platforms / NSFW), 16 charts (Recharts), bulk edit/delete via the GitHub Git Data API, and a one-click "add" flow that dispatches the scraper workflow with a URL input.

  • Web build: deployed to https://freeitchgames.win by Cloudflare Workers Builds from webapp/wrangler.jsonc — every push to main ships automatically (no GitHub Actions deploy). The old GitHub Pages address only redirects there.
  • Desktop build (optional): same React code wraps as a Tauri 2 native app for Windows / macOS / Linux. See webapp/TAURI.md for prerequisites and npm run tauri:dev. Multi-platform installers are built by .github/workflows/release_desktop.yml on tag push (v*).
  • Android build (optional): the same app also ships as a sideloadable .apk (arm64-v8a) via Tauri mobile. A signed APK is built by .github/workflows/release_android.yml and attached to the same draft Release. See Android (download & install) below and the Android section of webapp/TAURI.md.
  • Auth: a fine-grained PAT with contents:write + workflow:write is encrypted in localStorage with a passphrase (AES-GCM, PBKDF2-SHA256). Decrypted token only lives in memory. Reads are public (no auth needed).
  • Edits commit as chore(webapp): … so they're easy to filter from the daily scraper commits.

Local dev:

cd webapp
npm install
npm run dev          # http://localhost:5173
npm run build        # writes to webapp/dist/
npm run tauri:dev    # native desktop (requires Rust)

Android (download & install)

No Play Store — just grab the .apk and sideload it (Android is fine with this, it just asks nicely first):

  1. Open the latest Release and download FreeGamesItchio_<version>_arm64-v8a.apk.
  2. Tap it. Android will ask to allow installs from this source (browser / file manager) — turn it on (Settings → Apps → Special access → Install unknown apps).
  3. Accept the "not from Play Store" warning and install. It's the same app as the web/desktop build, just signed by me instead of Google.

Notes: arm64-v8a only (every phone since ~2017 — 32-bit-only devices aren't supported), and Android 11+ (API 30). Older versions can't install it — that's deliberate: it keeps the floor at a release that still gets security hardening and that we actually test on (≈87% of active devices; why). Building it yourself is in the Android section of webapp/TAURI.md.

How it works

temp_link.json ─→ update_info.py ─┐                     ┌─→ data_game/*.json ─→ freeitchgames.win
(queued URLs)     (scrape, free?) ├─→ patch.json ─→ apply_patch.py
data_game/     ─→ refresh.py    ──┘   (URL-keyed)    (validate + push)
(1/7 per day)     (alive / paid / rating / status)
  1. Add links — itch.io URLs are queued in scripts/temp_link.json (by the companion browser extension, or manually / via PR).
  2. Ingest — update.yml runs update_info.py as soon as the queue changes (and daily as a fallback). Each link is canonicalized, fetched, checked for free status and scraped; paid, dead, duplicate and previously removed games are skipped. Transient failures stay queued and are retried (up to 3 runs).
  3. Refresh — refresh.yml checks one seventh of the catalog every day (one request per game), so every game is re-checked weekly: dead links (404/410), games that became paid, rating and status. A game is only removed when the same problem is seen again at least 20 hours later; removals are logged with a reason, and a run that would remove an implausible number of games holds them back and alerts instead.
  4. Commit safely — both steps emit a URL-keyed patch; apply_patch.py applies it to the latest main, validates every data file (validate.py) and only then pushes, retrying if another writer got there first.

Project structure

data_game/              # Game database (chunked JSON, max 500 per file)
├── game_info_001.json
├── game_info_002.json
└── ...

scripts/
├── scraper.py          # Shared: HTTP session, pacing / 429 back-off, free detection, parsing
├── data_store.py       # Shared: chunked load/save (minimal diffs), index.json, count_history.json
├── canonical.py        # One canonical form for itch.io game URLs
├── update_info.py      # Queued URLs → patch (new games)
├── refresh.py          # Rotating health check → patch (alive / paid / rating / status); --full re-scrape
├── apply_patch.py      # Apply a patch to the latest main, then validate
├── validate.py         # Schema + consistency checks for every data file
├── temp_link.json      # Input queue for new URLs
├── deleted_games.json  # Log of removed games with reasons
└── state/              # Pipeline bookkeeping (last check, strikes, retries) — not published

bash/commit_push.sh     # fetch main → apply patch → validate → commit → push (with retries)
tests/                  # pytest suite (no network) + fixtures shared with the web Worker

.github/workflows/
├── update.yml             # Ingest queued games (on queue change + daily)
├── refresh.yml            # Daily rotating catalog check
├── force_update.yml       # Manual full re-scrape (one URL or the next batch)
├── python-ci.yml          # Lint, tests, data validation
├── webapp-ci.yml          # Lint, type-check, build (no deploy)
├── release_desktop.yml    # Build Tauri installers (Win/macOS/Linux) on v* tag
└── release_android.yml    # Build signed Android APK (arm64-v8a) on v* tag

webapp/                 # React + TS SPA + Tauri desktop & Android wrapper
├── src/                # React app (routes, components, hooks, stores)
├── src-tauri/          # Rust + Tauri 2 config (icons, capabilities, main.rs)
├── TAURI.md            # Desktop build prerequisites and instructions
├── package.json
├── vite.config.ts
└── tailwind.config.ts

Automation (GitHub Actions)

Workflow Schedule Purpose
Ingest queued games On queue change + daily 01:23 UTC Scrape links from temp_link.json (+ optional url input), skip paid / dead / removed
Refresh catalog Daily 02:47 UTC Check 1/7 of the catalog: dead links, now-paid games, rating, status
Force update Manual Re-scrape every field (one URL or the next batch); keeps safe_virus / notes / nsfw
Python CI / Webapp CI Pull requests Lint, tests, data validation / type-check and build
Deploy webapp On push to main Cloudflare Workers Builds (webapp/wrangler.jsonc) → freeitchgames.win
Release desktop On v* tag push Build Tauri installers (Win/macOS/Linux) → draft Release
Release Android On v* tag push Build signed APK (arm64-v8a) → draft Release

Scrapers identify themselves (FreeItchGamesBot), pace requests (random delays, batch pauses) and back off on HTTP 429. Network errors are treated as transient; a game is removed only after the same 404/410 or paid status is seen again at least 20 hours after the first sighting. Failures, cancellations, rate limits and suspicious mass changes are reported to Discord.

Data fields

Each game entry in data_game/game_info_NNN.json contains:

Field Description
url Game page URL
name Game title
dev Developer / author name(s)
description Short description (first sentence, max 200 chars)
genre Genre(s) from itch.io
tags All tags
status Release status
platforms Available platforms (Windows, macOS, Linux, Web)
publisher Publisher (if different from author)
release_date Full release date from page metadata
made_with Engine / tools used
rating Average rating (from itch.io aggregate)
rating_count Number of ratings
average_session Typical play session length
languages Supported languages
inputs Input methods (keyboard, mouse, gamepad)
nsfw NSFW flag (auto-detected from tags, warnings, description)
safe_virus Manual safety note (default: ?)
notes Manual notes
thumbnail Thumbnail image URL

All fields default to N/A when not available on the game page.

Contributing

See CONTRIBUTING.md for full details.

  • Add games — use the "Add New Games" issue template (max 50 links per issue).
  • Add games via Telegram bot — alternative path for batch submissions through @my_skull_bot; DM the owner to be whitelisted. Full flow + privacy notes: CONTRIBUTING §1b · Bot repo: poli0981/telegram-scraper-bot (USER_GUIDE.md).
  • Report bugs — use the "Bug Report" template.
  • Request features — open an issue or submit a PR.

Contributors are credited in ACKNOWLEDGEMENTS.md. Maintainer hardware + dev environment: docs/pc_spec.md, docs/dev_env.md.

Connect / support

Two Discord servers exist now (the "if I ever make one" disclaimer is officially obsolete):

DMs are open everywhere — replies slow, introvert max level. The About page in the webapp has the same list with one-click buttons.

Licensed under MIT.

S
Description
No description provided
Readme CC-BY-4.0
12 MiB
0 Stars 1 Watchers 0 Forks
Languages
TypeScript 81.7%
Python 13.8%
JavaScript 1.3%
Rich Text Format 1.1%
CSS 1%
Other 1.1%