* feat(pipeline): rotating refresh + patch/apply pipeline, drop lists/
Replace check_paid / check_alive / update_reviews / update_status: each
crawled all 2,600+ pages sequentially, overran its job timeout, saved
only at the end and has lost every run since late June (unreported:
timeouts end as "cancelled").
- scripts/refresh.py: daily slice of ~1/7 of the catalog (pending
strikes first, then least recently checked), one GET per game for
alive / paid / rating / status (+ release_date, thumbnail, updated_at
backfill). Removal needs the same 404/410 or paid strike on two
different UTC days. Paced requests, capped Retry-After back-off, stop
after 3x 429, checkpoints and an in-script deadline. --full replaces
force_update.py (rotating batches, editable fields preserved).
- Scanners emit a URL-keyed patch (patch.py); apply_patch.py applies it
to the latest main and runs validate.py; bash/commit_push.sh fetches,
re-applies and retries pushes, so concurrent writers never clobber
each other.
- update_info.py: canonical URLs (canonical.py), skip games in the
deleted log, keep transient failures queued (max 3 runs, tracked in
scripts/state/), stamp added_at, never drop links queued meanwhile.
- data_store.py: games stay in their chunk, re-slice only past
thresholds, write only changed bytes; index/count_history change only
with data (no more no-op commits).
- validate.py (schema, enums, canonical/unique URLs, index consistency)
+ normalize one invalid safe_virus ("y" -> "Yes").
- tests/: 85 network-free pytest cases, HTML fixtures (from sample/),
shared URL vectors and golden chunk/serialization fixtures for the
future Worker port.
- Workflows: refresh.yml (scan -> apply -> notify), update.yml (runs on
queue change), force_update.yml (refresh.yml full mode),
notify-failure.yml (Discord on failure/cancel/rate limit),
python-ci.yml, webapp-ci.yml, deploy-status.yml; draft-first release
with one create-release job and tauri-action v1, no caches in release
builds; SHA-pinned actions, permissions: {} defaults, reusable
workflows pinned with explicit secrets, injection fixes; dependabot.yml.
- Remove lists/, generate_md, generate_table, log_deleted,
deleted_games.txt, the Telegram bot-ingest path and old bash wrappers;
update README (EN/VI), CONTRIBUTING, templates, CLAUDE.md, CHANGELOG
and mark the retired Telegram path in the policies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(pipeline): review findings on the refresh / ingest rewrite
Removals
- Mass-change guard: a run whose removals or new strikes exceed
max(10, 10%) of the games it checked withholds all its removals and
alerts (mass_change). Withheld games are marked withheld_at and stay
withheld in later (even single-URL or rate-limited) runs until a healthy
check clears them or a run is dispatched with allow_mass_removal.
- Checkpoints never carry removals, so a scan that dies before its guard
cannot delete games; apply_patch re-checks the per-run limit and caps
any patch at max(25, 5%) of the catalog (ALLOW_MASS_REMOVAL=1 overrides).
- Strikes mature by time, not calendar day: a second sighting must come
>= 20 h after the first (strike_at), so two checks around midnight UTC no
longer remove a game.
- Overlapping scans can't undo each other: refresh_state merges keep the
newest check and the newest full stamp in either apply order, and a
removal is skipped when a newer check has landed since.
- A corrupt count_history.json fails the apply instead of being replaced
by a one-row history.
Ingest
- A second spelling of a queued game no longer drops the first copy
before it reaches a final outcome.
- A dispatched url is processed first and kept in temp_link.json
(queue_add) until it finishes, so a retry, deadline or rate-limit stop
keeps it.
- Failures count as an attempt only >= 6 h apart and a 429 counts as none,
so a burst of queued runs can't burn a link's three attempts.
- Non-string queue entries are cleaned on every apply instead of crashing
it; {"url": ...} objects are unwrapped.
CI
- queue: max on the ingest, refresh-apply and force-update concurrency
groups: the default queue replaced (and silently dropped) pending runs.
- Seed scripts/state/*.json and mkdir -p before git add (the pathspec
failed on an empty patch until the directory existed on main).
- Cancelling a refresh / force-update / desktop release really stops it
(!cancelled() instead of always()).
- deploy-status.yml becomes an hourly check of main's latest finished
"Workers Builds: free-games-itchio-list" run (bash/cf_build_status.sh):
GitHub never starts check_run workflows for GITHUB_TOKEN data commits.
- The reusable release notifiers still take their script from
poli0981/.github@main; the caller comments now say so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(pipeline): close the remaining mass-removal and monitor gaps
From the second review round of the pipeline fixes:
- A strike spike now marks every strike of that run withheld_at (the
checkpoint of a run that dies mid-spike too), so a later short,
rate-limited or single-URL run can't remove them without an
allow_mass_removal dispatch.
- Withheld games take at most half of a run's budget (all of it for the
allow_mass_removal run that confirms them), and a withheld full-mode
game moves its `full` stamp, so neither rotation stalls on them.
- refresh.yml runs one scan -> apply pipeline at a time (workflow-level
concurrency refresh-pipeline, queue: max), so an older removal can't be
applied after a newer healthy check of the same game.
- deploy-status.yml alerts once per failed Cloudflare build, keyed on its
check-run id and marked only after the Discord post, instead of by age
(which could drop an alert after a missed run, or repeat it); the
monitor walks main's first-parent history so preview builds of merged
branch commits are never mistaken for the site's build.
- CLAUDE.md: merge PRs with squash or a merge commit, never by fast-forward.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Free Itch.io Games List
🇻🇳 Tiếng Việt: see
README.vi.md. Vietnamese translations of the policy docs live indocs/i18n/vi/. The English versions in this repo remain controlling for legal interpretation.
A curated, auto-updating catalog of free games on itch.io. Games are scraped, validated, and published as a browsable, filterable website — https://freeitchgames.win — refreshed daily via GitHub Actions.
Table of contents
- Browse the catalog
- Webapp (browse + edit + analytics)
- How it works
- Project structure
- Automation (GitHub Actions)
- Data fields
- Contributing
- Legal
Browse the catalog
The catalog lives on the website: https://freeitchgames.win — search, filter by genre / platform /
status, sort, and open any game on itch.io. (The per-genre markdown tables under lists/ were retired in
favour of the site.) The raw data is plain JSON in data_game/.
Webapp (browse + edit + analytics)
A React + TypeScript SPA in webapp/ provides a browsable UI on top of the same JSON
catalog: virtualized DataTable for all 2,600+ games, faceted filters (genre / status / platforms /
NSFW), 16 charts (Recharts), bulk edit/delete via the GitHub Git Data API, and a one-click "add"
flow that dispatches the scraper workflow with a URL input.
- Web build: deployed to https://freeitchgames.win by Cloudflare Workers Builds from
webapp/wrangler.jsonc— every push tomainships automatically (no GitHub Actions deploy). The old GitHub Pages address only redirects there. - Desktop build (optional): same React code wraps as a Tauri 2 native app for Windows / macOS
/ Linux. See
webapp/TAURI.mdfor prerequisites andnpm run tauri:dev. Multi-platform installers are built by.github/workflows/release_desktop.ymlon tag push (v*). - Android build (optional): the same app also ships as a sideloadable
.apk(arm64-v8a) via Tauri mobile. A signed APK is built by.github/workflows/release_android.ymland attached to the same draft Release. See Android (download & install) below and the Android section ofwebapp/TAURI.md. - Auth: a fine-grained PAT with
contents:write+workflow:writeis encrypted in localStorage with a passphrase (AES-GCM, PBKDF2-SHA256). Decrypted token only lives in memory. Reads are public (no auth needed). - Edits commit as
chore(webapp): …so they're easy to filter from the daily scraper commits.
Local dev:
cd webapp
npm install
npm run dev # http://localhost:5173
npm run build # writes to webapp/dist/
npm run tauri:dev # native desktop (requires Rust)
Android (download & install)
No Play Store — just grab the .apk and sideload it (Android is fine with this, it just
asks nicely first):
- Open the latest Release and
download
FreeGamesItchio_<version>_arm64-v8a.apk. - Tap it. Android will ask to allow installs from this source (browser / file manager) — turn it on (Settings → Apps → Special access → Install unknown apps).
- Accept the "not from Play Store" warning and install. It's the same app as the web/desktop build, just signed by me instead of Google.
Notes: arm64-v8a only (every phone since ~2017 — 32-bit-only devices aren't supported), and
Android 11+ (API 30). Older versions can't install it — that's deliberate: it keeps the floor
at a release that still gets security hardening and that we actually test on (≈87% of active
devices; why). Building it yourself is in the Android
section of webapp/TAURI.md.
How it works
temp_link.json ─→ update_info.py ─┐ ┌─→ data_game/*.json ─→ freeitchgames.win
(queued URLs) (scrape, free?) ├─→ patch.json ─→ apply_patch.py
data_game/ ─→ refresh.py ──┘ (URL-keyed) (validate + push)
(1/7 per day) (alive / paid / rating / status)
- Add links — itch.io URLs are queued in
scripts/temp_link.json(by the companion browser extension, or manually / via PR). - Ingest —
update.ymlrunsupdate_info.pyas soon as the queue changes (and daily as a fallback). Each link is canonicalized, fetched, checked for free status and scraped; paid, dead, duplicate and previously removed games are skipped. Transient failures stay queued and are retried (up to 3 runs). - Refresh —
refresh.ymlchecks one seventh of the catalog every day (one request per game), so every game is re-checked weekly: dead links (404/410), games that became paid, rating and status. A game is only removed when the same problem is seen again at least 20 hours later; removals are logged with a reason, and a run that would remove an implausible number of games holds them back and alerts instead. - Commit safely — both steps emit a URL-keyed patch;
apply_patch.pyapplies it to the latestmain, validates every data file (validate.py) and only then pushes, retrying if another writer got there first.
Project structure
data_game/ # Game database (chunked JSON, max 500 per file)
├── game_info_001.json
├── game_info_002.json
└── ...
scripts/
├── scraper.py # Shared: HTTP session, pacing / 429 back-off, free detection, parsing
├── data_store.py # Shared: chunked load/save (minimal diffs), index.json, count_history.json
├── canonical.py # One canonical form for itch.io game URLs
├── update_info.py # Queued URLs → patch (new games)
├── refresh.py # Rotating health check → patch (alive / paid / rating / status); --full re-scrape
├── apply_patch.py # Apply a patch to the latest main, then validate
├── validate.py # Schema + consistency checks for every data file
├── temp_link.json # Input queue for new URLs
├── deleted_games.json # Log of removed games with reasons
└── state/ # Pipeline bookkeeping (last check, strikes, retries) — not published
bash/commit_push.sh # fetch main → apply patch → validate → commit → push (with retries)
tests/ # pytest suite (no network) + fixtures shared with the web Worker
.github/workflows/
├── update.yml # Ingest queued games (on queue change + daily)
├── refresh.yml # Daily rotating catalog check
├── force_update.yml # Manual full re-scrape (one URL or the next batch)
├── python-ci.yml # Lint, tests, data validation
├── webapp-ci.yml # Lint, type-check, build (no deploy)
├── release_desktop.yml # Build Tauri installers (Win/macOS/Linux) on v* tag
└── release_android.yml # Build signed Android APK (arm64-v8a) on v* tag
webapp/ # React + TS SPA + Tauri desktop & Android wrapper
├── src/ # React app (routes, components, hooks, stores)
├── src-tauri/ # Rust + Tauri 2 config (icons, capabilities, main.rs)
├── TAURI.md # Desktop build prerequisites and instructions
├── package.json
├── vite.config.ts
└── tailwind.config.ts
Automation (GitHub Actions)
| Workflow | Schedule | Purpose |
|---|---|---|
| Ingest queued games | On queue change + daily 01:23 UTC | Scrape links from temp_link.json (+ optional url input), skip paid / dead / removed |
| Refresh catalog | Daily 02:47 UTC | Check 1/7 of the catalog: dead links, now-paid games, rating, status |
| Force update | Manual | Re-scrape every field (one URL or the next batch); keeps safe_virus / notes / nsfw |
| Python CI / Webapp CI | Pull requests | Lint, tests, data validation / type-check and build |
| Deploy webapp | On push to main | Cloudflare Workers Builds (webapp/wrangler.jsonc) → freeitchgames.win |
| Release desktop | On v* tag push |
Build Tauri installers (Win/macOS/Linux) → draft Release |
| Release Android | On v* tag push |
Build signed APK (arm64-v8a) → draft Release |
Scrapers identify themselves (FreeItchGamesBot), pace requests (random delays, batch pauses) and back off on
HTTP 429. Network errors are treated as transient; a game is removed only after the same 404/410 or paid status is
seen again at least 20 hours after the first sighting. Failures, cancellations, rate limits and
suspicious mass changes are reported to Discord.
Data fields
Each game entry in data_game/game_info_NNN.json contains:
| Field | Description |
|---|---|
url |
Game page URL |
name |
Game title |
dev |
Developer / author name(s) |
description |
Short description (first sentence, max 200 chars) |
genre |
Genre(s) from itch.io |
tags |
All tags |
status |
Release status |
platforms |
Available platforms (Windows, macOS, Linux, Web) |
publisher |
Publisher (if different from author) |
release_date |
Full release date from page metadata |
made_with |
Engine / tools used |
rating |
Average rating (from itch.io aggregate) |
rating_count |
Number of ratings |
average_session |
Typical play session length |
languages |
Supported languages |
inputs |
Input methods (keyboard, mouse, gamepad) |
nsfw |
NSFW flag (auto-detected from tags, warnings, description) |
safe_virus |
Manual safety note (default: ?) |
notes |
Manual notes |
thumbnail |
Thumbnail image URL |
All fields default to N/A when not available on the game page.
Contributing
See CONTRIBUTING.md for full details.
- Add games — use the "Add New Games" issue template (max 50 links per issue).
- Add games via Telegram bot — alternative path for batch submissions through @my_skull_bot; DM the owner to be whitelisted. Full flow + privacy notes: CONTRIBUTING §1b · Bot repo: poli0981/telegram-scraper-bot (USER_GUIDE.md).
- Report bugs — use the "Bug Report" template.
- Request features — open an issue or submit a PR.
Contributors are credited in ACKNOWLEDGEMENTS.md.
Maintainer hardware + dev environment: docs/pc_spec.md, docs/dev_env.md.
Connect / support
Two Discord servers exist now (the "if I ever make one" disclaimer is officially obsolete):
- Chat: Discord — Repo discussion · Game chat
- Social: X/@SkullMute0011 · YouTube/@SkullMute · Bluesky · Mastodon
- Messaging: Telegram (DM) · Telegram bot — game submission (DM your numeric ID privately, never in public channels)
- Support (totally optional, mirrors
.github/FUNDING.yml): GitHub Sponsors · Patreon · Ko-fi · Buy Me a Coffee · PayPal - Gaming: Steam profile
DMs are open everywhere — replies slow, introvert max level. The About page in the webapp has the same list with one-click buttons.
Legal
Licensed under MIT.