5 Commits
Author SHA1 Message Date
poli0981andClaude Opus 5.5 40b17e620d ci(sonar): analyse in GitHub Actions with test coverage (#157)
SonarQube Cloud analyses in GitHub Actions instead of Automatic
Analysis, with test coverage (sonarcloud.yml): token (skips Dependabot
and fork PRs, which get no SONAR_TOKEN) → coverage (pytest-cov and
Vitest v8, no secrets) → scan (sonarqube-scan-action v8.3.0, full
history, sonar.projectVersion from webapp/package.json).

sonar-project.properties replaces .sonarcloud.properties: tests marked
as tests, coverage exclusions for code without unit tests by design,
Python 3.14 (the analyzer has no 3.15 yet), binary images and the
SQLite migrations (claimed by the PL/SQL analyzer, which only warned)
left out.
Adds pytest-cov / coverage (dev lock recompiled on Python 3.14) and
@vitest/coverage-v8 (Dependabot group with vitest). Docs: CLAUDE.md
gotcha 41, THIRD_PARTY, dev_env, CONTRIBUTING (EN + VI).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 20:30:20 +07:00
poli0981andClaude Opus 5.5 3e82f720dd Upgrade toolchain: TypeScript 7, Node 26, Python 3.15
TypeScript 7.0.2 (the native compiler) now runs `tsc -b` in `npm run build`.
It has no JavaScript API until 7.1, and typescript-eslint (peer
typescript <6.1) and scripts/i18n-edit.mjs need one, so the two are
installed side by side: `@typescript/native` = npm:typescript@^7.0.2
(bin tsc) and `typescript` = npm:@typescript/typescript6@^6.0.2 (the 6.0
API, bin tsc6). No tsconfig or source change: TS 7 reports 0 errors.

Node 26: webapp/.node-version 24 -> 26 (CI and both release workflows read
it), engines >=26, @types/node 26. npm 11.20 gates dependency install
scripts on `allowScripts`, so esbuild and workerd are approved by name.

Python 3.15: .python-version 3.14 -> 3.15, ruff target py315, urllib3
declared (scraper.py imports it). UTF-8 mode is the default since 3.15,
so the workflows drop PYTHONUTF8.

Dependabot: keep the `typescript` alias off new majors, group
@typescript/*, @types/node up to 26. Docs and CLAUDE.md follow (gotcha 6
rewritten, gotcha 38 for allowScripts).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 08:41:26 +07:00
poli0981andClaude Opus 5 e020be12fa feat(pipeline): rotating refresh + patch/apply pipeline, drop lists/ (v4 Phase 1) (#82)
* feat(pipeline): rotating refresh + patch/apply pipeline, drop lists/

Replace check_paid / check_alive / update_reviews / update_status: each
crawled all 2,600+ pages sequentially, overran its job timeout, saved
only at the end and has lost every run since late June (unreported:
timeouts end as "cancelled").

- scripts/refresh.py: daily slice of ~1/7 of the catalog (pending
  strikes first, then least recently checked), one GET per game for
  alive / paid / rating / status (+ release_date, thumbnail, updated_at
  backfill). Removal needs the same 404/410 or paid strike on two
  different UTC days. Paced requests, capped Retry-After back-off, stop
  after 3x 429, checkpoints and an in-script deadline. --full replaces
  force_update.py (rotating batches, editable fields preserved).
- Scanners emit a URL-keyed patch (patch.py); apply_patch.py applies it
  to the latest main and runs validate.py; bash/commit_push.sh fetches,
  re-applies and retries pushes, so concurrent writers never clobber
  each other.
- update_info.py: canonical URLs (canonical.py), skip games in the
  deleted log, keep transient failures queued (max 3 runs, tracked in
  scripts/state/), stamp added_at, never drop links queued meanwhile.
- data_store.py: games stay in their chunk, re-slice only past
  thresholds, write only changed bytes; index/count_history change only
  with data (no more no-op commits).
- validate.py (schema, enums, canonical/unique URLs, index consistency)
  + normalize one invalid safe_virus ("y" -> "Yes").
- tests/: 85 network-free pytest cases, HTML fixtures (from sample/),
  shared URL vectors and golden chunk/serialization fixtures for the
  future Worker port.
- Workflows: refresh.yml (scan -> apply -> notify), update.yml (runs on
  queue change), force_update.yml (refresh.yml full mode),
  notify-failure.yml (Discord on failure/cancel/rate limit),
  python-ci.yml, webapp-ci.yml, deploy-status.yml; draft-first release
  with one create-release job and tauri-action v1, no caches in release
  builds; SHA-pinned actions, permissions: {} defaults, reusable
  workflows pinned with explicit secrets, injection fixes; dependabot.yml.
- Remove lists/, generate_md, generate_table, log_deleted,
  deleted_games.txt, the Telegram bot-ingest path and old bash wrappers;
  update README (EN/VI), CONTRIBUTING, templates, CLAUDE.md, CHANGELOG
  and mark the retired Telegram path in the policies.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(pipeline): review findings on the refresh / ingest rewrite

Removals
- Mass-change guard: a run whose removals or new strikes exceed
  max(10, 10%) of the games it checked withholds all its removals and
  alerts (mass_change). Withheld games are marked withheld_at and stay
  withheld in later (even single-URL or rate-limited) runs until a healthy
  check clears them or a run is dispatched with allow_mass_removal.
- Checkpoints never carry removals, so a scan that dies before its guard
  cannot delete games; apply_patch re-checks the per-run limit and caps
  any patch at max(25, 5%) of the catalog (ALLOW_MASS_REMOVAL=1 overrides).
- Strikes mature by time, not calendar day: a second sighting must come
  >= 20 h after the first (strike_at), so two checks around midnight UTC no
  longer remove a game.
- Overlapping scans can't undo each other: refresh_state merges keep the
  newest check and the newest full stamp in either apply order, and a
  removal is skipped when a newer check has landed since.
- A corrupt count_history.json fails the apply instead of being replaced
  by a one-row history.

Ingest
- A second spelling of a queued game no longer drops the first copy
  before it reaches a final outcome.
- A dispatched url is processed first and kept in temp_link.json
  (queue_add) until it finishes, so a retry, deadline or rate-limit stop
  keeps it.
- Failures count as an attempt only >= 6 h apart and a 429 counts as none,
  so a burst of queued runs can't burn a link's three attempts.
- Non-string queue entries are cleaned on every apply instead of crashing
  it; {"url": ...} objects are unwrapped.

CI
- queue: max on the ingest, refresh-apply and force-update concurrency
  groups: the default queue replaced (and silently dropped) pending runs.
- Seed scripts/state/*.json and mkdir -p before git add (the pathspec
  failed on an empty patch until the directory existed on main).
- Cancelling a refresh / force-update / desktop release really stops it
  (!cancelled() instead of always()).
- deploy-status.yml becomes an hourly check of main's latest finished
  "Workers Builds: free-games-itchio-list" run (bash/cf_build_status.sh):
  GitHub never starts check_run workflows for GITHUB_TOKEN data commits.
- The reusable release notifiers still take their script from
  poli0981/.github@main; the caller comments now say so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(pipeline): close the remaining mass-removal and monitor gaps

From the second review round of the pipeline fixes:
- A strike spike now marks every strike of that run withheld_at (the
  checkpoint of a run that dies mid-spike too), so a later short,
  rate-limited or single-URL run can't remove them without an
  allow_mass_removal dispatch.
- Withheld games take at most half of a run's budget (all of it for the
  allow_mass_removal run that confirms them), and a withheld full-mode
  game moves its `full` stamp, so neither rotation stalls on them.
- refresh.yml runs one scan -> apply pipeline at a time (workflow-level
  concurrency refresh-pipeline, queue: max), so an older removal can't be
  applied after a newer healthy check of the same game.
- deploy-status.yml alerts once per failed Cloudflare build, keyed on its
  check-run id and marked only after the Discord post, instead of by age
  (which could drop an alert after a missed run, or repeat it); the
  monitor walks main's first-parent history so preview builds of merged
  branch commits are never mistaken for the site's build.
- CLAUDE.md: merge PRs with squash or a merge commit, never by fast-forward.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 17:52:57 +07:00
poli0981andClaude Opus 4.7 199497a5a6 release(v3.5.2): dead-code cleanup + knip/vulture tooling (#52)
- Wire in knip (webapp) and vulture (Python) as dead-code detectors
- Remove 8 unused deps, the unwired Tauri scrape-preview scaffold, and
  ~20 unused exports/types across the webapp
- Add docs/THIRD_PARTY.md attributing the dev tooling; trim TAURI.md
- Bump version badges, about.ts, and CHANGELOG to 3.5.2

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 00:42:36 +07:00
poli0981andClaude Opus 4.7 ac8db6ab1d release(v3.5.0): charts expansion, game-count history, pipeline cleanup (#46)
- Add 6 charts (game-count line, genre treemap, KPI cards, deletions
  over time, deletion reasons, most-rated) and split charts.tsx into a
  charts/ module (one file per chart, shared chart-card + palette).
- Track catalog size over time: data_store.py appends data_game/
  count_history.json daily; backfill_count_history.py seeds it from git.
- Add scripts/json_io.py with shared load/save and a URL-keyed
  dedup_deleted so the deleted-games log can't double-log re-added games.
- Add a scroll-to-top button and a ruff config (pyproject.toml).
- Refactor docs for v3.5.0 (CHANGELOG, README badges, CLAUDE.md).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 13:32:52 +07:00