The scheduled pipeline has been failing daily since 2026-08-13 (last successful
`Auto update` was e8b7d1b2). Root cause: `secrets.GH_TOKEN`, a PAT, expired.
Every workflow passing it to actions/checkout dies at fetch with exit 128.
No workflow actually needs a PAT. Each already declares
`permissions: {contents: write}`, so the built-in GITHUB_TOKEN has push rights —
proven by mark-dead-games.yml, the only green data workflow and the only one
with a bare checkout. Removed the `token:` input from all 12 sites. This also
ends the silent-expiry failure mode rather than rescheduling it a year out.
`workflow_run` triggers are unaffected: GITHUB_TOKEN suppression applies to
push/issues/release events, and update-daily.yml plus the three notify-*
workflows all key off workflow_run, which the Actions service emits on run
completion regardless of the checkout token.
Also in this commit:
- ingest-from-issue.yml gated on `author_association`. Without it, any user
could open an issue titled [add-game] and have their JSON committed and
pushed to main with `contents: write`. A title match is not an authorization
check. Its `&&`/`||` commit chain is also parenthesised — left-associativity
meant it pushed even when there was nothing staged.
- bot-ingest.yml's final step deleted. It referenced `secrets.BOT_TOKEN` while
the rest of the file uses TELEGRAM_BOT_TOKEN, `cat`-ed a result.txt no step
creates, ran under `if: always()`, and interpolated a secret straight into a
shell command line. The Python step above it already does the same Telegram
callback correctly. `secrets.BOT_TOKEN` is now unreferenced.
- The snapshot feature is committed. snapshot-daily.yml gains the
`concurrency: {group: data-write}` guard every other data-writing workflow
has — without it a 23:00 run can push against jobs that take ~2 hours. The
`pip install requests` in bash/snapshot.sh is dropped: snapshot.py and
scripts/core/* import only stdlib.
- New web-ci.yml. deploy-pages.yml was the only thing running `npm run
typecheck`, so deleting it during the Cloudflare migration would leave the
frontend unchecked. Uses a plain `npm ci` — the old `npm ci || npm install`
fallback is what let the lockfile conflict above go unnoticed.
- New dependabot.yml. NOTE: an in-repo config replaces the one in repo
settings and supersedes open PR #111. Groups security updates so the
transitive backlog collapses into one PR instead of one per package.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
75 lines
2.3 KiB
Python
75 lines
2.3 KiB
Python
#!/usr/bin/env python3
|
|
"""Daily snapshot — capture per-game time-series data.
|
|
|
|
Writes one JSONL line per game with fields needed for trend charts:
|
|
{appid, name, current_players, peak_today, reviews_pct, captured_at}
|
|
|
|
Output: data/snapshots/YYYY-MM-DD.jsonl (one file per UTC day, idempotent — overwrites same-day).
|
|
|
|
Manual fields and full game records are NOT duplicated here; consumers join on appid
|
|
against the live shards if they need the rest.
|
|
"""
|
|
import json
|
|
import os
|
|
import re
|
|
import sys
|
|
from datetime import datetime, timezone
|
|
from pathlib import Path
|
|
|
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
|
from core.data_store import load_main, extract_appid
|
|
|
|
SNAPSHOTS_DIR = Path("data/snapshots")
|
|
PCT_RE = re.compile(r"(\d{1,3})%")
|
|
|
|
|
|
def _parse_players(s: str) -> int:
|
|
if not s:
|
|
return 0
|
|
cleaned = s.replace(",", "").strip()
|
|
return int(cleaned) if cleaned.isdigit() else 0
|
|
|
|
|
|
def _review_pct(reviews: str) -> int | None:
|
|
if not reviews:
|
|
return None
|
|
m = PCT_RE.search(reviews)
|
|
return int(m.group(1)) if m else None
|
|
|
|
|
|
def main() -> int:
|
|
games = load_main()
|
|
if not games:
|
|
print("No games loaded — aborting snapshot.")
|
|
return 1
|
|
SNAPSHOTS_DIR.mkdir(parents=True, exist_ok=True)
|
|
today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
|
out_path = SNAPSHOTS_DIR / f"{today}.jsonl"
|
|
captured_at = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
|
written = 0
|
|
tmp = out_path.with_suffix(".jsonl.tmp")
|
|
with tmp.open("w", encoding="utf-8") as f:
|
|
for g in games:
|
|
appid = extract_appid(g.get("link", "") or "")
|
|
if not appid:
|
|
continue
|
|
row = {
|
|
"appid": appid,
|
|
"name": g.get("name", ""),
|
|
"current_players": _parse_players(g.get("current_players", "")),
|
|
"peak_today": _parse_players(g.get("peak_today", "")),
|
|
"reviews_pct": _review_pct(g.get("reviews", "")),
|
|
"type_game": g.get("type_game", ""),
|
|
"status": g.get("status", "active"),
|
|
"captured_at": captured_at,
|
|
}
|
|
f.write(json.dumps(row, ensure_ascii=False) + "\n")
|
|
written += 1
|
|
tmp.replace(out_path)
|
|
print(f"[OK] {out_path} ({written} rows)")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|