Files
free-steam-games-list/.github/workflows/ingest-new.yml
T
poli0981andClaude Opus 5.5 529b8603c3 ci(pipeline): queue data-write runs and start each on the latest main
- queue: max on every data-write workflow (GitHub, 2026-05-07): up to
  100 runs wait, first in, first out. The default kept one, and a run
  queueing behind a running job cancelled it.
- ref: main on every checkout. A run's commit is fixed when it is
  created, so a run that waited behind another data job wrote shards
  from before that job's commit; Auto Update JSON failed four days in
  ten on the rebase conflict in data/index.json.
- timeout-minutes everywhere, and JOB_BUDGET_MINUTES at least 20 below
  it for the long loops (never for the ingest, which empties its queue).
- pipeline.test.ts fails on a data-write workflow missing any of these.
- The admin's Pipeline and Automation pages, docs/ADMIN.md and the
  auto-approve comment no longer say a push evicts a waiting job; the
  idle gate stays, so a batch's ingest starts at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 22:22:08 +07:00

47 lines
2.1 KiB
YAML

name : Ingest New Game Links
on :
# branches filter is REQUIRED: without it every v*/desktop-v* tag push
# matches the paths filter (tag pushes compare against an empty base),
# fires the ingest on a detached HEAD, and dies with exit 128 on push.
#
# data/blocklist.jsonl is here too: this job also applies removals
# (bash/ingest.sh runs scripts/remove_games.py first), so one run does both.
push : { branches : [ main ], paths : [ 'scripts/temp_info.jsonl', 'data/blocklist.jsonl' ] }
# A safety net, not a poll. Before data-write queued its runs (queue: max),
# a push here that waited behind a long data job was evicted the moment any
# other data-write workflow queued, and approved games sat in
# temp_info.jsonl until the next unrelated push. Nothing is evicted now, but
# a failed run still leaves its links queued. Re-running on a clear file is
# free: ingest_new.py returns immediately and `git diff --staged --quiet`
# skips the commit, so this is idempotent by construction.
schedule : [ { cron : '0 */3 * * *' } ]
workflow_dispatch :
permissions : { contents : write }
# queue: max - a run that finds another data job running waits its turn
# (first in, first out) instead of evicting the run already waiting.
concurrency :
group : data-write
cancel-in-progress : false
queue : max
jobs :
ingest :
runs-on : ubuntu-latest
# About seven seconds a game, and an /admin batch holds at most 100. No
# time budget: ingest_new.py empties the whole queue file when it is done,
# so a game it had not reached would be lost.
timeout-minutes : 120
steps :
-
uses : actions/checkout@v6
# The tip of main when the job STARTS, not the commit that triggered
# it. A run that waited behind a long data job would otherwise work on
# shards that job has since rewritten, and its push would fail to
# rebase.
with : { ref : main }
-
uses : actions/setup-python@v6
with : { python-version : '3.15' }
-
run : chmod +x ./bash/ingest.sh && ./bash/ingest.sh
env : { STEAM_API_KEY : '${{ secrets.STEAM_API_KEY }}' }