- queue: max on every data-write workflow (GitHub, 2026-05-07): up to 100 runs wait, first in, first out. The default kept one, and a run queueing behind a running job cancelled it. - ref: main on every checkout. A run's commit is fixed when it is created, so a run that waited behind another data job wrote shards from before that job's commit; Auto Update JSON failed four days in ten on the rebase conflict in data/index.json. - timeout-minutes everywhere, and JOB_BUDGET_MINUTES at least 20 below it for the long loops (never for the ingest, which empties its queue). - pipeline.test.ts fails on a data-write workflow missing any of these. - The admin's Pipeline and Automation pages, docs/ADMIN.md and the auto-approve comment no longer say a push evicts a waiting job; the idle gate stays, so a batch's ingest starts at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
47 lines
2.1 KiB
YAML
47 lines
2.1 KiB
YAML
name : Ingest New Game Links
|
|
on :
|
|
# branches filter is REQUIRED: without it every v*/desktop-v* tag push
|
|
# matches the paths filter (tag pushes compare against an empty base),
|
|
# fires the ingest on a detached HEAD, and dies with exit 128 on push.
|
|
#
|
|
# data/blocklist.jsonl is here too: this job also applies removals
|
|
# (bash/ingest.sh runs scripts/remove_games.py first), so one run does both.
|
|
push : { branches : [ main ], paths : [ 'scripts/temp_info.jsonl', 'data/blocklist.jsonl' ] }
|
|
# A safety net, not a poll. Before data-write queued its runs (queue: max),
|
|
# a push here that waited behind a long data job was evicted the moment any
|
|
# other data-write workflow queued, and approved games sat in
|
|
# temp_info.jsonl until the next unrelated push. Nothing is evicted now, but
|
|
# a failed run still leaves its links queued. Re-running on a clear file is
|
|
# free: ingest_new.py returns immediately and `git diff --staged --quiet`
|
|
# skips the commit, so this is idempotent by construction.
|
|
schedule : [ { cron : '0 */3 * * *' } ]
|
|
workflow_dispatch :
|
|
permissions : { contents : write }
|
|
# queue: max - a run that finds another data job running waits its turn
|
|
# (first in, first out) instead of evicting the run already waiting.
|
|
concurrency :
|
|
group : data-write
|
|
cancel-in-progress : false
|
|
queue : max
|
|
jobs :
|
|
ingest :
|
|
runs-on : ubuntu-latest
|
|
# About seven seconds a game, and an /admin batch holds at most 100. No
|
|
# time budget: ingest_new.py empties the whole queue file when it is done,
|
|
# so a game it had not reached would be lost.
|
|
timeout-minutes : 120
|
|
steps :
|
|
-
|
|
uses : actions/checkout@v6
|
|
# The tip of main when the job STARTS, not the commit that triggered
|
|
# it. A run that waited behind a long data job would otherwise work on
|
|
# shards that job has since rewritten, and its push would fail to
|
|
# rebase.
|
|
with : { ref : main }
|
|
-
|
|
uses : actions/setup-python@v6
|
|
with : { python-version : '3.15' }
|
|
-
|
|
run : chmod +x ./bash/ingest.sh && ./bash/ingest.sh
|
|
env : { STEAM_API_KEY : '${{ secrets.STEAM_API_KEY }}' }
|