- queue: max on every data-write workflow (GitHub, 2026-05-07): up to 100 runs wait, first in, first out. The default kept one, and a run queueing behind a running job cancelled it. - ref: main on every checkout. A run's commit is fixed when it is created, so a run that waited behind another data job wrote shards from before that job's commit; Auto Update JSON failed four days in ten on the rebase conflict in data/index.json. - timeout-minutes everywhere, and JOB_BUDGET_MINUTES at least 20 below it for the long loops (never for the ingest, which empties its queue). - pipeline.test.ts fails on a data-write workflow missing any of these. - The admin's Pipeline and Automation pages, docs/ADMIN.md and the auto-approve comment no longer say a push evicts a waiting job; the idle gate stays, so a batch's ingest starts at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
23 lines
982 B
Bash
Executable File
23 lines
982 B
Bash
Executable File
#!/bin/bash
|
|
# The ingest job, which is also the removal job.
|
|
#
|
|
# remove_games.py applies data/blocklist.jsonl (games the maintainer removed in
|
|
# /admin or with delete_game.py) before ingest_new.py adds the queued links.
|
|
# One workflow does both: a commit to either file starts one run that settles
|
|
# the two, in that order. (It was also the only way to do it while
|
|
# `data-write` kept a single pending run and a separate removal workflow and
|
|
# the ingest kept evicting each other; the group queues them now.)
|
|
set -euo pipefail
|
|
pip install --quiet --require-hashes --only-binary :all: -r requirements.txt
|
|
python scripts/remove_games.py
|
|
python scripts/ingest_new.py
|
|
|
|
# Name the commit after what changed: the queue file is emptied only when
|
|
# ingest_new.py processed something.
|
|
if git diff --quiet -- scripts/temp_info.jsonl; then
|
|
msg="Apply removals [$(date +'%Y-%m-%d %H:%M')]"
|
|
else
|
|
msg="Ingest games [$(date +'%Y-%m-%d %H:%M')]"
|
|
fi
|
|
bash "$(dirname "$0")/commit_push.sh" "$msg"
|