Files
free-steam-games-list/bash/ingest.sh
T
poli0981andClaude Opus 5.5 529b8603c3 ci(pipeline): queue data-write runs and start each on the latest main
- queue: max on every data-write workflow (GitHub, 2026-05-07): up to
  100 runs wait, first in, first out. The default kept one, and a run
  queueing behind a running job cancelled it.
- ref: main on every checkout. A run's commit is fixed when it is
  created, so a run that waited behind another data job wrote shards
  from before that job's commit; Auto Update JSON failed four days in
  ten on the rebase conflict in data/index.json.
- timeout-minutes everywhere, and JOB_BUDGET_MINUTES at least 20 below
  it for the long loops (never for the ingest, which empties its queue).
- pipeline.test.ts fails on a data-write workflow missing any of these.
- The admin's Pipeline and Automation pages, docs/ADMIN.md and the
  auto-approve comment no longer say a push evicts a waiting job; the
  idle gate stays, so a batch's ingest starts at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 22:22:08 +07:00

23 lines
982 B
Bash
Executable File

#!/bin/bash
# The ingest job, which is also the removal job.
#
# remove_games.py applies data/blocklist.jsonl (games the maintainer removed in
# /admin or with delete_game.py) before ingest_new.py adds the queued links.
# One workflow does both: a commit to either file starts one run that settles
# the two, in that order. (It was also the only way to do it while
# `data-write` kept a single pending run and a separate removal workflow and
# the ingest kept evicting each other; the group queues them now.)
set -euo pipefail
pip install --quiet --require-hashes --only-binary :all: -r requirements.txt
python scripts/remove_games.py
python scripts/ingest_new.py
# Name the commit after what changed: the queue file is emptied only when
# ingest_new.py processed something.
if git diff --quiet -- scripts/temp_info.jsonl; then
msg="Apply removals [$(date +'%Y-%m-%d %H:%M')]"
else
msg="Ingest games [$(date +'%Y-%m-%d %H:%M')]"
fi
bash "$(dirname "$0")/commit_push.sh" "$msg"