Files
poli0981andClaude Opus 5.5 1da4d5940e feat(pipeline): keep the store, peaks and re-fetch runs bounded
- Refresh Store Data: the page and appdetails cycles grow past 2,000
  pages / 500 appdetails a day (7 -> 14, 30 -> 60) instead of the run,
  the slice comes in a daily order, and the store step has a budget. A
  game appdetails no longer calls free is listed in a ::warning::, the
  case the purge's two-signal rule cannot see; nothing is written.
- Refresh All-Time Peaks: 30 -> 60 days past 400 pages a day, never
  shorter, so SteamCharts is still asked at most once a month a game;
  the budget stops it with a warning like a refusal does.
- Force Re-fetch: --part K/N, oldest last_updated first, each game
  cleared on a copy that replaces it only once appdetails has answered;
  no final loop stamping every game. Parts queue one after another.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 22:22:08 +07:00

61 lines
1.8 KiB
YAML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
name : Force Re-fetch All (v2.1 Schema)
# ⚠ MANUAL ONLY – this re-fetches games from Steam from scratch, about six
# seconds a game: far longer than one job may run once the catalogue is in
# the thousands. Run it in PARTS: part "1/8" takes an eighth of the games
# (crc32 slices, scripts/refetch_all.py), and the runs for 1/8 ... 8/8 can be
# started together - the data-write group queues them (queue: max). A part cut
# short by its budget keeps what it did; starting it again carries on with the
# games it did not reach. Creates a backup at data.bak before starting.
on :
workflow_dispatch :
inputs :
confirm :
description : 'Type "yes" to confirm force re-fetch'
required : true
type : string
part :
description : 'Which part, K/N (1/8 ... 8/8 for the whole catalogue)'
required : true
default : '1/8'
type : string
permissions :
contents : write
concurrency :
group : data-write
cancel-in-progress : false
queue : max
jobs :
refetch :
if : ${{ github.event.inputs.confirm == 'yes' }}
runs-on : ubuntu-latest
timeout-minutes : 350
steps :
-
name : Checkout repo
uses : actions/checkout@v6
# main as it is when the job STARTS: a part that waited its turn must
# work on what the part before it committed.
with :
ref : main
-
name : Setup Python
uses : actions/setup-python@v6
with :
python-version : '3.15'
-
name : Force re-fetch one part
env :
STEAM_API_KEY : ${{ secrets.STEAM_API_KEY }}
REFETCH_PART : ${{ inputs.part }}
JOB_BUDGET_MINUTES : '320'
run : |
chmod +x ./bash/refetch_all.sh
./bash/refetch_all.sh