Files
poli0981andClaude Opus 5.5 1da4d5940e feat(pipeline): keep the store, peaks and re-fetch runs bounded
- Refresh Store Data: the page and appdetails cycles grow past 2,000
  pages / 500 appdetails a day (7 -> 14, 30 -> 60) instead of the run,
  the slice comes in a daily order, and the store step has a budget. A
  game appdetails no longer calls free is listed in a ::warning::, the
  case the purge's two-signal rule cannot see; nothing is written.
- Refresh All-Time Peaks: 30 -> 60 days past 400 pages a day, never
  shorter, so SteamCharts is still asked at most once a month a game;
  the budget stops it with a warning like a refusal does.
- Force Re-fetch: --part K/N, oldest last_updated first, each game
  cleared on a copy that replaces it only once appdetails has answered;
  no final loop stamping every game. Parts queue one after another.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 22:22:08 +07:00

52 lines
1.9 KiB
YAML

name: Refresh All-Time Peaks
# all_time_peak - the most players a game has had at once - from SteamCharts'
# recorded peak, a thirtieth of the catalogue a day (~330 pages, 2-3 s apart:
# about a quarter of an hour), so every game is asked once a month. Past 400
# pages a day the cycle grows to 60 days and more instead: never more often
# than once a month a game. Every run also raises the field from our own
# Steam API samples, which needs no requests. scripts/refresh_peaks.py has
# the rules: the field only ever goes up, and a refusal from SteamCharts - or
# the time budget running out - ends the run early with a warning, not a
# failure.
#
# 19:30 UTC sits between Refresh Store Data (16:00, about an hour) and the
# Daily Snapshot (23:00). A data job still running only delays this one: the
# data-write group queues runs (queue: max) instead of evicting them.
#
# The one-time backfill is a manual run with the box ticked: every game, about
# four hours. Start it in the evening UTC, after Refresh Store Data.
on:
schedule: [{cron: '30 19 * * *'}]
workflow_dispatch:
inputs:
backfill:
description: 'Ask SteamCharts about every game, not just today''s slice (~4 h)'
required: false
default: false
type: boolean
permissions: {contents: write}
concurrency:
group: data-write
cancel-in-progress: false
queue: max
jobs:
refresh:
runs-on: ubuntu-latest
timeout-minutes: ${{ inputs.backfill && 340 || 45 }}
steps:
# main as it is when the job STARTS: a run that waited its turn must
# work on what the job before it committed.
- uses: actions/checkout@v6
with: {ref: main}
- uses: actions/setup-python@v6
with: {python-version: '3.15'}
- run: chmod +x ./bash/refresh_peaks.sh && ./bash/refresh_peaks.sh
env:
PEAKS_ALL: ${{ inputs.backfill }}
JOB_BUDGET_MINUTES: ${{ inputs.backfill && '300' || '25' }}