Files
poli0981andClaude Opus 5.5 1da4d5940e feat(pipeline): keep the store, peaks and re-fetch runs bounded
- Refresh Store Data: the page and appdetails cycles grow past 2,000
  pages / 500 appdetails a day (7 -> 14, 30 -> 60) instead of the run,
  the slice comes in a daily order, and the store step has a budget. A
  game appdetails no longer calls free is listed in a ::warning::, the
  case the purge's two-signal rule cannot see; nothing is written.
- Refresh All-Time Peaks: 30 -> 60 days past 400 pages a day, never
  shorter, so SteamCharts is still asked at most once a month a game;
  the budget stops it with a warning like a refusal does.
- Force Re-fetch: --part K/N, oldest last_updated first, each game
  cleared on a copy that replaces it only once appdetails has answered;
  no final loop stamping every game. Parts queue one after another.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 22:22:08 +07:00

61 lines
2.4 KiB
YAML

name: Refresh Store Data
# The store fields update_data.py fills once and never revisits (it only
# enriches INCOMPLETE records): tags, languages and the paid-DLC flag from the
# store page, every game once a week; name, header image, description,
# developer/publisher, release date, platforms, Metacritic and DRM notes from
# appdetails, every game once a month. Each daily run takes one slice of each
# rotation - about 1,400 store pages and 330 appdetails calls at 9,800 games,
# an hour. Past 2,000 pages or 500 appdetails a day a cycle doubles instead of
# the run growing, so it stays under an hour and a half.
# scripts/refresh_store_data.py explains the rotation and the rules (never
# blanks a value, never touches MANUAL_FIELDS).
#
# Then the anti-cheat sweep, committed separately: every game's anti-cheat as
# its developer declares it on Steam, about 200 GetItems requests plus a store
# page for each disclosure Steam does not name there (~70), a few minutes.
# scripts/refresh_anticheat.py has its rules: Steam wins, an override wins
# over Steam, and no disclosure never clears a value. `only: anticheat` runs
# just the sweep.
#
# 16:00 UTC is the quiet part of this repository's day: scheduled runs here
# start up to an hour late, and the morning jobs are done by then. A data job
# that is still running only delays this one - the data-write group queues
# runs (queue: max) instead of evicting the one waiting.
on:
schedule: [{cron: '0 16 * * *'}]
workflow_dispatch:
inputs:
only:
description: 'One part only: a store group, or the anti-cheat sweep; leave empty for all'
required: false
default: ''
type: choice
options: ['', 'page', 'details', 'anticheat']
permissions: {contents: write}
concurrency:
group: data-write
cancel-in-progress: false
queue: max
jobs:
refresh:
runs-on: ubuntu-latest
timeout-minutes: 150
steps:
# main, not the triggering commit: a run that waited its turn in
# data-write must work on the data as it is now, not as it was queued.
- uses: actions/checkout@v6
with: {ref: main}
- uses: actions/setup-python@v6
with: {python-version: '3.15'}
- run: chmod +x ./bash/refresh_store.sh && ./bash/refresh_store.sh
env:
REFRESH_ONLY: ${{ inputs.only }}
# The store refresh's budget (core/budget.py); the anti-cheat sweep
# after it takes a few minutes and has none.
JOB_BUDGET_MINUTES: '100'