fix(workflows): force_update dispatches refresh.yml; queued runs scan the latest main (#118)
force_update.yml called refresh.yml through `uses:` and failed at startup ("workflow file issue", no annotation); it had never run since v4. It now dispatches refresh.yml with full=true (new workflow_dispatch input; refresh.yml drops workflow_call). refresh.yml's scan checks out the branch as it is when the job starts, so runs waiting in the refresh-pipeline queue see the previous run's state and consecutive force-update dispatches cover consecutive batches. Verified end to end on the branch (one URL re-scraped and applied to main as f03211d).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
f03211d5d2
commit
8ba21485ff
3 files changed
+39
-42
No files matched your search
@@ -1,7 +1,12 @@
|
||||
# Manual re-scrape of every scraper-owned field (repairs mojibake, backfills
|
||||
# fields after a scraper upgrade). Always preserves safe_virus / notes / nsfw.
|
||||
# Runs through refresh.yml in full mode: one URL, or a rotating batch of the
|
||||
# games whose last full re-scrape is oldest — dispatch again to continue.
|
||||
# Queues refresh.yml in full mode: one URL, or a rotating batch of the games
|
||||
# whose last full re-scrape is oldest. Dispatch again to continue: the runs
|
||||
# wait FIFO in refresh.yml's concurrency group and each batch starts from the
|
||||
# latest main, so several dispatches in a row cover consecutive batches.
|
||||
#
|
||||
# A dispatch rather than a reusable-workflow call: called through `uses:`,
|
||||
# refresh.yml failed at startup ("workflow file issue", no annotation).
|
||||
name: Force update
|
||||
|
||||
on:
|
||||
@@ -22,22 +27,24 @@ on:
|
||||
|
||||
permissions: {}
|
||||
|
||||
# FIFO (`queue: max`): with the default queue a newer pending dispatch would
|
||||
# replace an older pending one and silently drop its `url`.
|
||||
concurrency:
|
||||
group: force-update
|
||||
cancel-in-progress: false
|
||||
queue: max
|
||||
|
||||
jobs:
|
||||
rescrape:
|
||||
queue:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: write
|
||||
uses: ./.github/workflows/refresh.yml
|
||||
with:
|
||||
full: true
|
||||
url: ${{ inputs.url }}
|
||||
budget: ${{ inputs.budget }}
|
||||
allow_mass_removal: ${{ inputs.allow_mass_removal }}
|
||||
secrets:
|
||||
DISCORD_CI_WEBHOOK: ${{ secrets.DISCORD_CI_WEBHOOK }}
|
||||
actions: write # dispatch refresh.yml (a workflow_dispatch from GITHUB_TOKEN does start a run)
|
||||
steps:
|
||||
- name: Queue a full refresh
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
GH_REPO: ${{ github.repository }}
|
||||
REF: ${{ github.ref_name }}
|
||||
URL: ${{ inputs.url }}
|
||||
BUDGET: ${{ inputs.budget }}
|
||||
ALLOW_MASS_REMOVAL: ${{ inputs.allow_mass_removal }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
gh workflow run refresh.yml --ref "$REF" \
|
||||
-f full=true -f budget="$BUDGET" -f url="$URL" -f allow_mass_removal="$ALLOW_MASS_REMOVAL"
|
||||
echo "Queued a full refresh on \`$REF\` (budget $BUDGET${URL:+, $URL}) — see the Refresh catalog runs." \
|
||||
>> "$GITHUB_STEP_SUMMARY"
|
||||
@@ -9,7 +9,8 @@
|
||||
# (a run that would remove or strike an implausible share of games
|
||||
# — usually itch.io markup changes — withholds the removals).
|
||||
#
|
||||
# Also called by force_update.yml with `full: true` (re-scrape every field).
|
||||
# Also dispatched with `full: true` (by force_update.yml, or by hand): re-scrape
|
||||
# every scraper-owned field of one URL or of the next batch.
|
||||
name: Refresh catalog
|
||||
|
||||
on:
|
||||
@@ -29,31 +30,17 @@ on:
|
||||
description: 'Apply removals even if the mass-change guard trips (confirm on itch.io first)'
|
||||
type: boolean
|
||||
default: false
|
||||
workflow_call:
|
||||
inputs:
|
||||
budget:
|
||||
type: number
|
||||
default: 0
|
||||
url:
|
||||
type: string
|
||||
required: false
|
||||
full:
|
||||
description: 'Re-scrape every scraper-owned field (rotates on the last full re-scrape)'
|
||||
type: boolean
|
||||
default: false
|
||||
allow_mass_removal:
|
||||
type: boolean
|
||||
default: false
|
||||
secrets:
|
||||
DISCORD_CI_WEBHOOK:
|
||||
required: false
|
||||
|
||||
permissions: {}
|
||||
|
||||
# One refresh pipeline (scan → apply) at a time, FIFO — a scheduled refresh, a
|
||||
# manual one and a force-update batch never start from the same state, so an
|
||||
# older removal can't undo a newer healthy check. The name differs from
|
||||
# force_update.yml's own group on purpose: a called workflow asking for its
|
||||
# caller's group is cancelled as a deadlock.
|
||||
# manual one and a force-update batch never start from the same state (the scan
|
||||
# checks out the branch as it is when the run starts), so an older removal can't
|
||||
# undo a newer healthy check, and queued full batches move on through the catalog.
|
||||
concurrency:
|
||||
group: refresh-pipeline
|
||||
cancel-in-progress: false
|
||||
@@ -73,9 +60,12 @@ jobs:
|
||||
rate_limited: ${{ steps.refresh.outputs.rate_limited }}
|
||||
mass_change: ${{ steps.refresh.outputs.mass_change }}
|
||||
steps:
|
||||
# The branch's latest commit, not the one it had when the run was queued:
|
||||
# a run waiting behind another must see that run's refresh state.
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||
with:
|
||||
ref: ${{ github.ref }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
|
||||
@@ -75,8 +75,8 @@ Each game has **20** fields (plus optional `added_at` / `updated_at`, appended b
|
||||
| Workflow | Trigger | Notes |
|
||||
|---|---|---|
|
||||
| `update.yml` | Push to `main` touching `scripts/temp_link.json` + daily 01:23 UTC + `workflow_dispatch` (`url`) | Ingest queue → patch → `commit_push.sh`. Workflow-level `concurrency: ingest` with **`queue: max`** (the default queue replaces a pending run, which would drop its dispatched `url`). A dispatched `url` is processed first and written to `temp_link.json` (`queue_add`) until it reaches a final outcome. Failed fetches count as an attempt only ≥ 6 h apart (`last_attempt`; 3 → given up), and a 429 counts as none. GitHub-App pushes (extension, admin Worker) trigger it; its own `GITHUB_TOKEN` push does not (no loop). Keep the filename. |
|
||||
| `refresh.yml` | Daily 02:47 UTC + `workflow_dispatch` (`budget`, `url`, `allow_mass_removal`) + `workflow_call` | scan (read-only, 55-min step timeout, patch artifact uploaded `if: always()`) → apply (`!cancelled()`: a failed / timed-out scan's checkpoint is applied, a cancelled run's is not) → notify. Workflow-level `concurrency: refresh-pipeline` + `queue: max` runs one whole scan → apply at a time (also for force-update's call; the group name must differ from the caller's or GitHub cancels it as a deadlock). Removal needs the same 404/410 or paid strike seen again ≥ 20 h after the first sighting (`strike_at`). Checkpoints never carry removals. **Mass-change guard:** a run whose removals or new strikes exceed max(10, 10 %) of the games it checked withholds all its removals and marks them — and, on a strike spike, all its strikes — `withheld_at`; later runs keep withholding those (they take at most half a run's budget) until a healthy check clears them, and alert (`mass_change`) — that pattern means itch.io markup changed; dispatch with `allow_mass_removal` only after checking on itch.io. `apply_patch.py` re-checks the same per-run limit and max(25, 5 %) of the catalog unless `ALLOW_MASS_REMOVAL=1`. |
|
||||
| `force_update.yml` | `workflow_dispatch` ONLY (`url`, `budget`, `allow_mass_removal`) | `queue: max`. Calls `refresh.yml` with `full: true`: re-scrapes every scraper field for one URL or the next batch (rotates on `full` timestamps). Always preserves `safe_virus` / `notes` / `nsfw`. Canonical tool for repairing mojibake. |
|
||||
| `refresh.yml` | Daily 02:47 UTC + `workflow_dispatch` (`budget`, `url`, `allow_mass_removal`, `full`) | scan (read-only, 55-min step timeout, checks out the branch as it is when the job starts — not the queued SHA — so runs waiting in the queue see the previous run's state; patch artifact uploaded `if: always()`) → apply (`!cancelled()`: a failed / timed-out scan's checkpoint is applied, a cancelled run's is not) → notify. Workflow-level `concurrency: refresh-pipeline` + `queue: max` runs one whole scan → apply at a time (scheduled, manual and force-update runs alike). Removal needs the same 404/410 or paid strike seen again ≥ 20 h after the first sighting (`strike_at`). Checkpoints never carry removals. **Mass-change guard:** a run whose removals or new strikes exceed max(10, 10 %) of the games it checked withholds all its removals and marks them — and, on a strike spike, all its strikes — `withheld_at`; later runs keep withholding those (they take at most half a run's budget) until a healthy check clears them, and alert (`mass_change`) — that pattern means itch.io markup changed; dispatch with `allow_mass_removal` only after checking on itch.io. `apply_patch.py` re-checks the same per-run limit and max(25, 5 %) of the catalog unless `ALLOW_MASS_REMOVAL=1`. |
|
||||
| `force_update.yml` | `workflow_dispatch` ONLY (`url`, `budget`, `allow_mass_removal`) | Dispatches `refresh.yml` with `full: true` (`actions: write`; it used to call it through `uses:`, which failed at startup with no annotation): re-scrapes every scraper field for one URL or the next batch (rotates on `full` timestamps). Dispatch it N times in a row to cover N consecutive batches (FIFO, each from the latest main). Always preserves `safe_virus` / `notes` / `nsfw`. Canonical tool for repairing mojibake. |
|
||||
| `notify-failure.yml` | `workflow_call` | Discord (`DISCORD_CI_WEBHOOK`) on failure / **cancelled** (job-level timeouts end as cancelled) / rate-limited / mass change / failed Cloudflare build. |
|
||||
| `python-ci.yml` / `webapp-ci.yml` | PRs (+ pushes for Python) | ruff + format + vulture + pytest + `validate.py` / `npm run cf:types -- --check` + lint + knip + vitest + build + `wrangler deploy --dry-run --env=""`. |
|
||||
| `deploy-status.yml` | Hourly + manual | Not a deploy. `bash/cf_build_status.sh` walks main's first-parent history to the latest finished `Workers Builds: free-games-itchio-list` check run (Cloudflare posts it only when a build ends; branch commits carry preview builds under the same name) and alerts once per failed build (cache key per check-run id, saved after the Discord post). Scheduled because GitHub never starts `check_run` workflows for `GITHUB_TOKEN` pushes (the data commits). |
|
||||
@@ -122,7 +122,7 @@ Code **MIT** (`LICENSE`); catalog data **CC BY 4.0** for the maintainer's contri
|
||||
14. **With `run_worker_first` as an array, the asset layer's SPA fallback answers every miss before the Worker** — a stale hashed chunk got `index.html` + the year-long immutable cache. Hence `not_found_handling: "none"` + `spa.ts`.
|
||||
15. **Never share an in-flight promise across requests in a Worker** (cache only resolved values): a pending subrequest dies with the request that started it (client disconnect) and every later awaiter hangs.
|
||||
16. **itch.io cover URLs are signed per size**, so you can't rewrite `/original/` to a smaller variant — resizing goes through Image Transformations (`cf.image`) and R2.
|
||||
17. **GitHub Actions concurrency keeps at most one *pending* run per group by default** and cancels the older pending one — use `queue: max` where a queued run carries unique input. A called workflow must not request its caller's group (deadlock → cancelled).
|
||||
17. **GitHub Actions concurrency keeps at most one *pending* run per group by default** and cancels the older pending one — use `queue: max` where a queued run carries unique input. A called workflow must not request its caller's group (deadlock → cancelled). **A queued run checks out the SHA from when it was triggered** (`workflow_dispatch`: the branch head at dispatch time), so runs waiting behind each other read the same state unless the job checks out `ref: ${{ github.ref }}` (refresh.yml's scan does).
|
||||
18. **Pushes made with `GITHUB_TOKEN` trigger no workflows** (no ingest loop), and GitHub never starts `check_run` workflows for them (hence the scheduled `deploy-status.yml`). Workers Builds posts its check run only when a build ends.
|
||||
19. **`wrangler types` reads `.dev.vars`** — `npm run cf:types` points it at the empty `types.env` so the committed `worker-configuration.d.ts` never depends on local secrets. Don't use `secrets.required` until every environment has the secrets (deploys fail without them).
|
||||
20. **D1 migrations are applied by hand** (`npx wrangler d1 migrations apply freeitchgames --remote`), never in the build — and before deploying code that needs them. Production `freeitchgames` was created by command (APAC); its `database_id` is in `wrangler.jsonc` (not a secret). **R2 must be enabled on the account** (dashboard, payment method) before `wrangler r2 bucket create`; a deploy whose bucket is missing fails.
|
||||
@@ -146,7 +146,7 @@ Code **MIT** (`LICENSE`); catalog data **CC BY 4.0** for the maintainer's contri
|
||||
|
||||
## Current state (Oct 2026)
|
||||
|
||||
**v4.3.0** (2026-10-10): itch.io's newer info rows in the catalog — `ai_disclosure` / `ai_content` (generative-AI disclosure) and `accessibility` — on the game page, as `/games` filters (`ai`, `access`) and as charts (Overview: "Generative AI"; Discovery: "Accessibility Features", "What AI Was Used For"); `description` = the creator's tagline. Rolled out with a full re-scrape: after the merge, `force_update.yml` dispatched ~6× (budget ≈550, one ~45-min batch each, FIFO); the daily refresh fills the new fields in on its own within a week. A 250-page sample before the release: No AI 47 %, AI Assisted 10 %, not disclosed 43 %; accessibility listed on ≈20 % of games.
|
||||
**v4.3.0** (2026-10-10): itch.io's newer info rows in the catalog — `ai_disclosure` / `ai_content` (generative-AI disclosure) and `accessibility` — on the game page, as `/games` filters (`ai`, `access`) and as charts (Overview: "Generative AI"; Discovery: "Accessibility Features", "What AI Was Used For"); `description` = the creator's tagline. Rolled out with a full re-scrape: after the merge, `force_update.yml` dispatched 6× (budget 600, one ~41-min batch each at ≈14.5 games/min, FIFO); the daily refresh fills the new fields in on its own within a week. A 250-page sample before the release: No AI 47 %, AI Assisted 10 %, not disclosed 43 %; accessibility listed on ≈20 % of games.
|
||||
|
||||
**v4.2.0** (2026-10-03, PR #103): catalog packs + lazy descriptions (first visit ≈ 264 KB instead of ≈ 520 KB), compressed IndexedDB cache, `/data/*.json` only for apps 4.0–4.1.1 (410 + repository links for everyone else), `/img` cache fixes, themed scrollbars + "Hide scrollbars", policies (`LEGAL_VERSION` 2026-10-03: data reuse goes through the repository). Checked on the live site the same day: TLS 1.2/1.3 only, HSTS preload header (domain not on the preload list), Cloudflare compression already on (zstd/br), Smart Tiered Cache on but irrelevant here (`docs/operations.md` §1 step 12).
|
||||
|
||||
|
||||
Reference in new issue
Block a user