docs: games in every language
- docs/plan/19-all-languages.md: what kept them out, what every workflow does with them, the re-sweep, and what is left. - CLAUDE.md: discovery's language rules, ASCII appids, search folds but storage never does, re-cut descriptions are not news. - docs/ADMIN.md: the filters as they are now, re-sweeping a whole year, the Languages line in the candidate dialog. - README.md, docs/plan/11 (the decision changed), CHANGELOG [Unreleased]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
51b7da62b6
commit
a618aeae2f
6 files changed
+301
-10
No files matched your search
@@ -4,6 +4,54 @@ All notable changes to this awesome noob repo will be documented here.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### ✨ Added
|
||||
|
||||
- **Games in every language.** Discovery used to search Steam only for free
|
||||
games that list English, which left out 1,592 of them: mostly Simplified
|
||||
Chinese (784), Japanese (317), Korean (165) and Traditional Chinese (164)
|
||||
only, the free Japanese visual novel `//トロイメライ・ノクターン` among them.
|
||||
It now proposes every free game whatever its languages, and every release
|
||||
window swept before gets one more pass. The pipeline still asks Steam in English, so
|
||||
tags, genres and language names stay in English; a name or description the
|
||||
developer never wrote in English stays in their language.
|
||||
- **A language filter on Games**, counted like the others and part of the
|
||||
link (`?lang=Japanese`), so a reader can keep to the languages they read.
|
||||
The About page says the catalogue now covers every language.
|
||||
- **Search across scripts and widths.** Every search box (Games, the command
|
||||
palette, developers, publishers and the anti-cheat list) now matches
|
||||
half-width and full-width forms and hiragana and katakana as one: `トロイ`,
|
||||
`とろい` and `トロイ` find the same game, and `4` finds `4`. One kanji, kana
|
||||
or Hangul syllable is enough in the command palette.
|
||||
- `/admin` shows a candidate's languages, with a **No English** badge when
|
||||
English is not among them. Such games are decided like any other.
|
||||
|
||||
### 🐛 Fixed
|
||||
|
||||
- The header search no longer jumps to Games in the middle of a word typed
|
||||
with a Japanese, Chinese or Korean input method; it waits until the word is
|
||||
chosen. And it keeps its focus when it does jump: until now the next letter
|
||||
typed after the jump went nowhere, whatever the language.
|
||||
- A CSV export opens in Excel with its Japanese, Chinese and Korean names
|
||||
intact (it now starts with a byte order mark).
|
||||
- A long Chinese or Japanese description is cut at the end of a sentence
|
||||
where it can be, and never ends in `、…` or `。…`.
|
||||
- A queued store link written in full-width digits (`5118700`) is read
|
||||
as the appid it is; digits of any other script are refused. Such an appid
|
||||
used to slip through, and the next daily refresh would have stopped on it.
|
||||
The blocklist now accepts exactly the lines the site's own reader does.
|
||||
|
||||
### 🧰 Pipeline
|
||||
|
||||
- Discovery runs wait in turn: backfill windows dispatched together no longer
|
||||
cancel each other, and a window can be swept a whole year at a time
|
||||
(`max_pages`). Each run says how many candidates do not list English.
|
||||
|
||||
### 📚 Docs
|
||||
|
||||
- `docs/plan/19-all-languages.md`: what kept these games out, what every
|
||||
workflow does with them, and what is left (games not sold in the US, CJK
|
||||
fonts).
|
||||
|
||||
## [v4.5.1] – 2026-10-27 (The "Speed Bump" Edition)
|
||||
|
||||
The site's Worker now limits how fast one address can call it, behind
|
||||
|
||||
@@ -15,7 +15,13 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
|
||||
- **`appid` is not a stored field.** It is parsed out of `link` with
|
||||
`/\/app\/(\d+)/` (`web/src/lib/data-store.ts`, `extract_appid` in
|
||||
`scripts/core/data_store.py`). Never add it as a column; derive it.
|
||||
`scripts/core/data_store.py`). Never add it as a column; derive it. It is
|
||||
ASCII digits on both sides: JavaScript's `\d` is ASCII, Python's is not, so
|
||||
`extract_appid` refuses a run of digits with any non-ASCII one in it
|
||||
(`/app/730٤` is `None`, never 730) and `normalize_link()` reads a queued line
|
||||
through NFKC first, which turns a Japanese keyboard's `5118700` into
|
||||
5118700. A non-ASCII appid would miss every comparison and make
|
||||
`rotation.slice_of()` (`.encode("ascii")`) stop every refresh.
|
||||
|
||||
- **Numeric-looking fields are formatted strings, not numbers.**
|
||||
`current_players: "492,197"`, `reviews: "86% (Very Positive)"`,
|
||||
@@ -233,8 +239,10 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
`notes`, trusts `has_paid_dlc` only from a real store page (an age gate has
|
||||
no language table or tags) or a gated game's DLC prices, compares every
|
||||
value as stored (normalised, `description` truncated - and a blurb merely
|
||||
cut off later, now that `"` is one character, is written without
|
||||
counting as news), and bumps `last_updated` only on a real change. A game
|
||||
cut off somewhere else is written without counting as news: later, now
|
||||
that `"` is one character, or earlier, at a CJK sentence end since
|
||||
v4.6.0; `_fuller_cut()` takes two cuts of one text, never a rewrite), and
|
||||
bumps `last_updated` only on a real change. A game
|
||||
with empty tags, languages or `language_details` is in the page group EVERY
|
||||
day, first, capped at `MISSING_DAILY_CAP`; `--missing-only` runs just those.
|
||||
A game whose appdetails no longer call it free is LISTED in a `::warning::`,
|
||||
@@ -372,6 +380,32 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
rule a reviewer turned on there (off by default, at most 500 a day). Either
|
||||
way it is a REQUEST in `temp_info.jsonl` that `ingest_new.py` still checks
|
||||
game by game.
|
||||
- **Discovery takes games in every language** (since v4.6.0). The store search
|
||||
(`SteamClient.fetch_search_page()`) sends no `supportedlang`; pinned to
|
||||
english it left out 1,592 of 18,790 free games (2026-10-30), mostly Chinese,
|
||||
Japanese or Korean only, and without it Steam filters on no language at all.
|
||||
Rules that are easy to break:
|
||||
- The search sends `l=english`. It fixes the language of the row TEXT, never
|
||||
which rows come back: discovery skips a row whose release reads "Coming
|
||||
soon" and the backfill parses English month names.
|
||||
- Every other request still asks Steam in English, so tags, genre, language
|
||||
names, review labels and dates stay English; a name or description the
|
||||
developer never wrote in English stays in their script. That is the
|
||||
record, not a parsing error.
|
||||
- A candidate's payload carries `languages` (appdetails'
|
||||
`supported_languages`, read by `appdetails_languages()`), which `/admin`
|
||||
shows with a "No English" badge. Auto-approve treats such a game like any
|
||||
other. Rows queued earlier have no list: the Worker never rewrites a
|
||||
queued row's `payload_json`.
|
||||
- Both workflows' guard step greps the two discovery scripts for
|
||||
`save_main|save_jsonl|clear_temp|fetch_full` as WORDS, comments and
|
||||
docstrings included: none of the four may appear in them at all.
|
||||
- The `discovery` group queues (`queue: max`) in both of its workflows, so
|
||||
backfill windows dispatched together wait in turn instead of cancelling
|
||||
each other; `pipeline.test.ts` holds a group to one setting.
|
||||
- A game that declares no language (29 of them) has empty `languages` once
|
||||
ingested: Refresh Store Data's repair group takes it every day and the
|
||||
admin Quality page flags it. Accepted.
|
||||
- **Auto-approve (`worker/lib/auto-approve.ts`, run by the cron after
|
||||
reconcile) commits a batch only when every gate says yes**, and a gate it
|
||||
cannot evaluate says no: enabled (or "Run one batch now"), its D1 lease,
|
||||
@@ -1076,6 +1110,31 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
|
||||
`url.search` in markup or a load - prerender throws on it. The page number
|
||||
lives in the filter store, keyed to the criteria (`filters.pageFor`), so Back
|
||||
from a game keeps the page and a changed filter starts again at page 1.
|
||||
The language filter (`?lang=`) offers the names stored in `languages`,
|
||||
Steam's English ones: data, never i18n keys.
|
||||
- **Search folds; storage never does.** `lib/search-fold.ts` `fold()` is NFKC
|
||||
plus hiragana to katakana, applied to BOTH sides of every search box
|
||||
(`/games`' Fuse index, the palette, /developers, /publishers, the anti-cheat
|
||||
list), so `トロイ`, `とろい` and `トロイ` find one game and a full-width `4`
|
||||
or `:` its ASCII twin. Never apply it to stored text: NFKC rewrites U+3000
|
||||
and half-width kana, which `save_main()` keeps as spelling. Rules:
|
||||
- Fuse gets one `getFn` per KEY. A wrapped `Fuse.config.getFn` hands back
|
||||
`{v, i}` objects for array fields (`tags`, `developer`, `publisher`), not
|
||||
strings, and folding those type-checks and then breaks.
|
||||
- Fold the query, THEN trim: NFKC turns U+3000 into a plain space.
|
||||
- Not Fuse's `ignoreDiacritics`: it strips the dakuten, making ガ equal カ.
|
||||
And no `includeMatches`: its positions would point into folded text.
|
||||
- The header search jumps to /games only once an input method has finished
|
||||
composing (`lib/layout/header-search.ts`); the value is still written on
|
||||
every keystroke, because Android keyboards compose Latin words too. The
|
||||
jump is `goto("/games", {reset: false})`: the default reset moved focus to
|
||||
`<body>` and the next keystroke was lost. The page scrolls inside `<main>`,
|
||||
which no reset touches, so focus is all `reset` changes there.
|
||||
- The palette takes one kanji, kana or Hangul syllable where Latin needs two.
|
||||
- A CSV download starts with a byte order mark (`csvForExcel()`), or Excel
|
||||
opens CJK names as mojibake. It is the escape `"\uFEFF"` in the source;
|
||||
the editing tools turn a typed escape into the raw character, so write it
|
||||
with a script and check the bytes.
|
||||
- **`safe` is classified by `lib/safety.ts`** (`y`, `n`, `?` = not reviewed,
|
||||
`""` = never set). `/games`, `/stats` and `/health` all use it; `?` is a
|
||||
to-do, not missing data, so /health reports it as its own group.
|
||||
|
||||
@@ -78,7 +78,7 @@ npm run tauri:build # release binary into web/src-tauri/target/release/bundle
|
||||
<details>
|
||||
<summary><b>The admin at <code>/admin</code></b></summary>
|
||||
|
||||
`Discover New Games` sweeps the Steam store daily and proposes candidates into a Cloudflare D1 queue. The maintainer reviews them in a small Svelte app embedded in the Worker (never shipped to visitors): approving commits the store link to `scripts/temp_info.jsonl`, and the pipeline does the rest, checking every game again (free, a game, reachable) before it is published. Approval can also run on its own, in batches with a daily cap, once the maintainer turns it on. Decided and already-published rows are read-only. Corrections to published games are saved as override files that survive every pipeline run, and a removed game goes into `data/blocklist.jsonl`, which keeps it out of the catalogue until it is restored. An overview, the pipeline's runs and a data-quality report complete it. See [`docs/ADMIN.md`](docs/ADMIN.md).
|
||||
`Discover New Games` sweeps the Steam store daily for free games in every language, not only the ones that support English, and proposes candidates into a Cloudflare D1 queue. The maintainer reviews them in a small Svelte app embedded in the Worker (never shipped to visitors): approving commits the store link to `scripts/temp_info.jsonl`, and the pipeline does the rest, checking every game again (free, a game, reachable) before it is published. Approval can also run on its own, in batches with a daily cap, once the maintainer turns it on. Decided and already-published rows are read-only. Corrections to published games are saved as override files that survive every pipeline run, and a removed game goes into `data/blocklist.jsonl`, which keeps it out of the catalogue until it is restored. An overview, the pipeline's runs and a data-quality report complete it. See [`docs/ADMIN.md`](docs/ADMIN.md).
|
||||
|
||||
</details>
|
||||
|
||||
@@ -177,7 +177,7 @@ scripts/
|
||||
| Workflow | Trigger | What it does |
|
||||
| --------------------- | -------------------------------- | --------------------------------------------------------- |
|
||||
| Auto Update JSON | Daily 00:00 UTC | Fills in incomplete records (API + HTML scrape) |
|
||||
| Discover New Games | Daily 05:00 UTC | Propose new free games into the `/admin` review queue |
|
||||
| Discover New Games | Daily 05:00 UTC | Propose new free games, any language, to the `/admin` queue |
|
||||
| Ingest New | On `scripts/temp_info.jsonl` or `data/blocklist.jsonl` push, and every 3 h | Apply removals, then add approved games |
|
||||
| Top Online | Sun/Wed/Fri 03:00 UTC | Live player leaderboard |
|
||||
| Top Offline | 1st + 15th of month 05:00 UTC | Leaderboard for single-player F2P concurrents |
|
||||
@@ -197,7 +197,7 @@ The site itself is deployed by Cloudflare Workers Builds on every push to `main`
|
||||
### Contribute
|
||||
|
||||
New games arrive on their own: `Discover New Games` sweeps the Steam store
|
||||
daily into a review queue, and the maintainer approves them at `/admin` — by
|
||||
daily, in every language, into a review queue, and the maintainer approves them at `/admin` — by
|
||||
hand, or with an auto-approve rule that has a daily cap and is off until turned
|
||||
on. Discovery only proposes, and every approved game is checked again by the
|
||||
pipeline (free, a game, reachable) before it is published. Missing a free game
|
||||
|
||||
+22
-4
@@ -329,9 +329,11 @@ account higher up this page.
|
||||
|
||||
## 8. Working through the backlog
|
||||
|
||||
Measured 2026-09-10: Steam lists **16,668** free games matching the
|
||||
catalogue's own filters (`category1=998`, `maxprice=free`,
|
||||
`supportedlang=english`); `data/` holds **3,424**.
|
||||
Measured 2026-09-10: Steam listed **16,668** free games that support English
|
||||
(`category1=998`, `maxprice=free`, `supportedlang=english`); `data/` held
|
||||
**3,424**. Since v4.6.0 discovery takes games in every language, with no
|
||||
`supportedlang`: **18,790** free games on 2026-10-30, 1,592 of them without
|
||||
English ([docs/plan/19-all-languages.md](plan/19-all-languages.md)).
|
||||
|
||||
Two jobs close that gap, and they are not the same kind of thing:
|
||||
|
||||
@@ -360,6 +362,19 @@ a re-run reports `candidates: 0`; the scheduled run's window is the
|
||||
`SWEEP_YEAR`/`SWEEP_QUARTER` block at the top of the workflow file, which you
|
||||
bump as you go.
|
||||
|
||||
Every window swept before v4.6.0 found only games that list English, so it
|
||||
needs one more pass. A whole year at a time works once `max_pages` covers it (a
|
||||
recent year is about 30 pages of 100):
|
||||
|
||||
```bash
|
||||
gh workflow run "Backfill Discovery (TEMPORARY)" -f year=2024 -f limit=1000 -f max_pages=60
|
||||
```
|
||||
|
||||
A window is done when its log says `walked past the window`; `stopping: hit
|
||||
--max-pages` means it was cut short. Runs dispatched together wait in turn
|
||||
(`queue: max`). Each log counts the candidates without English, and a dry run
|
||||
lists their languages.
|
||||
|
||||
When the backlog is closed, delete `scripts/backfill_discover.py`, delete
|
||||
`.github/workflows/backfill-discover.yml`, and drop
|
||||
`"Backfill Discovery (TEMPORARY)"` from the allowlist in
|
||||
@@ -421,7 +436,10 @@ a dialog is open, and never interrupt IME composition.
|
||||
|
||||
Click a game (or press `e`) for its details: the payload as submitted, the
|
||||
durable decision for the game, and every other row the queue holds for the same
|
||||
appid. A `rejected` row has **Reopen** there. It moves that one row back to
|
||||
appid. **Languages** lists what the game supports, as Steam names them, with a
|
||||
**No English** badge when English is not among them; such a game is decided
|
||||
like any other, and auto-approve treats it the same. Rows queued before v4.6.0
|
||||
show "—": discovery did not record languages then. A `rejected` row has **Reopen** there. It moves that one row back to
|
||||
`pending` and deletes the rejection, but only if no other row for the game is
|
||||
open or committed and the game has no approved decision; otherwise the dialog
|
||||
says which of those stopped it. There is no bulk reopen, on purpose. A game
|
||||
|
||||
@@ -667,6 +667,7 @@ workflow_dispatch: mode=backfill, start_offset=0, max_pages=9, budget=700
|
||||
4. **Playtest apps' `type` value.** `--strict-names` covers the case either way, but the frequency is unmeasured.
|
||||
5. **Region-locked `unavailable` rate from a GitHub runner IP.** This determines whether the optional negative cache is worth building. The first backfill run's counters measure it — that is why `Stats` breaks the drop reasons out individually rather than reporting one "skipped" number.
|
||||
6. **`supportedlang=english`.** Keeping it excludes 1,537 titles (8.4%) **[measured]**. Matching the existing dataset's character is the v1 default; it is a one-line knob if you later decide the tracker should be language-neutral.
|
||||
*Decided 2026-10-30, v4.6.0: language-neutral. The search sends no `supportedlang` (1,592 titles were excluded by then); see [19-all-languages.md](19-all-languages.md).*
|
||||
|
||||
### Files this design touches
|
||||
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
# Every language — games whose store page has no English
|
||||
|
||||
## Why this note exists
|
||||
|
||||
On 2026-10-30 the maintainer asked why a free Japanese game,
|
||||
[`//トロイメライ・ノクターン`](https://store.steampowered.com/app/5118700/)
|
||||
(5118700), was nowhere in the catalogue, and for an assessment of every
|
||||
workflow a game like it passes through, a plan, and the change. This is that
|
||||
assessment and the decisions it led to. Every number below was measured on
|
||||
the day with read-only requests to Steam and the repository's own data
|
||||
(`main` at 60b710a6, 12,854 games).
|
||||
|
||||
The short version: one parameter kept them out. Discovery searched Steam with
|
||||
`supportedlang=english`, a choice made when discovery was designed
|
||||
([11-design-discovery.md](11-design-discovery.md) §6, "a one-line knob if you
|
||||
later decide the tracker should be language-neutral"). Everything after
|
||||
discovery already handled these games: 275 of them were in the catalogue,
|
||||
added before discovery existed, and were displayed, reviewed, refreshed and
|
||||
purged like any other.
|
||||
|
||||
## The example game
|
||||
|
||||
| Question | Answer |
|
||||
| --- | --- |
|
||||
| Free, a game? | `is_free: true`, `type: game` (appdetails, US, JP and VN) |
|
||||
| Sold in the US? | Yes: GetItems (US) `visible`, free, 4 reviews, 100 % |
|
||||
| Languages | `Japanese` only |
|
||||
| Store text | An English description exists; the name is Japanese only |
|
||||
| In the repository? | No: not in `data/`, the queue, `removed_games.jsonl` or the blocklist |
|
||||
|
||||
## What Steam lists
|
||||
|
||||
Steam's search (`category1=998`, `maxprice=free`), walked to the end both ways:
|
||||
|
||||
| Search | Free games |
|
||||
| --- | --- |
|
||||
| `supportedlang=english` | 17,198 |
|
||||
| No language filter | 18,790 |
|
||||
| No filter, with `l=english` or `Accept-Language: ja` | 18,790 |
|
||||
|
||||
Every game of the first list is in the second, so the difference is exactly
|
||||
**1,592 games that do not list English**. Without the parameter Steam applies
|
||||
no language filter of its own. These counts were taken from Vietnam; search
|
||||
results depend on the country (GitHub's US runners saw 17,259 English-listing
|
||||
games the same morning).
|
||||
|
||||
Supported languages of those 1,592, from `supportedlang=english,<X>` minus the
|
||||
English count (a game can count under several):
|
||||
|
||||
| Language | Games |
|
||||
| --- | --- |
|
||||
| Simplified Chinese | 784 |
|
||||
| Japanese | 317 |
|
||||
| Korean | 165 |
|
||||
| Traditional Chinese | 164 |
|
||||
| French / Spanish / Russian / German / Portuguese (Brazil) | 51 / 45 / 40 / 32 / 31 |
|
||||
| Any other single language | 23 or fewer |
|
||||
| No language declared at all | 29 |
|
||||
|
||||
Of the 1,592, **275 were already in the catalogue**; the other **1,317** were
|
||||
not. By release year:
|
||||
|
||||
| ≤2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | 2026 |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| 62 | 63 | 70 | 83 | 139 | 175 | 229 | 270 | 226 |
|
||||
|
||||
The English backlog had been swept back to 2019 (the last Backfill Discovery
|
||||
run that morning was 2019 Q4), so every window from 2019 to 2026 needed one
|
||||
more pass; older windows get these games as they are swept for the first time.
|
||||
|
||||
## What each workflow does with such a game
|
||||
|
||||
| Workflow | Before this change | Change |
|
||||
| --- | --- | --- |
|
||||
| Discover New Games (daily, ≤5 newest pages) | Never sees it | Search every language |
|
||||
| Backfill Discovery (by year or quarter) | Never sees it | Same search; re-sweep 2019-2026 |
|
||||
| `/admin` queue, auto-approve | No language rule, but nothing tells a reviewer which languages a game supports | Candidates carry their languages; "No English" badge |
|
||||
| Ingest (`ingest_new.py`, `fetch_full`) | Any link form (percent-encoded slug, raw Japanese slug, `?l=`) becomes `/app/<id>/`. Name, description and developer keep their script; tags, genre, language names, review labels and dates are English, because every request asks Steam in English | None, apart from the two points below |
|
||||
| Update Reviews, anti-cheat, Peaks, Purge | Nothing depends on the language. Reviews are GetItems' `summary_filtered`, which counts every language, so a Japanese game with Japanese reviews has its full score | None |
|
||||
| Refresh Store Data | As above. A game that declares no language keeps empty `languages`, so the daily repair group takes it every day and the admin Quality page flags it | None; accepted for the 29 such games |
|
||||
| Web app | Names and descriptions display (CJK through the system's fonts); search finds exact substrings | Language filter, CJK-aware search, CSV for Excel |
|
||||
| Region | Every request goes out as the US (runners, `country_code: US`): a game not sold there is refused as `unavailable` | Out of scope |
|
||||
|
||||
Two smaller problems turned up in the pipeline:
|
||||
|
||||
- **Digits that are not ASCII.** Python's `\d` also matches full-width
|
||||
(`5118700`, what a Japanese keyboard types) and other scripts' digits.
|
||||
A queued link in such digits, which only a pull request could add, gave an
|
||||
appid that missed every comparison and made `rotation.slice_of()` raise on
|
||||
`.encode("ascii")`, stopping Refresh Store Data, Peaks and Refetch for every
|
||||
game. The Worker and `web/shared/removal.ts` already took ASCII only;
|
||||
`scripts/core/blocklist.py` did not, and its `$` also let `"730\n"` through.
|
||||
- **Cutting a CJK blurb.** `truncate_description()` backed off only to an
|
||||
ASCII space and trimmed only Latin punctuation before the ellipsis, so a
|
||||
Chinese or Japanese blurb was cut mid-phrase, and two stored ones end in
|
||||
`。…`.
|
||||
|
||||
And in the web app:
|
||||
|
||||
- `/games` had no language filter, and `languages` is not a search key, so a
|
||||
reader of English had no way to leave out the games they cannot play.
|
||||
- Every search box only lowercased, so `トロイ` (half-width), `とろい` (hiragana)
|
||||
and `トロイ` were three different searches, as were `4` and `4`.
|
||||
- The header search jumped to `/games` on the first keystroke an input method
|
||||
had not finished composing.
|
||||
- The command palette waited for two characters; one kanji found nothing.
|
||||
- The CSV download had no byte order mark, so Excel, Japanese Windows
|
||||
included, opened every CJK name as mojibake.
|
||||
|
||||
## What changed (v4.6.0)
|
||||
|
||||
- **Discovery searches every language.** `fetch_search_page()` sends no
|
||||
`supportedlang`, and sends `l=english`, which fixes the language of the row
|
||||
text (discovery skips "Coming soon" rows, the backfill reads English month
|
||||
names) without changing which rows come back.
|
||||
- **Candidates carry their languages.** `appdetails_languages()` reads
|
||||
appdetails' `supported_languages` HTML into a list in the payload; dry runs
|
||||
print "no English: …" and each run counts the candidates without English.
|
||||
`/admin`'s candidate dialog shows the list and a "No English" badge.
|
||||
Auto-approve is unchanged: the maintainer chose to treat these games like
|
||||
any other.
|
||||
- **The `discovery` group queues** (`queue: max` in both workflows), and the
|
||||
backfill takes a `max_pages` input. `pipeline.test.ts` now holds every
|
||||
shared group to one queue setting.
|
||||
- **ASCII appids.** `extract_appid` refuses a run of digits with a non-ASCII
|
||||
one in it; `normalize_link()` reads NFKC first, so `5118700` is
|
||||
5118700; the blocklist uses ASCII digits and `fullmatch`.
|
||||
- **CJK blurbs** are cut just after the last `。!?` when that is later than
|
||||
the last space and in the last 40 %, and `、。,.:;` are trimmed before
|
||||
the ellipsis. Stored descriptions (all within the cap) never change;
|
||||
`_fuller_cut()` lets Refresh Store Data write a re-cut without counting it
|
||||
as news.
|
||||
- **Web:** a language filter on `/games` (`?lang=`, counted like the others),
|
||||
`fold()` (NFKC plus hiragana to katakana) on both sides of every search box,
|
||||
the header search waiting for the input method, one CJK character enough in
|
||||
the palette, and a byte order mark on CSV downloads. The About page says
|
||||
the catalogue covers every language.
|
||||
|
||||
The legal documents were not edited. DISCLAIMER §2 and ToS item 2 say the
|
||||
pipeline asks Steam in English and parses English answers, which is still
|
||||
true; an edit would change their hash and reopen the consent gate for
|
||||
everyone. No filter is stored under an `f2p:` key, so the privacy policy is
|
||||
unchanged too.
|
||||
|
||||
## The re-sweep
|
||||
|
||||
After the merge, Backfill Discovery was dispatched once per year from 2019 to
|
||||
2026, `limit=1000`, `max_pages=60` (a whole recent year is about 30 pages).
|
||||
A window counts as done only when its log says "walked past the window"; a
|
||||
run that reaches its limit is run again. Windows before 2019 get their
|
||||
non-English games when they are first swept.
|
||||
|
||||
## Left for later
|
||||
|
||||
- **Genre "Free To Play".** `SKIP_GENRE_TAGS` holds `"Free to Play"`, but
|
||||
appdetails writes `"Free To Play"` and the comparison is case-sensitive:
|
||||
1,084 records have that genre, 809 of them added since 2026-10-01. `genre`
|
||||
is a human-judgement field, so the fix needs its own decision.
|
||||
- **Games not sold in the US.** Every request goes out as the US; covering
|
||||
region-locked games needs another country or runner.
|
||||
- **CJK fonts and `lang`.** CJK text renders in the system's font (a Linux
|
||||
machine without one shows boxes), and no element marks a game's name with
|
||||
its language, so shared Han characters may take Chinese shapes in Japanese
|
||||
names. A self-hosted CJK font is megabytes; guessing a name's language from
|
||||
its script would be wrong for many Han-only names.
|
||||
Reference in new issue
Block a user