docs: games in every language

- docs/plan/19-all-languages.md: what kept them out, what every workflow
  does with them, the re-sweep, and what is left.
- CLAUDE.md: discovery's language rules, ASCII appids, search folds but
  storage never does, re-cut descriptions are not news.
- docs/ADMIN.md: the filters as they are now, re-sweeping a whole year,
  the Languages line in the candidate dialog.
- README.md, docs/plan/11 (the decision changed), CHANGELOG [Unreleased].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 5.5 committed 2026-10-30 17:28:47 +07:00
1 parent 51b7da62b6
commit a618aeae2f
6 files changed
+301 -10

No files matched your search

+48
View File
@@ -4,6 +4,54 @@ All notable changes to this awesome noob repo will be documented here.
## [Unreleased]
### ✨ Added
- **Games in every language.** Discovery used to search Steam only for free
games that list English, which left out 1,592 of them: mostly Simplified
Chinese (784), Japanese (317), Korean (165) and Traditional Chinese (164)
only, the free Japanese visual novel `//トロイメライ・ノクターン` among them.
It now proposes every free game whatever its languages, and every release
window swept before gets one more pass. The pipeline still asks Steam in English, so
tags, genres and language names stay in English; a name or description the
developer never wrote in English stays in their language.
- **A language filter on Games**, counted like the others and part of the
link (`?lang=Japanese`), so a reader can keep to the languages they read.
The About page says the catalogue now covers every language.
- **Search across scripts and widths.** Every search box (Games, the command
palette, developers, publishers and the anti-cheat list) now matches
half-width and full-width forms and hiragana and katakana as one: `トロイ`,
`とろい` and `トロイ` find the same game, and `4` finds `4`. One kanji, kana
or Hangul syllable is enough in the command palette.
- `/admin` shows a candidate's languages, with a **No English** badge when
English is not among them. Such games are decided like any other.
### 🐛 Fixed
- The header search no longer jumps to Games in the middle of a word typed
with a Japanese, Chinese or Korean input method; it waits until the word is
chosen. And it keeps its focus when it does jump: until now the next letter
typed after the jump went nowhere, whatever the language.
- A CSV export opens in Excel with its Japanese, Chinese and Korean names
intact (it now starts with a byte order mark).
- A long Chinese or Japanese description is cut at the end of a sentence
where it can be, and never ends in `、…` or `。…`.
- A queued store link written in full-width digits (`5118700`) is read
as the appid it is; digits of any other script are refused. Such an appid
used to slip through, and the next daily refresh would have stopped on it.
The blocklist now accepts exactly the lines the site's own reader does.
### 🧰 Pipeline
- Discovery runs wait in turn: backfill windows dispatched together no longer
cancel each other, and a window can be swept a whole year at a time
(`max_pages`). Each run says how many candidates do not list English.
### 📚 Docs
- `docs/plan/19-all-languages.md`: what kept these games out, what every
workflow does with them, and what is left (games not sold in the US, CJK
fonts).
## [v4.5.1] – 2026-10-27 (The "Speed Bump" Edition)
The site's Worker now limits how fast one address can call it, behind
+62 -3
View File
@@ -15,7 +15,13 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
- **`appid` is not a stored field.** It is parsed out of `link` with
`/\/app\/(\d+)/` (`web/src/lib/data-store.ts`, `extract_appid` in
`scripts/core/data_store.py`). Never add it as a column; derive it.
`scripts/core/data_store.py`). Never add it as a column; derive it. It is
ASCII digits on both sides: JavaScript's `\d` is ASCII, Python's is not, so
`extract_appid` refuses a run of digits with any non-ASCII one in it
(`/app/730٤` is `None`, never 730) and `normalize_link()` reads a queued line
through NFKC first, which turns a Japanese keyboard's `5118700` into
5118700. A non-ASCII appid would miss every comparison and make
`rotation.slice_of()` (`.encode("ascii")`) stop every refresh.
- **Numeric-looking fields are formatted strings, not numbers.**
`current_players: "492,197"`, `reviews: "86% (Very Positive)"`,
@@ -233,8 +239,10 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
`notes`, trusts `has_paid_dlc` only from a real store page (an age gate has
no language table or tags) or a gated game's DLC prices, compares every
value as stored (normalised, `description` truncated - and a blurb merely
cut off later, now that `&quot;` is one character, is written without
counting as news), and bumps `last_updated` only on a real change. A game
cut off somewhere else is written without counting as news: later, now
that `&quot;` is one character, or earlier, at a CJK sentence end since
v4.6.0; `_fuller_cut()` takes two cuts of one text, never a rewrite), and
bumps `last_updated` only on a real change. A game
with empty tags, languages or `language_details` is in the page group EVERY
day, first, capped at `MISSING_DAILY_CAP`; `--missing-only` runs just those.
A game whose appdetails no longer call it free is LISTED in a `::warning::`,
@@ -372,6 +380,32 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
rule a reviewer turned on there (off by default, at most 500 a day). Either
way it is a REQUEST in `temp_info.jsonl` that `ingest_new.py` still checks
game by game.
- **Discovery takes games in every language** (since v4.6.0). The store search
(`SteamClient.fetch_search_page()`) sends no `supportedlang`; pinned to
english it left out 1,592 of 18,790 free games (2026-10-30), mostly Chinese,
Japanese or Korean only, and without it Steam filters on no language at all.
Rules that are easy to break:
- The search sends `l=english`. It fixes the language of the row TEXT, never
which rows come back: discovery skips a row whose release reads "Coming
soon" and the backfill parses English month names.
- Every other request still asks Steam in English, so tags, genre, language
names, review labels and dates stay English; a name or description the
developer never wrote in English stays in their script. That is the
record, not a parsing error.
- A candidate's payload carries `languages` (appdetails'
`supported_languages`, read by `appdetails_languages()`), which `/admin`
shows with a "No English" badge. Auto-approve treats such a game like any
other. Rows queued earlier have no list: the Worker never rewrites a
queued row's `payload_json`.
- Both workflows' guard step greps the two discovery scripts for
`save_main|save_jsonl|clear_temp|fetch_full` as WORDS, comments and
docstrings included: none of the four may appear in them at all.
- The `discovery` group queues (`queue: max`) in both of its workflows, so
backfill windows dispatched together wait in turn instead of cancelling
each other; `pipeline.test.ts` holds a group to one setting.
- A game that declares no language (29 of them) has empty `languages` once
ingested: Refresh Store Data's repair group takes it every day and the
admin Quality page flags it. Accepted.
- **Auto-approve (`worker/lib/auto-approve.ts`, run by the cron after
reconcile) commits a batch only when every gate says yes**, and a gate it
cannot evaluate says no: enabled (or "Run one batch now"), its D1 lease,
@@ -1076,6 +1110,31 @@ scraping pipeline (`scripts/`) driven by GitHub Actions.
`url.search` in markup or a load - prerender throws on it. The page number
lives in the filter store, keyed to the criteria (`filters.pageFor`), so Back
from a game keeps the page and a changed filter starts again at page 1.
The language filter (`?lang=`) offers the names stored in `languages`,
Steam's English ones: data, never i18n keys.
- **Search folds; storage never does.** `lib/search-fold.ts` `fold()` is NFKC
plus hiragana to katakana, applied to BOTH sides of every search box
(`/games`' Fuse index, the palette, /developers, /publishers, the anti-cheat
list), so `トロイ`, `とろい` and `トロイ` find one game and a full-width `4`
or `:` its ASCII twin. Never apply it to stored text: NFKC rewrites U+3000
and half-width kana, which `save_main()` keeps as spelling. Rules:
- Fuse gets one `getFn` per KEY. A wrapped `Fuse.config.getFn` hands back
`{v, i}` objects for array fields (`tags`, `developer`, `publisher`), not
strings, and folding those type-checks and then breaks.
- Fold the query, THEN trim: NFKC turns U+3000 into a plain space.
- Not Fuse's `ignoreDiacritics`: it strips the dakuten, making ガ equal カ.
And no `includeMatches`: its positions would point into folded text.
- The header search jumps to /games only once an input method has finished
composing (`lib/layout/header-search.ts`); the value is still written on
every keystroke, because Android keyboards compose Latin words too. The
jump is `goto("/games", {reset: false})`: the default reset moved focus to
`<body>` and the next keystroke was lost. The page scrolls inside `<main>`,
which no reset touches, so focus is all `reset` changes there.
- The palette takes one kanji, kana or Hangul syllable where Latin needs two.
- A CSV download starts with a byte order mark (`csvForExcel()`), or Excel
opens CJK names as mojibake. It is the escape `"\uFEFF"` in the source;
the editing tools turn a typed escape into the raw character, so write it
with a script and check the bytes.
- **`safe` is classified by `lib/safety.ts`** (`y`, `n`, `?` = not reviewed,
`""` = never set). `/games`, `/stats` and `/health` all use it; `?` is a
to-do, not missing data, so /health reports it as its own group.
+3 -3
View File
@@ -78,7 +78,7 @@ npm run tauri:build # release binary into web/src-tauri/target/release/bundle
<details>
<summary><b>The admin at <code>/admin</code></b></summary>
`Discover New Games` sweeps the Steam store daily and proposes candidates into a Cloudflare D1 queue. The maintainer reviews them in a small Svelte app embedded in the Worker (never shipped to visitors): approving commits the store link to `scripts/temp_info.jsonl`, and the pipeline does the rest, checking every game again (free, a game, reachable) before it is published. Approval can also run on its own, in batches with a daily cap, once the maintainer turns it on. Decided and already-published rows are read-only. Corrections to published games are saved as override files that survive every pipeline run, and a removed game goes into `data/blocklist.jsonl`, which keeps it out of the catalogue until it is restored. An overview, the pipeline's runs and a data-quality report complete it. See [`docs/ADMIN.md`](docs/ADMIN.md).
`Discover New Games` sweeps the Steam store daily for free games in every language, not only the ones that support English, and proposes candidates into a Cloudflare D1 queue. The maintainer reviews them in a small Svelte app embedded in the Worker (never shipped to visitors): approving commits the store link to `scripts/temp_info.jsonl`, and the pipeline does the rest, checking every game again (free, a game, reachable) before it is published. Approval can also run on its own, in batches with a daily cap, once the maintainer turns it on. Decided and already-published rows are read-only. Corrections to published games are saved as override files that survive every pipeline run, and a removed game goes into `data/blocklist.jsonl`, which keeps it out of the catalogue until it is restored. An overview, the pipeline's runs and a data-quality report complete it. See [`docs/ADMIN.md`](docs/ADMIN.md).
</details>
@@ -177,7 +177,7 @@ scripts/
| Workflow | Trigger | What it does |
| --------------------- | -------------------------------- | --------------------------------------------------------- |
| Auto Update JSON | Daily 00:00 UTC | Fills in incomplete records (API + HTML scrape) |
| Discover New Games | Daily 05:00 UTC | Propose new free games into the `/admin` review queue |
| Discover New Games | Daily 05:00 UTC | Propose new free games, any language, to the `/admin` queue |
| Ingest New | On `scripts/temp_info.jsonl` or `data/blocklist.jsonl` push, and every 3 h | Apply removals, then add approved games |
| Top Online | Sun/Wed/Fri 03:00 UTC | Live player leaderboard |
| Top Offline | 1st + 15th of month 05:00 UTC | Leaderboard for single-player F2P concurrents |
@@ -197,7 +197,7 @@ The site itself is deployed by Cloudflare Workers Builds on every push to `main`
### Contribute
New games arrive on their own: `Discover New Games` sweeps the Steam store
daily into a review queue, and the maintainer approves them at `/admin` — by
daily, in every language, into a review queue, and the maintainer approves them at `/admin` — by
hand, or with an auto-approve rule that has a daily cap and is off until turned
on. Discovery only proposes, and every approved game is checked again by the
pipeline (free, a game, reachable) before it is published. Missing a free game
+22 -4
View File
@@ -329,9 +329,11 @@ account higher up this page.
## 8. Working through the backlog
Measured 2026-09-10: Steam lists **16,668** free games matching the
catalogue's own filters (`category1=998`, `maxprice=free`,
`supportedlang=english`); `data/` holds **3,424**.
Measured 2026-09-10: Steam listed **16,668** free games that support English
(`category1=998`, `maxprice=free`, `supportedlang=english`); `data/` held
**3,424**. Since v4.6.0 discovery takes games in every language, with no
`supportedlang`: **18,790** free games on 2026-10-30, 1,592 of them without
English ([docs/plan/19-all-languages.md](plan/19-all-languages.md)).
Two jobs close that gap, and they are not the same kind of thing:
@@ -360,6 +362,19 @@ a re-run reports `candidates: 0`; the scheduled run's window is the
`SWEEP_YEAR`/`SWEEP_QUARTER` block at the top of the workflow file, which you
bump as you go.
Every window swept before v4.6.0 found only games that list English, so it
needs one more pass. A whole year at a time works once `max_pages` covers it (a
recent year is about 30 pages of 100):
```bash
gh workflow run "Backfill Discovery (TEMPORARY)" -f year=2024 -f limit=1000 -f max_pages=60
```
A window is done when its log says `walked past the window`; `stopping: hit
--max-pages` means it was cut short. Runs dispatched together wait in turn
(`queue: max`). Each log counts the candidates without English, and a dry run
lists their languages.
When the backlog is closed, delete `scripts/backfill_discover.py`, delete
`.github/workflows/backfill-discover.yml`, and drop
`"Backfill Discovery (TEMPORARY)"` from the allowlist in
@@ -421,7 +436,10 @@ a dialog is open, and never interrupt IME composition.
Click a game (or press `e`) for its details: the payload as submitted, the
durable decision for the game, and every other row the queue holds for the same
appid. A `rejected` row has **Reopen** there. It moves that one row back to
appid. **Languages** lists what the game supports, as Steam names them, with a
**No English** badge when English is not among them; such a game is decided
like any other, and auto-approve treats it the same. Rows queued before v4.6.0
show "—": discovery did not record languages then. A `rejected` row has **Reopen** there. It moves that one row back to
`pending` and deletes the rejection, but only if no other row for the game is
open or committed and the game has no approved decision; otherwise the dialog
says which of those stopped it. There is no bulk reopen, on purpose. A game
+1
View File
@@ -667,6 +667,7 @@ workflow_dispatch: mode=backfill, start_offset=0, max_pages=9, budget=700
4. **Playtest apps' `type` value.** `--strict-names` covers the case either way, but the frequency is unmeasured.
5. **Region-locked `unavailable` rate from a GitHub runner IP.** This determines whether the optional negative cache is worth building. The first backfill run's counters measure it — that is why `Stats` breaks the drop reasons out individually rather than reporting one "skipped" number.
6. **`supportedlang=english`.** Keeping it excludes 1,537 titles (8.4%) **[measured]**. Matching the existing dataset's character is the v1 default; it is a one-line knob if you later decide the tracker should be language-neutral.
*Decided 2026-10-30, v4.6.0: language-neutral. The search sends no `supportedlang` (1,592 titles were excluded by then); see [19-all-languages.md](19-all-languages.md).*
### Files this design touches
+165
View File
@@ -0,0 +1,165 @@
# Every language — games whose store page has no English
## Why this note exists
On 2026-10-30 the maintainer asked why a free Japanese game,
[`//トロイメライ・ノクターン`](https://store.steampowered.com/app/5118700/)
(5118700), was nowhere in the catalogue, and for an assessment of every
workflow a game like it passes through, a plan, and the change. This is that
assessment and the decisions it led to. Every number below was measured on
the day with read-only requests to Steam and the repository's own data
(`main` at 60b710a6, 12,854 games).
The short version: one parameter kept them out. Discovery searched Steam with
`supportedlang=english`, a choice made when discovery was designed
([11-design-discovery.md](11-design-discovery.md) §6, "a one-line knob if you
later decide the tracker should be language-neutral"). Everything after
discovery already handled these games: 275 of them were in the catalogue,
added before discovery existed, and were displayed, reviewed, refreshed and
purged like any other.
## The example game
| Question | Answer |
| --- | --- |
| Free, a game? | `is_free: true`, `type: game` (appdetails, US, JP and VN) |
| Sold in the US? | Yes: GetItems (US) `visible`, free, 4 reviews, 100 % |
| Languages | `Japanese` only |
| Store text | An English description exists; the name is Japanese only |
| In the repository? | No: not in `data/`, the queue, `removed_games.jsonl` or the blocklist |
## What Steam lists
Steam's search (`category1=998`, `maxprice=free`), walked to the end both ways:
| Search | Free games |
| --- | --- |
| `supportedlang=english` | 17,198 |
| No language filter | 18,790 |
| No filter, with `l=english` or `Accept-Language: ja` | 18,790 |
Every game of the first list is in the second, so the difference is exactly
**1,592 games that do not list English**. Without the parameter Steam applies
no language filter of its own. These counts were taken from Vietnam; search
results depend on the country (GitHub's US runners saw 17,259 English-listing
games the same morning).
Supported languages of those 1,592, from `supportedlang=english,<X>` minus the
English count (a game can count under several):
| Language | Games |
| --- | --- |
| Simplified Chinese | 784 |
| Japanese | 317 |
| Korean | 165 |
| Traditional Chinese | 164 |
| French / Spanish / Russian / German / Portuguese (Brazil) | 51 / 45 / 40 / 32 / 31 |
| Any other single language | 23 or fewer |
| No language declared at all | 29 |
Of the 1,592, **275 were already in the catalogue**; the other **1,317** were
not. By release year:
| ≤2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | 2026 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 62 | 63 | 70 | 83 | 139 | 175 | 229 | 270 | 226 |
The English backlog had been swept back to 2019 (the last Backfill Discovery
run that morning was 2019 Q4), so every window from 2019 to 2026 needed one
more pass; older windows get these games as they are swept for the first time.
## What each workflow does with such a game
| Workflow | Before this change | Change |
| --- | --- | --- |
| Discover New Games (daily, ≤5 newest pages) | Never sees it | Search every language |
| Backfill Discovery (by year or quarter) | Never sees it | Same search; re-sweep 2019-2026 |
| `/admin` queue, auto-approve | No language rule, but nothing tells a reviewer which languages a game supports | Candidates carry their languages; "No English" badge |
| Ingest (`ingest_new.py`, `fetch_full`) | Any link form (percent-encoded slug, raw Japanese slug, `?l=`) becomes `/app/<id>/`. Name, description and developer keep their script; tags, genre, language names, review labels and dates are English, because every request asks Steam in English | None, apart from the two points below |
| Update Reviews, anti-cheat, Peaks, Purge | Nothing depends on the language. Reviews are GetItems' `summary_filtered`, which counts every language, so a Japanese game with Japanese reviews has its full score | None |
| Refresh Store Data | As above. A game that declares no language keeps empty `languages`, so the daily repair group takes it every day and the admin Quality page flags it | None; accepted for the 29 such games |
| Web app | Names and descriptions display (CJK through the system's fonts); search finds exact substrings | Language filter, CJK-aware search, CSV for Excel |
| Region | Every request goes out as the US (runners, `country_code: US`): a game not sold there is refused as `unavailable` | Out of scope |
Two smaller problems turned up in the pipeline:
- **Digits that are not ASCII.** Python's `\d` also matches full-width
(`5118700`, what a Japanese keyboard types) and other scripts' digits.
A queued link in such digits, which only a pull request could add, gave an
appid that missed every comparison and made `rotation.slice_of()` raise on
`.encode("ascii")`, stopping Refresh Store Data, Peaks and Refetch for every
game. The Worker and `web/shared/removal.ts` already took ASCII only;
`scripts/core/blocklist.py` did not, and its `$` also let `"730\n"` through.
- **Cutting a CJK blurb.** `truncate_description()` backed off only to an
ASCII space and trimmed only Latin punctuation before the ellipsis, so a
Chinese or Japanese blurb was cut mid-phrase, and two stored ones end in
`。…`.
And in the web app:
- `/games` had no language filter, and `languages` is not a search key, so a
reader of English had no way to leave out the games they cannot play.
- Every search box only lowercased, so `トロイ` (half-width), `とろい` (hiragana)
and `トロイ` were three different searches, as were `4` and `4`.
- The header search jumped to `/games` on the first keystroke an input method
had not finished composing.
- The command palette waited for two characters; one kanji found nothing.
- The CSV download had no byte order mark, so Excel, Japanese Windows
included, opened every CJK name as mojibake.
## What changed (v4.6.0)
- **Discovery searches every language.** `fetch_search_page()` sends no
`supportedlang`, and sends `l=english`, which fixes the language of the row
text (discovery skips "Coming soon" rows, the backfill reads English month
names) without changing which rows come back.
- **Candidates carry their languages.** `appdetails_languages()` reads
appdetails' `supported_languages` HTML into a list in the payload; dry runs
print "no English: …" and each run counts the candidates without English.
`/admin`'s candidate dialog shows the list and a "No English" badge.
Auto-approve is unchanged: the maintainer chose to treat these games like
any other.
- **The `discovery` group queues** (`queue: max` in both workflows), and the
backfill takes a `max_pages` input. `pipeline.test.ts` now holds every
shared group to one queue setting.
- **ASCII appids.** `extract_appid` refuses a run of digits with a non-ASCII
one in it; `normalize_link()` reads NFKC first, so `5118700` is
5118700; the blocklist uses ASCII digits and `fullmatch`.
- **CJK blurbs** are cut just after the last `。!?` when that is later than
the last space and in the last 40 %, and `、。,.:;` are trimmed before
the ellipsis. Stored descriptions (all within the cap) never change;
`_fuller_cut()` lets Refresh Store Data write a re-cut without counting it
as news.
- **Web:** a language filter on `/games` (`?lang=`, counted like the others),
`fold()` (NFKC plus hiragana to katakana) on both sides of every search box,
the header search waiting for the input method, one CJK character enough in
the palette, and a byte order mark on CSV downloads. The About page says
the catalogue covers every language.
The legal documents were not edited. DISCLAIMER §2 and ToS item 2 say the
pipeline asks Steam in English and parses English answers, which is still
true; an edit would change their hash and reopen the consent gate for
everyone. No filter is stored under an `f2p:` key, so the privacy policy is
unchanged too.
## The re-sweep
After the merge, Backfill Discovery was dispatched once per year from 2019 to
2026, `limit=1000`, `max_pages=60` (a whole recent year is about 30 pages).
A window counts as done only when its log says "walked past the window"; a
run that reaches its limit is run again. Windows before 2019 get their
non-English games when they are first swept.
## Left for later
- **Genre "Free To Play".** `SKIP_GENRE_TAGS` holds `"Free to Play"`, but
appdetails writes `"Free To Play"` and the comparison is case-sensitive:
1,084 records have that genre, 809 of them added since 2026-10-01. `genre`
is a human-judgement field, so the fix needs its own decision.
- **Games not sold in the US.** Every request goes out as the US; covering
region-locked games needs another country or runner.
- **CJK fonts and `lang`.** CJK text renders in the system's font (a Linux
machine without one shows boxes), and no element marks a game's name with
its language, so shared Han characters may take Chinese shapes in Japanese
names. A self-hosted CJK font is megabytes; guessing a name's language from
its script would be wrong for many Han-only names.