changelog/v<major>/<major>.<minor>.x/<version>.md for each of the 42 releases from v1.0.0 to v4.7.1, and changelog/beta/ for pre-releases (none has been published; its README says how one would be filed). CHANGELOG.md becomes the index: what the layout is, [Unreleased] exactly as it was, and a table per major version (version, date, edition) that links each file. Each file is its section, copied byte for byte except that its heading is the file's H1 (so the sub-headings move up one level, outside code fences), relative links gain ../../../ so they name the same repository paths, and the --- separators between sections are gone. Checked by reversing all three changes on every file and comparing with the original section: 40 of 40 equal. Two releases had no section: v1.0.0 (its GitHub release notes, which the changelog pointed to as "Initial stable") and v1.1.3 (its tag message). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.3 KiB
5.3 KiB
v2.1.0 - 2026-03-28 (The "Extension-Ready + Performance" Edition)
💥 Breaking Changes
descfield removed from schema. Usedescriptiononly. Migration script handles upgrade automatically.free_typefield removed (unreliable from API, extension handles classification client-side).jsonlinespip dependency dropped – all I/O now uses stdlibjson(faster, zero deps).fetcher.pyfunction renamed:_process_batch()→process_batch()(public API).
🏗️ Schema v2.1 (Extension-Compatible)
- 7 new fields added to every game record, matching Chrome extension output:
publisher(list) – e.g.["Square Enix", "Feral Interactive (Mac)"]platforms(list) – e.g.["Windows", "macOS", "Linux"]languages(list) – e.g.["English", "French", "Japanese"]language_details(list[dict]) – per-language{name, interface, audio, subtitles}booleanstags(list) – all user-defined Steam tags including hidden overflowanti_cheat_note(str) – raw anti-cheat name from store pageis_kernel_ac(bool|null) –true= kernel-level,false= non-kernel,null= unknown
developernormalized: string → list everywhere (API, extension, migration).descmerged intodescription, then dropped. One field, no duplication.
🔌 Extension Data Integration
- New
merge_extension_data()indata_store.py– smart merge with 3-tier priority:- MANUAL_FIELDS (notes, safe, type_game, genre, anti_cheat): only fill if empty (preserves user edits).
- ARRAY_FIELDS (developer, publisher, platforms, languages, tags): replace wholesale (extension data is richer).
- Other fields: overwrite if extension provides non-empty value.
ingest_new.pynow accepts full extension output (20+ fields per entry) – merges before fetch, re-applies manual overrides after fetch.- Ingest summary shows field counts:
dev=DONTNOD plat=[Windows, macOS, Linux] langs=6 tags=20.
🔍 HTML Scraper (New Module)
- New
scraper.py– extracts data unavailable from Steam API:- Language table: parses
<table class="game_language_options">→ accurate Interface/Audio/Subtitles per language (API only gives a broken HTML string with no subtitle info). - DLC pricing: parses
#gameAreaDLCSection→ checksdata-price-finalper DLC row. Only markshas_paid_dlc=trueif at least one DLC has actual non-zero price (API just lists DLC IDs with no prices). - Tags: parses
.app_tagelements includingdisplay:noneoverflow tags (APIappdetailsdoesn't return user tags at all).
- Language table: parses
- All 3 extracted from single GET request to store page – one HTML fetch per game, not three.
- 8 regex patterns pre-compiled at module level.
⚡ Performance Optimizations
- Eliminated double-fetch in ingest:
health_checkernow caches API response inHealthResult.data.ingest_new.pypassesprefetched_details=health.datatofetch_full()→ skips redundantfetch_app_details()call. Saves 1 API request per game (300 games = ~450s saved). - Dropped jsonlines dependency:
load_jsonl()usesjson.loads()line-by-line,save_jsonl()usesjson.dumps()+f.write()directly. ~20-30% faster I/O for large files, zero pip deps for core. - Pre-compiled regex:
scraper.pycompiles all 8 patterns at import time, not per-call. 300 games = 8 compiles instead of 2400. - Faster
is_empty(): usesfrozensetlookup for empty string variants, early returns forNone/list. SteamClient.__slots__: reduced memory per instance, faster attribute access.extract_appid()regex simplified:/app/(\d+)instead of full domain match – shorter, same correctness.make_skeleton()template: shallow-copies from frozen template dict instead of building from scratch.generate_tables.py:lines.extend(generator)instead ofappend()loop.
🔄 New Scripts & Workflows
refetch_all.py+refetch-all.yml– manual-only force re-fetch of ALL games into v2.1 schema. Creates backup → migrates → clears fetchable fields → re-fetches from Steam API + HTML scrape. Preserves manual fields (notes, safe, type_game). Requires typing"yes"to confirm. Timeout: 120 minutes.migrate_schema.py+migrate-schema.yml– upgrades existingdata.jsonlrecords: adds missing fields, normalizes types, drops deprecated fields. Idempotent.
🐛 Bug Fixes
languagescorrupted: API HTML string"Brazil<br>languages with full audio"was parsed as"Brazillanguages...". Fixed by switching to HTML table scraping.subtitlesalwaysfalse: API has no subtitle breakdown. Fixed by parsing store page language table ( Interface/Audio/Subtitles columns).has_paid_dlcfalse positives: Games with free-only DLC were markedtrue. Fixed by scraping actualdata-price-finalattributes.descriptionduplicated: Bothdescanddescriptionstored same text. Unified todescriptiononly.
🧹 Improved
- Tables: Publisher column, Platforms column, Language count, kernel anti-cheat badge (🔴/🟢).
health_checker.py:@dataclass(slots=True)for leaner objects.top_online.py: tier thresholds as tuple list (cleaner).- All scripts: consistent
sys.path.insert(0, ...)for imports.