release(v3.5.0): charts expansion, game-count history, pipeline cleanup (#46)

- Add 6 charts (game-count line, genre treemap, KPI cards, deletions
  over time, deletion reasons, most-rated) and split charts.tsx into a
  charts/ module (one file per chart, shared chart-card + palette).
- Track catalog size over time: data_store.py appends data_game/
  count_history.json daily; backfill_count_history.py seeds it from git.
- Add scripts/json_io.py with shared load/save and a URL-keyed
  dedup_deleted so the deleted-games log can't double-log re-added games.
- Add a scroll-to-top button and a ruff config (pyproject.toml).
- Refactor docs for v3.5.0 (CHANGELOG, README badges, CLAUDE.md).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
poli0981andClaude Opus 4.7 authored and GitHub committed 2026-05-20 13:32:52 +07:00
1 parent 67ad34c3fb
commit ac8db6ab1d
49 files changed
+1255 -410

No files matched your search

+30
View File
@@ -2,6 +2,36 @@
All notable changes to this project will be documented here.
## [3.5.0] - 2026-05-20 (Charts expansion + game-count history + pipeline cleanup)
### Added
- **Six new charts on `/charts`.**
- **Game count over time** — a `LineChart` of the catalog size by day, backed by a new `data_game/count_history.json` (date-keyed `{date, total}` series). [`scripts/data_store.py`](scripts/data_store.py) `_append_count_history()` upserts today's total on every catalog write (last-write-wins per UTC day). [`scripts/backfill_count_history.py`](scripts/backfill_count_history.py) is a one-time dev tool that reconstructs the series from git history (`data_game/index.json`, plus the pre-refactor `scripts/game_info.json`). The webapp loads it via `loadCountHistory` / `useCountHistory`.
- **Genre treemap** — a Recharts `Treemap` with a custom cell renderer that hides labels on rectangles too small to fit them, so it stays legible on mobile.
- **KPI summary cards** — total games / online / NSFW / deleted / average rating, above the Overview grid.
- **Deletions over time** (bar, by month), **Deletion reasons** (pie: became-paid vs page-removed), and **Most rated games** (top 10 by `rating_count`).
- **Scroll-to-top button** — a floating bottom-right button that appears after scrolling the main pane. New [`webapp/src/components/scroll-to-top.tsx`](webapp/src/components/scroll-to-top.tsx).
- **Python linting** — `ruff` config in a new root `pyproject.toml`; run `ruff check scripts/`.
### Changed
- **`charts.tsx` split into a module.** The monolith is now a `webapp/src/components/charts/` folder — one file per chart, a shared `chart-card.tsx` (`ChartCard` + `PALETTE`), and an `index.ts` barrel. 16 charts across the 4 tabs.
- **`scripts/json_io.py`** — new shared module. `load_json` / `save_json` (previously copy-pasted into both `check_alive.py` and `check_paid.py`) live here now, alongside `dedup_deleted`.
- README + README.vi badges bumped to `3.5.0`.
### Fixed
- **The deleted-games log can no longer accumulate duplicate URLs.** `check_alive.py` and `check_paid.py` appended to `scripts/deleted_games.json` with no guard — a game removed, re-added by the scraper, then removed again would be logged twice. The new `dedup_deleted()` (keyed by URL, keeping the earliest `deleted_at`) runs on every write and inside `log_deleted.py`. Two genuinely different games that share a title but have different urls are kept separate.
### Notes
- No Tauri / Rust binary boundary change — `Cargo.toml` and `tauri.conf.json` stay at `0.1.1`, no `THIRD_PARTY` entries change (Recharts already shipped; `Treemap` / `LineChart` need no new dependency).
- `data_game/count_history.json` is auto-committed by the scrape workflows — the bash wrappers already `git add data_game/` wholesale, so no workflow changes were needed.
- The dead-code sweep found the project already tidy (TS strict mode + ESLint catch unused webapp code). `webapp/src/lib/tauri-scrape.ts` `tauriScrapePreview` was reviewed and intentionally kept as a Phase-8b stub.
---
## [3.4.1] - 2026-05-17 (Hotfix: DataTable column alignment + Discussion-announce auto-discovery)
### Fixed
+6 -5
View File
@@ -7,8 +7,9 @@ Quick context so future Claude sessions don't have to re-derive it.
Two layers stacked on the same JSON catalog of free itch.io games:
1. **Python pipeline** — the data layer.
- `data_game/game_info_NNN.json` — chunked records, max 500 per file, ~514 games as of v3.1.1.
- `scripts/` — `scraper.py` (shared session + parsing), `data_store.py` (chunk load/save/rebalance), `update_info.py`, `check_paid.py`, `check_alive.py`, `generate_md.py`, `log_deleted.py`.
- `data_game/game_info_NNN.json` — chunked records, max 500 per file, ~1,470 games as of v3.5.0.
- `data_game/index.json` — chunk manifest (total + per-file counts). `data_game/count_history.json` — date-keyed `{date,total}` series powering the "games over time" chart.
- `scripts/` — `scraper.py` (shared session + parsing), `data_store.py` (chunk load/save/rebalance, writes `index.json` + `count_history.json`), `json_io.py` (shared `load_json`/`save_json` + `dedup_deleted`), `update_info.py`, `check_paid.py`, `check_alive.py`, `check_duplicate.py`, `update_reviews.py`, `update_status.py`, `force_update.py`, `generate_md.py`, `log_deleted.py`, `backfill_count_history.py` (one-time count-history seed). Lint: `ruff check scripts/` (config in root `pyproject.toml`).
- `lists/{genre}.md` — auto-generated, **never edit by hand**.
- `bash/` — wrappers used by the GitHub Actions workflows.
2. **Webapp** — `webapp/` (React 19 + TypeScript 6 + Vite 8 + Tailwind v3 + shadcn/ui, Tauri 2 desktop wrapper in `webapp/src-tauri/`). Reads from `raw.githubusercontent.com`; writes via GitHub API with an encrypted PAT.
@@ -36,7 +37,7 @@ Path alias: `@/*` → `webapp/src/*`. Vite `define`s `__BUILD_DATE__` so About p
- `src/routes/*.tsx` — one file per route; `App.tsx` wires `react-router-dom` HashRouter.
- `src/components/data-table/*` — `DataTable` (TanStack Table + Virtual + pagination), `data-table-toolbar`, `data-table-pagination`, `faceted-filter`, `columns`. Row keys are URL-based (`getRowId: g => g.url`) so selection survives sort/filter/page changes.
- `src/components/ui/*` — shadcn primitives (Button, Card, Input, Label, Badge, Skeleton, Separator, Tabs, Popover, Checkbox, Select, Switch, Textarea, Dialog).
- `src/components/charts.tsx` — 9 Recharts charts grouped into 4 tabs.
- `src/components/charts/` — chart module (one file per chart, shared `chart-card.tsx` with `ChartCard` + `PALETTE`, `index.ts` barrel). 16 charts across 4 tabs.
- `src/lib/github/` — `client.ts` (Octokit factory), `data-store.ts` (TS port of `scripts/data_store.py`), `contents.ts` (single-file commits — routed through Git Data API so they can be signed), `git-data.ts` (atomic multi-file commit; optional signer for GPG), `signer.ts` (builds a `CommitSigner` from gpg store + auth + prefs), `workflow.ts` (dispatch + poll).
- `src/lib/gpg/` — `canonicalize.ts` (pure: builds the byte string Git would hash for a commit), `sign.ts` (lazy-imports `openpgp`, exports `loadPrivateKey` / `signCommit` / `verifyDetached`), `storage.ts` (encrypted-key + metadata in localStorage).
- `src/lib/crypto.ts` — AES-GCM 256 + PBKDF2-SHA256 100k rounds. **Uint8Array generic must be `<ArrayBuffer>`** under TS 6, not `<ArrayBufferLike>` — Web Crypto rejects the latter.
@@ -107,6 +108,6 @@ Fine-grained PAT scoped to **only this repo** with:
- Don't commit a real PAT, ever. Settings page handles it client-side; CI uses `secrets.GH_TOKEN`.
- When adding an npm dep, also add it to `webapp/src/lib/about.ts` so the About page lists it.
## Current state (as of v3.1.1)
## Current state (as of v3.5.0)
`main` is the trunk. All feature work has merged. Tags `v3.0.0`, `v3.0.1`, `v3.1.0`, `v3.1.1` exist (all GPG-signed). The webapp is published at `https://poli0981.github.io/free-games-itchio-list/app/` once the user enables Pages → Source = GitHub Actions in repo Settings.
`main` is the trunk. All feature work has merged. Tags `v3.0.0` through `v3.5.0` exist (all GPG-signed). The webapp is published at `https://poli0981.github.io/free-games-itchio-list/app/`.
+2 -2
View File
@@ -1,6 +1,6 @@
# Free Itch.io Games List
[![Version](https://img.shields.io/badge/version-3.4.1-blue.svg)](https://github.com/poli0981/free-games-itchio-list/)
[![Version](https://img.shields.io/badge/version-3.5.0-blue.svg)](https://github.com/poli0981/free-games-itchio-list/)
[![Stars](https://img.shields.io/github/stars/poli0981/free-games-itchio-list?style=social)](https://github.com/poli0981/free-games-itchio-list/stargazers)
[![Forks](https://img.shields.io/github/forks/poli0981/free-games-itchio-list?style=social)](https://github.com/poli0981/free-games-itchio-list/network/members)
[![Last Updated](https://img.shields.io/github/last-commit/poli0981/free-games-itchio-list?label=last%20updated)](https://github.com/poli0981/free-games-itchio-list/commits/main)
@@ -42,7 +42,7 @@ are added.
A React + TypeScript SPA in [`webapp/`](webapp/) provides a browsable UI on top of the same JSON
catalog: virtualized DataTable for all 500+ games, faceted filters (genre / status / platforms /
NSFW), 9 charts (Recharts), bulk edit/delete via the GitHub Git Data API, and a one-click "add"
NSFW), 16 charts (Recharts), bulk edit/delete via the GitHub Git Data API, and a one-click "add"
flow that dispatches the scraper workflow with a URL input.
- **Web build**: deployed to GitHub Pages by [`.github/workflows/deploy_webapp.yml`](.github/workflows/deploy_webapp.yml)
+2 -2
View File
@@ -1,6 +1,6 @@
# Danh sách Game Itch.io Miễn Phí
[![Version](https://img.shields.io/badge/version-3.4.1-blue.svg)](https://github.com/poli0981/free-games-itchio-list/)
[![Version](https://img.shields.io/badge/version-3.5.0-blue.svg)](https://github.com/poli0981/free-games-itchio-list/)
[![License: MIT](https://img.shields.io/badge/license-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![English](https://img.shields.io/badge/lang-English-blue.svg)](README.md)
@@ -35,7 +35,7 @@ Các bảng được tự tạo và chia theo thể loại chính (tối đa 300
## Webapp (duyệt + sửa + analytics)
Một SPA React + TypeScript trong [`webapp/`](webapp/) cung cấp giao diện duyệt trên cùng catalog JSON: DataTable virtualized cho 500+ game, faceted filter (thể loại / trạng thái / nền tảng / NSFW), 9 chart (Recharts), bulk edit/delete qua GitHub Git Data API, và một flow "add" một-click dispatch workflow scraper với input URL.
Một SPA React + TypeScript trong [`webapp/`](webapp/) cung cấp giao diện duyệt trên cùng catalog JSON: DataTable virtualized cho 500+ game, faceted filter (thể loại / trạng thái / nền tảng / NSFW), 16 chart (Recharts), bulk edit/delete qua GitHub Git Data API, và một flow "add" một-click dispatch workflow scraper với input URL.
- **Bản web**: deploy lên GitHub Pages bởi [`.github/workflows/deploy_webapp.yml`](.github/workflows/deploy_webapp.yml) — push vào `main` chạm `webapp/` sẽ tự ship. (Setup một lần: repo Settings → Pages → Source = "GitHub Actions".)
- **Bản desktop (tùy chọn)**: cùng code React đóng gói thành app native Tauri 2 cho Windows / macOS / Linux. Xem [`webapp/TAURI.md`](webapp/TAURI.md) để biết yêu cầu và `npm run tauri:dev`.
+202
View File
@@ -0,0 +1,202 @@
[
{
"date": "2025-12-27",
"total": 6
},
{
"date": "2025-12-28",
"total": 6
},
{
"date": "2026-01-01",
"total": 6
},
{
"date": "2026-01-05",
"total": 47
},
{
"date": "2026-01-11",
"total": 80
},
{
"date": "2026-01-12",
"total": 78
},
{
"date": "2026-01-16",
"total": 96
},
{
"date": "2026-01-17",
"total": 109
},
{
"date": "2026-01-18",
"total": 120
},
{
"date": "2026-01-19",
"total": 132
},
{
"date": "2026-03-11",
"total": 174
},
{
"date": "2026-03-13",
"total": 187
},
{
"date": "2026-03-14",
"total": 222
},
{
"date": "2026-03-21",
"total": 221
},
{
"date": "2026-03-24",
"total": 266
},
{
"date": "2026-03-26",
"total": 268
},
{
"date": "2026-03-27",
"total": 396
},
{
"date": "2026-03-28",
"total": 396
},
{
"date": "2026-03-29",
"total": 398
},
{
"date": "2026-03-30",
"total": 398
},
{
"date": "2026-03-31",
"total": 398
},
{
"date": "2026-04-01",
"total": 397
},
{
"date": "2026-04-03",
"total": 396
},
{
"date": "2026-04-07",
"total": 395
},
{
"date": "2026-04-09",
"total": 390
},
{
"date": "2026-04-13",
"total": 389
},
{
"date": "2026-04-16",
"total": 392
},
{
"date": "2026-04-17",
"total": 393
},
{
"date": "2026-04-19",
"total": 433
},
{
"date": "2026-04-20",
"total": 470
},
{
"date": "2026-04-21",
"total": 469
},
{
"date": "2026-04-23",
"total": 499
},
{
"date": "2026-04-24",
"total": 519
},
{
"date": "2026-04-25",
"total": 518
},
{
"date": "2026-04-27",
"total": 517
},
{
"date": "2026-04-29",
"total": 517
},
{
"date": "2026-05-01",
"total": 514
},
{
"date": "2026-05-03",
"total": 514
},
{
"date": "2026-05-04",
"total": 518
},
{
"date": "2026-05-05",
"total": 520
},
{
"date": "2026-05-07",
"total": 537
},
{
"date": "2026-05-08",
"total": 538
},
{
"date": "2026-05-09",
"total": 543
},
{
"date": "2026-05-10",
"total": 576
},
{
"date": "2026-05-11",
"total": 576
},
{
"date": "2026-05-13",
"total": 663
},
{
"date": "2026-05-15",
"total": 779
},
{
"date": "2026-05-16",
"total": 1076
},
{
"date": "2026-05-17",
"total": 1425
},
{
"date": "2026-05-19",
"total": 1467
}
]
+1
View File
@@ -32,6 +32,7 @@ checked into the repo.
```sh
pip install -r scripts/requirements.txt # if/when present, else: beautifulsoup4 requests
python scripts/update_info.py
ruff check scripts/ # lint Python (pip install ruff; config in root pyproject.toml)
```
The pipeline reads `scripts/temp_link.json` (a JSON array of itch.io URLs)
+1
View File
@@ -35,6 +35,7 @@ VS Code, Sublime, vim, v.v. đều dùng được; không có gì JetBrains-spec
```sh
pip install beautifulsoup4 requests
python scripts/update_info.py
ruff check scripts/ # lint Python (pip install ruff; cấu hình ở pyproject.toml gốc)
```
Pipeline đọc `scripts/temp_link.json` (mảng JSON các URL itch.io)
+12
View File
@@ -0,0 +1,12 @@
# Project tooling configuration.
#
# Ruff is the only tool configured here: the repo has no Python build
# system — the scraper pipeline runs the scripts/ modules directly.
# Run it with: ruff check scripts/
[tool.ruff]
line-length = 100
target-version = "py310"
[tool.ruff.lint]
select = ["E", "F", "I", "W", "UP", "B"]
+125
View File
@@ -0,0 +1,125 @@
"""
backfill_count_history.py — One-time seed for data_game/count_history.json.
Walks the git history of the game catalogue and reconstructs the total
game count per day, so the "games over time" chart has data from day one.
Two eras are covered:
- scripts/game_info.json — the pre-refactor flat array (its length).
- data_game/index.json — the current multi-file layout (total_games,
or the sum of files[].count for the oldest shape).
For each calendar day the latest commit wins. This is a developer tool:
run it once locally before tagging a release, then commit the result.
Any unreadable commit is skipped with a warning — it never aborts.
Usage (from the repo root):
python scripts/backfill_count_history.py
"""
import json
import os
import subprocess
import sys
sys.path.insert(0, os.path.dirname(__file__))
from json_io import save_json
OUTPUT_FILE = os.path.join("data_game", "count_history.json")
# (path, kind) pairs, oldest era first.
SOURCES = [
("scripts/game_info.json", "array"),
("data_game/index.json", "index"),
]
def run_git(args: list[str]) -> str:
"""Run a git command from the repo root and return stdout."""
result = subprocess.run(
["git", *args],
capture_output=True,
text=True,
encoding="utf-8",
check=True,
)
return result.stdout
def git_log_commits(path: str):
"""Yield (sha, iso_datetime) for every commit that touched `path`."""
try:
out = run_git(["log", "--format=%H%x09%cI", "--", path])
except subprocess.CalledProcessError as exc:
print(f" ! git log failed for {path}: {exc}")
return
for line in out.splitlines():
line = line.strip()
if "\t" not in line:
continue
sha, iso = line.split("\t", 1)
yield sha, iso
def extract_total(data, kind: str) -> int:
"""Compute the game count from a parsed JSON blob of the given kind."""
if kind == "array":
if not isinstance(data, list):
raise ValueError("expected a JSON array")
return len(data)
# kind == "index"
if "total_games" in data:
return int(data["total_games"])
files = data.get("files")
if files:
return sum(int(f["count"]) for f in files)
raise ValueError("index has no total_games and no files")
def commit_total(sha: str, path: str, kind: str) -> int | None:
"""Return the game count at `sha` for `path`, or None if unreadable."""
try:
blob = run_git(["show", f"{sha}:{path}"])
return extract_total(json.loads(blob), kind)
except (
subprocess.CalledProcessError,
json.JSONDecodeError,
KeyError,
TypeError,
ValueError,
) as exc:
print(f" ! skip {sha[:9]} {path}: {exc}")
return None
def main() -> None:
records: list[tuple[str, str, int]] = [] # (iso, date, total)
for path, kind in SOURCES:
commits = list(git_log_commits(path))
print(f"{path}: {len(commits)} commit(s)")
for sha, iso in commits:
total = commit_total(sha, path, kind)
if total is not None:
records.append((iso, iso[:10], total))
if not records:
print("No history found — nothing written.")
return
# Latest commit of each calendar day wins.
records.sort(key=lambda r: r[0])
by_date: dict[str, int] = {}
for _iso, date, total in records:
by_date[date] = total
history = [{"date": d, "total": by_date[d]} for d in sorted(by_date)]
save_json(OUTPUT_FILE, history)
print(
f"Wrote {len(history)} day(s) -> {OUTPUT_FILE} "
f"({history[0]['date']}: {history[0]['total']} ... "
f"{history[-1]['date']}: {history[-1]['total']})"
)
if __name__ == "__main__":
main()
+12 -26
View File
@@ -12,20 +12,20 @@ Flow:
Uses lighter delays than full scrape since we don't parse HTML.
"""
import json
import os
import sys
sys.path.insert(0, os.path.dirname(__file__))
from scraper import (
create_session,
check_url_alive,
light_delay,
batch_pause,
should_batch_pause,
now_iso,
)
from data_store import load_all_games, save_all_games
from json_io import dedup_deleted, load_json, save_json
from scraper import (
batch_pause,
check_url_alive,
create_session,
light_delay,
now_iso,
should_batch_pause,
)
DELETED_LOG = "scripts/deleted_games.json"
@@ -33,20 +33,6 @@ DELETED_LOG = "scripts/deleted_games.json"
DEAD_CODES = {404, 410} # Not Found, Gone
def load_json(path: str, default=None):
if default is None:
default = []
if os.path.exists(path):
with open(path, "r", encoding="utf-8") as f:
return json.load(f)
return default
def save_json(path: str, data) -> None:
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=4)
def main() -> None:
games: list[dict] = load_all_games()
if not games:
@@ -70,7 +56,7 @@ def main() -> None:
if status is None:
# Connection failed entirely — keep game
print(f" → Connection error, keeping game.")
print(" → Connection error, keeping game.")
keep.append(game)
errors += 1
elif status in DEAD_CODES:
@@ -96,7 +82,7 @@ def main() -> None:
# Save
save_all_games(keep)
save_json(DELETED_LOG, deleted_log)
save_json(DELETED_LOG, dedup_deleted(deleted_log))
print(
f"\nDone — {removed} removed (dead), {errors} errors, "
@@ -105,4 +91,4 @@ def main() -> None:
if __name__ == "__main__":
main()
main()
+13 -27
View File
@@ -9,38 +9,24 @@ Flow:
5. Save updated games (auto-rebalanced chunks)
"""
import json
import os
import sys
sys.path.insert(0, os.path.dirname(__file__))
from scraper import (
create_session,
check_still_free,
polite_delay,
batch_pause,
should_batch_pause,
now_iso,
)
from data_store import load_all_games, save_all_games
from json_io import dedup_deleted, load_json, save_json
from scraper import (
batch_pause,
check_still_free,
create_session,
now_iso,
polite_delay,
should_batch_pause,
)
DELETED_LOG = "scripts/deleted_games.json"
def load_json(path: str, default=None):
if default is None:
default = []
if os.path.exists(path):
with open(path, "r", encoding="utf-8") as f:
return json.load(f)
return default
def save_json(path: str, data) -> None:
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=4)
def main() -> None:
games: list[dict] = load_all_games()
if not games:
@@ -64,7 +50,7 @@ def main() -> None:
if result is None:
# Network error — don't delete, keep game
print(f" → Network error, keeping game.")
print(" → Network error, keeping game.")
keep.append(game)
errors += 1
elif result is True:
@@ -72,7 +58,7 @@ def main() -> None:
keep.append(game)
else:
# Became paid → remove
print(f" ✘ Game became paid — removing.")
print(" ✘ Game became paid — removing.")
deleted_log.append({
"url": url,
"name": name,
@@ -89,7 +75,7 @@ def main() -> None:
# Save
save_all_games(keep)
save_json(DELETED_LOG, deleted_log)
save_json(DELETED_LOG, dedup_deleted(deleted_log))
print(
f"\nDone — {removed} removed (paid), {errors} network errors, "
@@ -98,4 +84,4 @@ def main() -> None:
if __name__ == "__main__":
main()
main()
+32 -2
View File
@@ -20,6 +20,7 @@ DATA_DIR = "data_game"
FILE_PREFIX = "game_info_"
FILE_SUFFIX = ".json"
INDEX_FILE = os.path.join(DATA_DIR, "index.json")
COUNT_HISTORY_FILE = os.path.join(DATA_DIR, "count_history.json")
CHUNK_SIZE = 500
@@ -38,15 +39,16 @@ def load_all_games() -> list[dict]:
"""Load every chunk file and return a single flat list."""
games: list[dict] = []
for path in _list_chunk_files():
with open(path, "r", encoding="utf-8") as f:
with open(path, encoding="utf-8") as f:
games.extend(json.load(f))
return games
def _write_index(chunks: list[tuple[str, int]]) -> None:
"""Write data_game/index.json with metadata about all chunk files."""
total = sum(count for _, count in chunks)
index = {
"total_games": sum(count for _, count in chunks),
"total_games": total,
"max_per_file": CHUNK_SIZE,
"last_updated": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
"files": [
@@ -56,6 +58,34 @@ def _write_index(chunks: list[tuple[str, int]]) -> None:
}
with open(INDEX_FILE, "w", encoding="utf-8") as f:
json.dump(index, f, ensure_ascii=False, indent=4)
_append_count_history(total)
def _append_count_history(total: int) -> None:
"""Record today's total game count in data_game/count_history.json.
Date-keyed upsert: each UTC day holds one row. Multiple pipeline runs
on the same day overwrite that day's count (last write wins), so the
series stays one point per day. A corrupt or unreadable file falls
back to an empty history rather than aborting the save.
"""
today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
history: list[dict] = []
if os.path.exists(COUNT_HISTORY_FILE):
try:
with open(COUNT_HISTORY_FILE, encoding="utf-8") as f:
history = json.load(f)
except (json.JSONDecodeError, OSError):
history = []
for row in history:
if row.get("date") == today:
row["total"] = total
break
else:
history.append({"date": today, "total": total})
history.sort(key=lambda r: r.get("date", ""))
with open(COUNT_HISTORY_FILE, "w", encoding="utf-8") as f:
json.dump(history, f, ensure_ascii=False, indent=4)
def save_all_games(games: list[dict]) -> None:
+4 -4
View File
@@ -28,14 +28,14 @@ import os
import sys
sys.path.insert(0, os.path.dirname(__file__))
from data_store import load_all_games, save_all_games
from scraper import (
create_session,
scrape_game_info,
polite_delay,
batch_pause,
create_session,
polite_delay,
scrape_game_info,
should_batch_pause,
)
from data_store import load_all_games, save_all_games
# Fields the scraper does NOT own — never overwrite, regardless of scrape result.
PRESERVE_FIELDS = ("safe_virus", "notes", "nsfw")
+46
View File
@@ -0,0 +1,46 @@
"""
json_io.py — Shared JSON helpers for the scraper pipeline.
Small utilities used by several scripts:
- load_json / save_json: read/write JSON files (UTF-8, pretty-printed).
- dedup_deleted: de-duplicate the deleted-games log by URL.
"""
import json
import os
def load_json(path: str, default=None):
"""Load a JSON file, returning `default` (default: []) when it doesn't exist."""
if default is None:
default = []
if os.path.exists(path):
with open(path, encoding="utf-8") as f:
return json.load(f)
return default
def save_json(path: str, data) -> None:
"""Write `data` to `path` as pretty-printed UTF-8 JSON."""
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, ensure_ascii=False, indent=4)
def dedup_deleted(entries: list[dict]) -> list[dict]:
"""De-duplicate the deleted-games log by URL.
Keeps the earliest `deleted_at` per URL, so a game that is removed,
re-added by the scraper, then removed again is logged only once.
Entries sharing a name but with different urls are kept separate
(they are genuinely different games). Entries without a url are dropped.
The result is sorted ascending by `deleted_at`.
"""
by_url: dict[str, dict] = {}
for entry in entries:
url = entry.get("url", "")
if not url:
continue
current = by_url.get(url)
if current is None or entry.get("deleted_at", "") < current.get("deleted_at", ""):
by_url[url] = entry
return sorted(by_url.values(), key=lambda e: e.get("deleted_at", ""))
+9 -3
View File
@@ -6,6 +6,10 @@ Produces a human-readable log file of all games removed from the list.
import json
import os
import sys
sys.path.insert(0, os.path.dirname(__file__))
from json_io import dedup_deleted
DELETED_JSON = "scripts/deleted_games.json"
DELETED_TXT = "deleted_games.txt"
@@ -16,9 +20,11 @@ def main() -> None:
print("No deleted_games.json found — nothing to export.")
return
with open(DELETED_JSON, "r", encoding="utf-8") as f:
with open(DELETED_JSON, encoding="utf-8") as f:
entries: list[dict] = json.load(f)
entries = dedup_deleted(entries)
if not entries:
print("deleted_games.json is empty — nothing to export.")
# Still write an empty-state file
@@ -48,8 +54,8 @@ def main() -> None:
f.write(f" Reason: {reason}\n")
f.write("\n")
print(f"Exported {len(entries)} deleted game(s) → {DELETED_TXT}")
print(f"Exported {len(entries)} deleted game(s) -> {DELETED_TXT}")
if __name__ == "__main__":
main()
main()
+2 -2
View File
@@ -13,10 +13,10 @@ Provides:
import random
import time
from datetime import datetime, timezone
from html import unescape
import requests
from bs4 import BeautifulSoup
from html import unescape
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
@@ -402,4 +402,4 @@ def check_still_free(session: requests.Session, url: str) -> bool | None:
return is_free_game(soup)
except requests.RequestException as e:
print(f" [NET ERROR] {url}: {e}")
return None
return None
+6 -6
View File
@@ -14,14 +14,14 @@ import os
import sys
sys.path.insert(0, os.path.dirname(__file__))
from data_store import get_all_urls, load_all_games, save_all_games
from scraper import (
create_session,
scrape_game_info,
polite_delay,
batch_pause,
create_session,
polite_delay,
scrape_game_info,
should_batch_pause,
)
from data_store import load_all_games, save_all_games, get_all_urls
TEMP_LINK = "scripts/temp_link.json"
@@ -30,7 +30,7 @@ def main() -> None:
new_links: list[str] = []
if os.path.exists(TEMP_LINK):
with open(TEMP_LINK, "r", encoding="utf-8") as f:
with open(TEMP_LINK, encoding="utf-8") as f:
new_links = json.load(f)
# Append a single URL passed via workflow_dispatch input (webapp uses this).
@@ -72,7 +72,7 @@ def main() -> None:
continue
if not info.get("is_free", True):
print(f" → Paid game, skipping.")
print(" → Paid game, skipping.")
skipped_paid += 1
polite_delay()
continue
+4 -4
View File
@@ -16,14 +16,14 @@ import os
import sys
sys.path.insert(0, os.path.dirname(__file__))
from data_store import load_all_games, save_all_games
from scraper import (
create_session,
scrape_game_info,
polite_delay,
batch_pause,
create_session,
polite_delay,
scrape_game_info,
should_batch_pause,
)
from data_store import load_all_games, save_all_games
def main() -> None:
+4 -4
View File
@@ -15,14 +15,14 @@ import os
import sys
sys.path.insert(0, os.path.dirname(__file__))
from data_store import load_all_games, save_all_games
from scraper import (
create_session,
scrape_game_info,
polite_delay,
batch_pause,
create_session,
polite_delay,
scrape_game_info,
should_batch_pause,
)
from data_store import load_all_games, save_all_games
def main() -> None:
+1 -1
View File
@@ -31,7 +31,7 @@ show the build date.
| `/games` | [src/routes/games.tsx](src/routes/games.tsx) | Virtualized DataTable + faceted filters; bulk edit / delete |
| `/games/:slug` | [src/routes/game-detail.tsx](src/routes/game-detail.tsx) | 23-field read view + edit form for `safe_virus` / `notes` / `nsfw` |
| `/add` | [src/routes/add.tsx](src/routes/add.tsx) | Single URL or bulk paste → dispatch scraper workflow |
| `/charts` | [src/routes/charts.tsx](src/routes/charts.tsx) | 9 Recharts visualizations across 4 tabs |
| `/charts` | [src/routes/charts.tsx](src/routes/charts.tsx) | 16 charts + KPI cards across 4 tabs |
| `/workflows` | [src/routes/workflows.tsx](src/routes/workflows.tsx) | Workflow run history + manual dispatch |
| `/deleted` | [src/routes/deleted.tsx](src/routes/deleted.tsx) | Tombstone log of removed games |
| `/settings` | [src/routes/settings.tsx](src/routes/settings.tsx) | PAT lock/unlock, theme, sidebar prefs |
+5 -2
View File
@@ -1,6 +1,7 @@
import { lazy, Suspense, useEffect } from 'react'
import { lazy, Suspense, useEffect, useRef } from 'react'
import { Routes, Route, useLocation } from 'react-router-dom'
import { Sidebar, MobileTopBar } from '@/components/sidebar'
import ScrollToTop from '@/components/scroll-to-top'
import { Skeleton } from '@/components/ui/skeleton'
import { useThemeEffect } from '@/hooks/useThemeEffect'
import { useDensityEffect } from '@/hooks/useDensityEffect'
@@ -45,13 +46,14 @@ function ScrollToHash() {
export default function App() {
useThemeEffect()
useDensityEffect()
const mainRef = useRef<HTMLElement>(null)
return (
<div className="flex h-[100dvh] overflow-hidden">
<ScrollToHash />
<Sidebar />
<div className="flex min-w-0 flex-1 flex-col">
<MobileTopBar />
<main className="flex-1 overflow-y-auto">
<main ref={mainRef} className="flex-1 overflow-y-auto">
<Suspense fallback={<RouteFallback />}>
<Routes>
<Route path="/" element={<Dashboard />} />
@@ -67,6 +69,7 @@ export default function App() {
</Routes>
</Suspense>
</main>
<ScrollToTop targetRef={mainRef} />
</div>
</div>
)
-310
View File
@@ -1,310 +0,0 @@
import { useMemo } from 'react'
import {
Bar,
BarChart,
CartesianGrid,
Cell,
Legend,
Pie,
PieChart,
ResponsiveContainer,
Tooltip,
XAxis,
YAxis,
} from 'recharts'
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card'
import { Badge } from '@/components/ui/badge'
import {
classifyOnline,
countBy,
countByArray,
ratingHistogram,
topMadeWith,
topN,
type CountEntry,
} from '@/lib/analytics'
import type { DeletedGameEntry, Game } from '@/types/game'
import { formatNumber } from '@/lib/utils'
const PALETTE = [
'hsl(217 91% 60%)',
'hsl(330 81% 60%)',
'hsl(142 71% 45%)',
'hsl(38 92% 50%)',
'hsl(262 83% 58%)',
'hsl(199 89% 48%)',
'hsl(20 90% 55%)',
'hsl(173 58% 39%)',
'hsl(280 65% 60%)',
'hsl(160 60% 45%)',
]
interface ChartCardProps {
title: string
description?: string
children: React.ReactNode
}
function ChartCard({ title, description, children }: ChartCardProps) {
return (
<Card>
<CardHeader>
<CardTitle className="text-base">{title}</CardTitle>
{description && (
<p className="text-xs text-muted-foreground">{description}</p>
)}
</CardHeader>
<CardContent className="h-72">{children}</CardContent>
</Card>
)
}
export function GenreChart({ games }: { games: Game[] }) {
const data = useMemo(() => topN(countBy(games, 'genre'), 15), [games])
return (
<ChartCard title="Genres" description="Top 15 by game count">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={data} layout="vertical" margin={{ left: 20 }}>
<CartesianGrid horizontal={false} strokeDasharray="3 3" />
<XAxis type="number" />
<YAxis dataKey="key" type="category" width={120} tick={{ fontSize: 12 }} />
<Tooltip
cursor={{ fill: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Bar dataKey="count" fill={PALETTE[0]} radius={[0, 4, 4, 0]} />
</BarChart>
</ResponsiveContainer>
</ChartCard>
)
}
export function PlatformChart({ games }: { games: Game[] }) {
const data = useMemo(() => countByArray(games, 'platforms'), [games])
return (
<ChartCard title="Platforms" description="Counted per game (a game may target several)">
<ResponsiveContainer width="100%" height="100%">
<PieChart>
<Pie
data={data}
dataKey="count"
nameKey="key"
innerRadius="50%"
outerRadius="80%"
paddingAngle={2}
>
{data.map((_, i) => (
<Cell key={i} fill={PALETTE[i % PALETTE.length]} />
))}
</Pie>
<Tooltip
contentStyle={{
background: 'hsl(var(--popover))',
border: '1px solid hsl(var(--border))',
}}
/>
<Legend />
</PieChart>
</ResponsiveContainer>
</ChartCard>
)
}
export function OnlineOfflineChart({ games }: { games: Game[] }) {
const split = useMemo(() => classifyOnline(games), [games])
const data = [
{ key: 'Online (HTML5)', count: split.online },
{ key: 'Offline only', count: split.offline },
]
return (
<ChartCard title="Online vs Offline" description="HTML5 means playable in the browser">
<ResponsiveContainer width="100%" height="100%">
<PieChart>
<Pie data={data} dataKey="count" nameKey="key" innerRadius="50%" outerRadius="80%">
<Cell fill={PALETTE[2]} />
<Cell fill={PALETTE[3]} />
</Pie>
<Tooltip
contentStyle={{
background: 'hsl(var(--popover))',
border: '1px solid hsl(var(--border))',
}}
/>
<Legend />
</PieChart>
</ResponsiveContainer>
</ChartCard>
)
}
export function LanguagesChart({ games }: { games: Game[] }) {
const data = useMemo(() => topN(countByArray(games, 'languages'), 10), [games])
return (
<ChartCard title="Top 10 Languages">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={data}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="key" tick={{ fontSize: 11 }} interval={0} angle={-25} textAnchor="end" height={60} />
<YAxis />
<Tooltip
cursor={{ fill: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Bar dataKey="count" fill={PALETTE[4]} radius={[4, 4, 0, 0]} />
</BarChart>
</ResponsiveContainer>
</ChartCard>
)
}
export function MadeWithChart({ games }: { games: Game[] }) {
const data = useMemo(() => topMadeWith(games, 8), [games])
return (
<ChartCard title="Game Engines" description="Primary engine per game (made_with[0])">
<ResponsiveContainer width="100%" height="100%">
<PieChart>
<Pie data={data} dataKey="count" nameKey="key" innerRadius="50%" outerRadius="80%" paddingAngle={2}>
{data.map((_, i) => (
<Cell key={i} fill={PALETTE[i % PALETTE.length]} />
))}
</Pie>
<Tooltip
contentStyle={{
background: 'hsl(var(--popover))',
border: '1px solid hsl(var(--border))',
}}
/>
<Legend />
</PieChart>
</ResponsiveContainer>
</ChartCard>
)
}
export function RatingChart({ games }: { games: Game[] }) {
const data = useMemo(() => ratingHistogram(games, 10), [games])
return (
<ChartCard title="Rating Histogram" description="Bins of 0.5 stars from 0 to 5">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={data}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="key" tick={{ fontSize: 10 }} />
<YAxis />
<Tooltip
cursor={{ fill: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Bar dataKey="count" fill={PALETTE[5]} radius={[4, 4, 0, 0]} />
</BarChart>
</ResponsiveContainer>
</ChartCard>
)
}
export function StatusChart({ games }: { games: Game[] }) {
const data = useMemo(() => countBy(games, 'status'), [games])
return (
<ChartCard title="Status Mix" description="Released vs prototype/in-progress">
<ResponsiveContainer width="100%" height="100%">
<PieChart>
<Pie data={data} dataKey="count" nameKey="key" outerRadius="80%">
{data.map((_, i) => (
<Cell key={i} fill={PALETTE[i % PALETTE.length]} />
))}
</Pie>
<Tooltip
contentStyle={{
background: 'hsl(var(--popover))',
border: '1px solid hsl(var(--border))',
}}
/>
<Legend />
</PieChart>
</ResponsiveContainer>
</ChartCard>
)
}
export function AvgSessionChart({ games }: { games: Game[] }) {
const data = useMemo(() => topN(countBy(games, 'average_session'), 8), [games])
return (
<ChartCard title="Average Session Length">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={data}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="key" tick={{ fontSize: 10 }} interval={0} angle={-20} textAnchor="end" height={60} />
<YAxis />
<Tooltip
cursor={{ fill: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Bar dataKey="count" fill={PALETTE[6]} radius={[4, 4, 0, 0]} />
</BarChart>
</ResponsiveContainer>
</ChartCard>
)
}
export function TopTagsChart({ games }: { games: Game[] }) {
const data = useMemo(() => topN(countByArray(games, 'tags'), 30), [games])
return (
<ChartCard title="Top 30 Tags" description="Word-cloud-style ranking (sized by count)">
<div className="flex h-full flex-wrap content-start items-baseline gap-2 overflow-y-auto p-2">
{data.map((t) => {
const fontSize = computeTagFontSize(t.count, data[0]?.count ?? 1)
return (
<span
key={t.key}
style={{ fontSize: `${fontSize}px`, lineHeight: 1.1 }}
className="font-medium text-foreground/80 hover:text-foreground"
title={`${t.count} games`}
>
{t.key}
</span>
)
})}
</div>
</ChartCard>
)
}
function computeTagFontSize(count: number, max: number): number {
const min = 11
const cap = 28
const ratio = count / Math.max(1, max)
return Math.round(min + (cap - min) * ratio)
}
export function RecentDeletedChart({ deleted }: { deleted: DeletedGameEntry[] }) {
const recent = useMemo(
() =>
[...deleted]
.sort((a, b) => (b.deleted_at ?? '').localeCompare(a.deleted_at ?? ''))
.slice(0, 12),
[deleted],
)
return (
<ChartCard title="Recent Deletions" description={`Last ${recent.length} of ${formatNumber(deleted.length)} total`}>
<div className="h-full overflow-y-auto pr-2">
<ul className="space-y-1.5 text-sm">
{recent.length === 0 ? (
<p className="text-muted-foreground">No deletions logged.</p>
) : (
recent.map((d) => (
<li key={`${d.url}-${d.deleted_at}`} className="flex items-center justify-between gap-2 border-b py-1 last:border-0">
<span className="truncate font-medium">{d.name}</span>
<Badge variant="outline" className="flex-shrink-0 text-xs">
{d.reason}
</Badge>
</li>
))
)}
</ul>
</div>
</ChartCard>
)
}
export function topGenresList(entries: CountEntry[]): string {
return entries.slice(0, 3).map((e) => e.key).join(', ')
}
@@ -0,0 +1,26 @@
import { useMemo } from 'react'
import { Bar, BarChart, CartesianGrid, ResponsiveContainer, Tooltip, XAxis, YAxis } from 'recharts'
import { countBy, topN } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function AvgSessionChart({ games }: { games: Game[] }) {
const data = useMemo(() => topN(countBy(games, 'average_session'), 8), [games])
return (
<ChartCard title="Average Session Length">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={data}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="key" tick={{ fontSize: 10 }} interval={0} angle={-20} textAnchor="end" height={60} />
<YAxis />
<Tooltip
cursor={{ fill: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Bar dataKey="count" fill={PALETTE[6]} radius={[4, 4, 0, 0]} />
</BarChart>
</ResponsiveContainer>
</ChartCard>
)
}
@@ -0,0 +1,21 @@
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card'
interface ChartCardProps {
title: string
description?: string
children: React.ReactNode
}
export function ChartCard({ title, description, children }: ChartCardProps) {
return (
<Card>
<CardHeader>
<CardTitle className="text-base">{title}</CardTitle>
{description && (
<p className="text-xs text-muted-foreground">{description}</p>
)}
</CardHeader>
<CardContent className="h-72">{children}</CardContent>
</Card>
)
}
@@ -0,0 +1,30 @@
import { useMemo } from 'react'
import { Cell, Legend, Pie, PieChart, ResponsiveContainer, Tooltip } from 'recharts'
import { deletionReasonCounts } from '@/lib/analytics'
import type { DeletedGameEntry } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function DeletionReasonsChart({ deleted }: { deleted: DeletedGameEntry[] }) {
const data = useMemo(() => deletionReasonCounts(deleted), [deleted])
return (
<ChartCard title="Deletion Reasons" description="Why games left the catalog">
<ResponsiveContainer width="100%" height="100%">
<PieChart>
<Pie data={data} dataKey="count" nameKey="key" innerRadius="50%" outerRadius="80%" paddingAngle={2}>
{data.map((_, i) => (
<Cell key={i} fill={PALETTE[i % PALETTE.length]} />
))}
</Pie>
<Tooltip
contentStyle={{
background: 'hsl(var(--popover))',
border: '1px solid hsl(var(--border))',
}}
/>
<Legend />
</PieChart>
</ResponsiveContainer>
</ChartCard>
)
}
@@ -0,0 +1,26 @@
import { useMemo } from 'react'
import { Bar, BarChart, CartesianGrid, ResponsiveContainer, Tooltip, XAxis, YAxis } from 'recharts'
import { deletionsByMonth } from '@/lib/analytics'
import type { DeletedGameEntry } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function DeletionsOverTimeChart({ deleted }: { deleted: DeletedGameEntry[] }) {
const data = useMemo(() => deletionsByMonth(deleted), [deleted])
return (
<ChartCard title="Deletions Over Time" description="Games removed per month">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={data}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="key" tick={{ fontSize: 10 }} interval={0} angle={-25} textAnchor="end" height={60} />
<YAxis allowDecimals={false} />
<Tooltip
cursor={{ fill: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Bar dataKey="count" fill={PALETTE[1]} radius={[4, 4, 0, 0]} />
</BarChart>
</ResponsiveContainer>
</ChartCard>
)
}
@@ -0,0 +1,29 @@
import { CartesianGrid, Line, LineChart, ResponsiveContainer, Tooltip, XAxis, YAxis } from 'recharts'
import type { CountHistoryPoint } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function GameCountChart({ history }: { history: CountHistoryPoint[] }) {
return (
<ChartCard title="Game Count Over Time" description="Total games in the catalog by day">
{history.length === 0 ? (
<p className="flex h-full items-center justify-center text-sm text-muted-foreground">
No history data yet.
</p>
) : (
<ResponsiveContainer width="100%" height="100%">
<LineChart data={history} margin={{ left: 4, right: 8 }}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="date" tick={{ fontSize: 10 }} minTickGap={24} />
<YAxis tick={{ fontSize: 11 }} width={48} allowDecimals={false} domain={['auto', 'auto']} />
<Tooltip
cursor={{ stroke: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Line type="monotone" dataKey="total" stroke={PALETTE[0]} strokeWidth={2} dot={false} />
</LineChart>
</ResponsiveContainer>
)}
</ChartCard>
)
}
@@ -0,0 +1,26 @@
import { useMemo } from 'react'
import { Bar, BarChart, CartesianGrid, ResponsiveContainer, Tooltip, XAxis, YAxis } from 'recharts'
import { countBy, topN } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function GenreChart({ games }: { games: Game[] }) {
const data = useMemo(() => topN(countBy(games, 'genre'), 15), [games])
return (
<ChartCard title="Genres" description="Top 15 by game count">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={data} layout="vertical" margin={{ left: 20 }}>
<CartesianGrid horizontal={false} strokeDasharray="3 3" />
<XAxis type="number" />
<YAxis dataKey="key" type="category" width={120} tick={{ fontSize: 12 }} />
<Tooltip
cursor={{ fill: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Bar dataKey="count" fill={PALETTE[0]} radius={[0, 4, 4, 0]} />
</BarChart>
</ResponsiveContainer>
</ChartCard>
)
}
@@ -0,0 +1,59 @@
import { useMemo } from 'react'
import { ResponsiveContainer, Tooltip, Treemap } from 'recharts'
import { genreTreemapData } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
interface TreemapCellProps {
x?: number
y?: number
width?: number
height?: number
index?: number
depth?: number
name?: string
}
// Custom cell renderer: hides the label on rectangles too small to fit it,
// so the treemap stays legible on narrow (mobile) viewports.
function TreemapCell(props: TreemapCellProps) {
const { x = 0, y = 0, width = 0, height = 0, index = 0, depth = 1, name = '' } = props
if (depth !== 1 || width <= 0 || height <= 0) return null
return (
<g>
<rect
x={x}
y={y}
width={width}
height={height}
fill={PALETTE[index % PALETTE.length]}
stroke="hsl(var(--background))"
strokeWidth={2}
/>
{width > 56 && height > 24 && (
<text x={x + 6} y={y + 18} fill="hsl(var(--background))" fontSize={12} fontWeight={600}>
{name}
</text>
)}
</g>
)
}
export function GenreTreemapChart({ games }: { games: Game[] }) {
const data = useMemo(() => genreTreemapData(games, 12), [games])
// Recharts' Treemap data prop demands an index signature; our typed nodes
// are valid treemap data at runtime, so cast through unknown.
const treemapData = data as unknown as React.ComponentProps<typeof Treemap>['data']
return (
<ChartCard title="Genre Treemap" description="Top 12 genres sized by game count">
<ResponsiveContainer width="100%" height="100%">
<Treemap data={treemapData} dataKey="size" nameKey="name" content={<TreemapCell />}>
<Tooltip
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
</Treemap>
</ResponsiveContainer>
</ChartCard>
)
}
+16
View File
@@ -0,0 +1,16 @@
export { GenreChart } from './genre-chart'
export { PlatformChart } from './platform-chart'
export { OnlineOfflineChart } from './online-offline-chart'
export { LanguagesChart } from './languages-chart'
export { MadeWithChart } from './made-with-chart'
export { RatingChart } from './rating-chart'
export { StatusChart } from './status-chart'
export { AvgSessionChart } from './avg-session-chart'
export { TopTagsChart } from './top-tags-chart'
export { RecentDeletedChart } from './recent-deleted-chart'
export { GameCountChart } from './game-count-chart'
export { GenreTreemapChart } from './genre-treemap-chart'
export { KpiCards } from './kpi-cards'
export { DeletionsOverTimeChart } from './deletions-over-time-chart'
export { DeletionReasonsChart } from './deletion-reasons-chart'
export { TopRatedCountChart } from './top-rated-count-chart'
@@ -0,0 +1,33 @@
import { useMemo } from 'react'
import { Card, CardContent } from '@/components/ui/card'
import { computeKpis } from '@/lib/analytics'
import { formatNumber } from '@/lib/utils'
import type { DeletedGameEntry, Game } from '@/types/game'
interface KpiCardsProps {
games: Game[]
deleted: DeletedGameEntry[]
}
export function KpiCards({ games, deleted }: KpiCardsProps) {
const kpis = useMemo(() => computeKpis(games, deleted), [games, deleted])
const items = [
{ label: 'Total games', value: formatNumber(kpis.totalGames) },
{ label: 'Online (HTML5)', value: formatNumber(kpis.onlineCount) },
{ label: 'NSFW', value: formatNumber(kpis.nsfwCount) },
{ label: 'Deleted (total)', value: formatNumber(kpis.totalDeleted) },
{ label: 'Avg rating', value: kpis.avgRating.toFixed(2) },
]
return (
<div className="grid grid-cols-2 gap-4 md:grid-cols-5">
{items.map((item) => (
<Card key={item.label}>
<CardContent className="p-4">
<p className="text-xs text-muted-foreground">{item.label}</p>
<p className="mt-1 text-2xl font-bold tracking-tight">{item.value}</p>
</CardContent>
</Card>
))}
</div>
)
}
@@ -0,0 +1,26 @@
import { useMemo } from 'react'
import { Bar, BarChart, CartesianGrid, ResponsiveContainer, Tooltip, XAxis, YAxis } from 'recharts'
import { countByArray, topN } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function LanguagesChart({ games }: { games: Game[] }) {
const data = useMemo(() => topN(countByArray(games, 'languages'), 10), [games])
return (
<ChartCard title="Top 10 Languages">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={data}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="key" tick={{ fontSize: 11 }} interval={0} angle={-25} textAnchor="end" height={60} />
<YAxis />
<Tooltip
cursor={{ fill: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Bar dataKey="count" fill={PALETTE[4]} radius={[4, 4, 0, 0]} />
</BarChart>
</ResponsiveContainer>
</ChartCard>
)
}
@@ -0,0 +1,30 @@
import { useMemo } from 'react'
import { Cell, Legend, Pie, PieChart, ResponsiveContainer, Tooltip } from 'recharts'
import { topMadeWith } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function MadeWithChart({ games }: { games: Game[] }) {
const data = useMemo(() => topMadeWith(games, 8), [games])
return (
<ChartCard title="Game Engines" description="Primary engine per game (made_with[0])">
<ResponsiveContainer width="100%" height="100%">
<PieChart>
<Pie data={data} dataKey="count" nameKey="key" innerRadius="50%" outerRadius="80%" paddingAngle={2}>
{data.map((_, i) => (
<Cell key={i} fill={PALETTE[i % PALETTE.length]} />
))}
</Pie>
<Tooltip
contentStyle={{
background: 'hsl(var(--popover))',
border: '1px solid hsl(var(--border))',
}}
/>
<Legend />
</PieChart>
</ResponsiveContainer>
</ChartCard>
)
}
@@ -0,0 +1,33 @@
import { useMemo } from 'react'
import { Cell, Legend, Pie, PieChart, ResponsiveContainer, Tooltip } from 'recharts'
import { classifyOnline } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function OnlineOfflineChart({ games }: { games: Game[] }) {
const split = useMemo(() => classifyOnline(games), [games])
const data = [
{ key: 'Online (HTML5)', count: split.online },
{ key: 'Offline only', count: split.offline },
]
return (
<ChartCard title="Online vs Offline" description="HTML5 means playable in the browser">
<ResponsiveContainer width="100%" height="100%">
<PieChart>
<Pie data={data} dataKey="count" nameKey="key" innerRadius="50%" outerRadius="80%">
<Cell fill={PALETTE[2]} />
<Cell fill={PALETTE[3]} />
</Pie>
<Tooltip
contentStyle={{
background: 'hsl(var(--popover))',
border: '1px solid hsl(var(--border))',
}}
/>
<Legend />
</PieChart>
</ResponsiveContainer>
</ChartCard>
)
}
+14
View File
@@ -0,0 +1,14 @@
// Shared chart colour palette. Kept in its own file so chart-card.tsx
// stays a component-only module (React Fast Refresh friendly).
export const PALETTE = [
'hsl(217 91% 60%)',
'hsl(330 81% 60%)',
'hsl(142 71% 45%)',
'hsl(38 92% 50%)',
'hsl(262 83% 58%)',
'hsl(199 89% 48%)',
'hsl(20 90% 55%)',
'hsl(173 58% 39%)',
'hsl(280 65% 60%)',
'hsl(160 60% 45%)',
]
@@ -0,0 +1,37 @@
import { useMemo } from 'react'
import { Cell, Legend, Pie, PieChart, ResponsiveContainer, Tooltip } from 'recharts'
import { countByArray } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function PlatformChart({ games }: { games: Game[] }) {
const data = useMemo(() => countByArray(games, 'platforms'), [games])
return (
<ChartCard title="Platforms" description="Counted per game (a game may target several)">
<ResponsiveContainer width="100%" height="100%">
<PieChart>
<Pie
data={data}
dataKey="count"
nameKey="key"
innerRadius="50%"
outerRadius="80%"
paddingAngle={2}
>
{data.map((_, i) => (
<Cell key={i} fill={PALETTE[i % PALETTE.length]} />
))}
</Pie>
<Tooltip
contentStyle={{
background: 'hsl(var(--popover))',
border: '1px solid hsl(var(--border))',
}}
/>
<Legend />
</PieChart>
</ResponsiveContainer>
</ChartCard>
)
}
@@ -0,0 +1,26 @@
import { useMemo } from 'react'
import { Bar, BarChart, CartesianGrid, ResponsiveContainer, Tooltip, XAxis, YAxis } from 'recharts'
import { ratingHistogram } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function RatingChart({ games }: { games: Game[] }) {
const data = useMemo(() => ratingHistogram(games, 10), [games])
return (
<ChartCard title="Rating Histogram" description="Bins of 0.5 stars from 0 to 5">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={data}>
<CartesianGrid strokeDasharray="3 3" />
<XAxis dataKey="key" tick={{ fontSize: 10 }} />
<YAxis />
<Tooltip
cursor={{ fill: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Bar dataKey="count" fill={PALETTE[5]} radius={[4, 4, 0, 0]} />
</BarChart>
</ResponsiveContainer>
</ChartCard>
)
}
@@ -0,0 +1,41 @@
import { useMemo } from 'react'
import { Badge } from '@/components/ui/badge'
import { formatNumber } from '@/lib/utils'
import type { DeletedGameEntry } from '@/types/game'
import { ChartCard } from './chart-card'
export function RecentDeletedChart({ deleted }: { deleted: DeletedGameEntry[] }) {
const recent = useMemo(
() =>
[...deleted]
.sort((a, b) => (b.deleted_at ?? '').localeCompare(a.deleted_at ?? ''))
.slice(0, 12),
[deleted],
)
return (
<ChartCard
title="Recent Deletions"
description={`Last ${recent.length} of ${formatNumber(deleted.length)} total`}
>
<div className="h-full overflow-y-auto pr-2">
<ul className="space-y-1.5 text-sm">
{recent.length === 0 ? (
<p className="text-muted-foreground">No deletions logged.</p>
) : (
recent.map((d) => (
<li
key={`${d.url}-${d.deleted_at}`}
className="flex items-center justify-between gap-2 border-b py-1 last:border-0"
>
<span className="truncate font-medium">{d.name}</span>
<Badge variant="outline" className="flex-shrink-0 text-xs">
{d.reason}
</Badge>
</li>
))
)}
</ul>
</div>
</ChartCard>
)
}
@@ -0,0 +1,30 @@
import { useMemo } from 'react'
import { Cell, Legend, Pie, PieChart, ResponsiveContainer, Tooltip } from 'recharts'
import { countBy } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function StatusChart({ games }: { games: Game[] }) {
const data = useMemo(() => countBy(games, 'status'), [games])
return (
<ChartCard title="Status Mix" description="Released vs prototype/in-progress">
<ResponsiveContainer width="100%" height="100%">
<PieChart>
<Pie data={data} dataKey="count" nameKey="key" outerRadius="80%">
{data.map((_, i) => (
<Cell key={i} fill={PALETTE[i % PALETTE.length]} />
))}
</Pie>
<Tooltip
contentStyle={{
background: 'hsl(var(--popover))',
border: '1px solid hsl(var(--border))',
}}
/>
<Legend />
</PieChart>
</ResponsiveContainer>
</ChartCard>
)
}
@@ -0,0 +1,32 @@
import { useMemo } from 'react'
import { Bar, BarChart, CartesianGrid, ResponsiveContainer, Tooltip, XAxis, YAxis } from 'recharts'
import { topByRatingCount } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
import { PALETTE } from './palette'
export function TopRatedCountChart({ games }: { games: Game[] }) {
const data = useMemo(() => topByRatingCount(games, 10), [games])
return (
<ChartCard title="Most Rated Games" description="Top 10 by number of ratings">
<ResponsiveContainer width="100%" height="100%">
<BarChart data={data} layout="vertical" margin={{ left: 20 }}>
<CartesianGrid horizontal={false} strokeDasharray="3 3" />
<XAxis type="number" allowDecimals={false} />
<YAxis
dataKey="key"
type="category"
width={140}
tick={{ fontSize: 11 }}
tickFormatter={(v: string) => (v.length > 20 ? `${v.slice(0, 19)}…` : v)}
/>
<Tooltip
cursor={{ fill: 'hsl(var(--accent))' }}
contentStyle={{ background: 'hsl(var(--popover))', border: '1px solid hsl(var(--border))' }}
/>
<Bar dataKey="count" fill={PALETTE[3]} radius={[0, 4, 4, 0]} />
</BarChart>
</ResponsiveContainer>
</ChartCard>
)
}
@@ -0,0 +1,34 @@
import { useMemo } from 'react'
import { countByArray, topN } from '@/lib/analytics'
import type { Game } from '@/types/game'
import { ChartCard } from './chart-card'
export function TopTagsChart({ games }: { games: Game[] }) {
const data = useMemo(() => topN(countByArray(games, 'tags'), 30), [games])
return (
<ChartCard title="Top 30 Tags" description="Word-cloud-style ranking (sized by count)">
<div className="flex h-full flex-wrap content-start items-baseline gap-2 overflow-y-auto p-2">
{data.map((t) => {
const fontSize = computeTagFontSize(t.count, data[0]?.count ?? 1)
return (
<span
key={t.key}
style={{ fontSize: `${fontSize}px`, lineHeight: 1.1 }}
className="font-medium text-foreground/80 hover:text-foreground"
title={`${t.count} games`}
>
{t.key}
</span>
)
})}
</div>
</ChartCard>
)
}
function computeTagFontSize(count: number, max: number): number {
const min = 11
const cap = 28
const ratio = count / Math.max(1, max)
return Math.round(min + (cap - min) * ratio)
}
+34
View File
@@ -0,0 +1,34 @@
import { useEffect, useState } from 'react'
import { ArrowUp } from 'lucide-react'
import { Button } from '@/components/ui/button'
interface ScrollToTopProps {
// The scroll container to watch — the app scrolls <main>, not window.
targetRef: React.RefObject<HTMLElement | null>
}
export default function ScrollToTop({ targetRef }: ScrollToTopProps) {
const [visible, setVisible] = useState(false)
useEffect(() => {
const el = targetRef.current
if (!el) return
const onScroll = () => setVisible(el.scrollTop > 400)
onScroll()
el.addEventListener('scroll', onScroll, { passive: true })
return () => el.removeEventListener('scroll', onScroll)
}, [targetRef])
if (!visible) return null
return (
<Button
size="icon"
aria-label="Scroll to top"
onClick={() => targetRef.current?.scrollTo({ top: 0, behavior: 'smooth' })}
className="fixed bottom-6 right-6 z-50 h-11 w-11 rounded-full shadow-lg"
>
<ArrowUp className="h-5 w-5" />
</Button>
)
}
+13 -1
View File
@@ -1,5 +1,10 @@
import { useQuery } from '@tanstack/react-query'
import { loadAllGames, loadIndex, loadDeletedLog } from '@/lib/github/data-store'
import {
loadAllGames,
loadIndex,
loadDeletedLog,
loadCountHistory,
} from '@/lib/github/data-store'
export function useIndex() {
return useQuery({
@@ -21,3 +26,10 @@ export function useDeletedGames() {
queryFn: loadDeletedLog,
})
}
export function useCountHistory() {
return useQuery({
queryKey: ['count-history'],
queryFn: loadCountHistory,
})
}
+1 -1
View File
@@ -8,7 +8,7 @@ export interface ThirdParty {
export const APP = {
name: 'Itch.io Free Games DB',
version: '3.4.1',
version: '3.5.0',
repo: 'https://github.com/poli0981/free-games-itchio-list',
license: 'MIT',
buildDate: typeof __BUILD_DATE__ !== 'undefined' ? __BUILD_DATE__ : 'dev',
+80 -2
View File
@@ -1,4 +1,4 @@
import type { Game } from '@/types/game'
import type { Game, DeletedGameEntry } from '@/types/game'
export interface CountEntry {
key: string
@@ -64,7 +64,7 @@ export function computeOverview(games: Game[]): OverviewStats {
}
}
export function parseRating(rating: string): number | null {
function parseRating(rating: string): number | null {
const n = parseFloat(rating)
return isFinite(n) ? n : null
}
@@ -107,3 +107,81 @@ export function ratingHistogram(games: Game[], bins = 10): CountEntry[] {
}
return out
}
export interface TreemapNode {
name: string
size: number
}
export function genreTreemapData(games: Game[], topN = 12): TreemapNode[] {
return countBy(games, 'genre')
.slice(0, topN)
.map((e) => ({ name: e.key, size: e.count }))
}
export interface KpiStats {
totalGames: number
onlineCount: number
nsfwCount: number
totalDeleted: number
avgRating: number
}
export function computeKpis(games: Game[], deleted: DeletedGameEntry[]): KpiStats {
const overview = computeOverview(games)
let ratingSum = 0
let ratingCount = 0
for (const g of games) {
const r = parseRating(g.rating)
if (r !== null) {
ratingSum += r
ratingCount += 1
}
}
return {
totalGames: overview.total,
onlineCount: overview.htmlCount,
nsfwCount: overview.nsfwCount,
totalDeleted: deleted.length,
avgRating: ratingCount ? ratingSum / ratingCount : 0,
}
}
export function deletionsByMonth(deleted: DeletedGameEntry[]): CountEntry[] {
const m = new Map<string, number>()
for (const d of deleted) {
const month = (d.deleted_at ?? '').slice(0, 7)
if (month.length !== 7) continue
m.set(month, (m.get(month) ?? 0) + 1)
}
return Array.from(m, ([key, count]) => ({ key, count })).sort((a, b) => a.key.localeCompare(b.key))
}
export function deletionReasonCounts(deleted: DeletedGameEntry[]): CountEntry[] {
let paid = 0
let removed = 0
let other = 0
for (const d of deleted) {
const reason = (d.reason ?? '').toLowerCase()
if (reason.includes('paid')) {
paid += 1
} else if (reason.includes('no longer exists')) {
removed += 1
} else {
other += 1
}
}
const out: CountEntry[] = []
if (paid) out.push({ key: 'Became paid', count: paid })
if (removed) out.push({ key: 'Page removed', count: removed })
if (other) out.push({ key: 'Other', count: other })
return out
}
export function topByRatingCount(games: Game[], n = 10): CountEntry[] {
return games
.map((g) => ({ key: g.name, count: Number(g.rating_count) || 0 }))
.filter((e) => e.count > 0)
.sort((a, b) => b.count - a.count)
.slice(0, n)
}
+1
View File
@@ -9,6 +9,7 @@ export const RAW_BASE = `https://raw.githubusercontent.com/${REPO.owner}/${REPO.
export const PATHS = {
index: 'data_game/index.json',
chunk: (filename: string) => `data_game/${filename}`,
countHistory: 'data_game/count_history.json',
deletedJson: 'scripts/deleted_games.json',
tempLink: 'scripts/temp_link.json',
} as const
+15 -1
View File
@@ -1,4 +1,10 @@
import type { Game, GameDbIndex, ChunkPlan, DeletedGameEntry } from '@/types/game'
import type {
Game,
GameDbIndex,
ChunkPlan,
DeletedGameEntry,
CountHistoryPoint,
} from '@/types/game'
import { MAX_PER_FILE, PATHS } from '../config'
import { fetchRaw } from './raw'
@@ -25,6 +31,14 @@ export async function loadDeletedLog(): Promise<DeletedGameEntry[]> {
}
}
export async function loadCountHistory(): Promise<CountHistoryPoint[]> {
try {
return await fetchRaw<CountHistoryPoint[]>(PATHS.countHistory)
} catch {
return []
}
}
export function rebalance(games: Game[]): ChunkPlan[] {
const numChunks = Math.max(1, Math.ceil(games.length / MAX_PER_FILE))
return Array.from({ length: numChunks }, (_, i) => ({
+18 -5
View File
@@ -2,7 +2,12 @@ import { Skeleton } from '@/components/ui/skeleton'
import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs'
import {
AvgSessionChart,
DeletionReasonsChart,
DeletionsOverTimeChart,
GameCountChart,
GenreChart,
GenreTreemapChart,
KpiCards,
LanguagesChart,
MadeWithChart,
OnlineOfflineChart,
@@ -10,15 +15,17 @@ import {
RatingChart,
RecentDeletedChart,
StatusChart,
TopRatedCountChart,
TopTagsChart,
} from '@/components/charts'
import { useAllGames, useDeletedGames } from '@/hooks/useGames'
import { useAllGames, useCountHistory, useDeletedGames } from '@/hooks/useGames'
import { useDocumentTitle } from '@/hooks/useDocumentTitle'
export default function Charts() {
useDocumentTitle('Charts')
const games = useAllGames()
const deleted = useDeletedGames()
const history = useCountHistory()
if (games.isLoading) {
return (
@@ -44,6 +51,7 @@ export default function Charts() {
const data = games.data?.games ?? []
const deletedList = deleted.data ?? []
const historyList = history.data ?? []
return (
<div className="container mx-auto p-6">
@@ -57,12 +65,14 @@ export default function Charts() {
<TabsTrigger value="discovery">Discovery</TabsTrigger>
</TabsList>
<TabsContent value="overview">
<TabsContent value="overview" className="space-y-4">
<KpiCards games={data} deleted={deletedList} />
<div className="grid gap-4 md:grid-cols-2">
<GenreChart games={data} />
<StatusChart games={data} />
<OnlineOfflineChart games={data} />
<MadeWithChart games={data} />
<GameCountChart history={historyList} />
</div>
</TabsContent>
@@ -76,15 +86,18 @@ export default function Charts() {
</TabsContent>
<TabsContent value="quality">
<div className="grid gap-4 md:grid-cols-1">
<div className="grid gap-4 md:grid-cols-2">
<RatingChart games={data} />
<RecentDeletedChart deleted={deletedList} />
<TopRatedCountChart games={data} />
<DeletionsOverTimeChart deleted={deletedList} />
<DeletionReasonsChart deleted={deletedList} />
</div>
</TabsContent>
<TabsContent value="discovery">
<div className="grid gap-4">
<div className="grid gap-4 md:grid-cols-2">
<TopTagsChart games={data} />
<GenreTreemapChart games={data} />
</div>
</TabsContent>
</Tabs>
+5
View File
@@ -38,6 +38,11 @@ export interface DeletedGameEntry {
deleted_at: string
}
export interface CountHistoryPoint {
date: string
total: number
}
export interface ChunkPlan {
name: string
games: Game[]