Files
poli0981andClaude Opus 5.5 12bf723daa ci: failures reach someone, audits run on what is merged, releases are attested
- notify-ci-failure.yml lists its workflows by name. `workflows: ['*']` is not a wildcard to
  workflow_run: it matched nothing, so `main` was red from 2026-09-23 with no one told.
- cargo audit runs a prebuilt cargo-audit (taiki-e/install-action, pinned by commit) with
  `--deny warnings`, replacing rustsec/audit-check@v2 - Node 20, a check-run write token, and on a
  pull request a failure only for "critical", which is how September's advisories passed every PR.
  .cargo/audit.toml's contract (a new informational advisory fails CI) is now what runs.
- security-scan.yml: both audits daily against `main`, so an advisory against something already
  merged is reported the day it is published, not at the next PR.
- cargo deny (licences, sources, bans) with deny.toml: the licence allowlist makes "every bundled
  licence is GPL-3.0-compatible" a test. The workspace declares `publish = false`, which it never
  had - and without which cargo-deny judged the app crates as published ones.
- CodeQL analyses rust and actions as well as javascript-typescript. The comment saying CodeQL had
  no Rust support stopped being true in 2025.
- ci-node.yml: Pages and id-token write move from the workflow to the one job that needs them
  (Sonar S8233); the audit installs with --ignore-scripts (S6505); a new `tauri lockstep` job
  fails when a Tauri crate and its npm package drift apart on Cargo.lock and package-lock.json.
- release.yml: build (read-only token, no persisted credentials) and publish (attest + draft) are
  separate jobs; the tag must match all five version fields; SHA256SUMS.txt is LF, which
  `sha256sum -c` elsewhere needs; actions/attest-build-provenance attests the .exe and .msi;
  softprops/action-gh-release is pinned by commit; `workflow_dispatch` with `dry_run` builds any
  ref without attesting or drafting.
- ci-integration.yml also runs for the CLI, exec, audit, release metadata, errors, store, index,
  Cargo.lock and rust-toolchain.toml - every one of which its end-to-end steps run through.
- dependabot.yml: cooldowns, a group per ecosystem for Tauri packages and for actions and pip,
  TypeScript held below 6.1 (typescript-eslint's peer range), and the rust-toolchain ecosystem.
  A multi-ecosystem group would put both Tauri halves in one PR, but what it does to dependencies
  outside its patterns is undocumented; the lockstep job covers the gap instead.
- Sonar: NOSONAR with reasons on the dev scripts' argv and PATH lookups; bundle-cli.mjs no longer
  claims to run from beforeBuildCommand. The dev CSP moves out of index.html into a Vite plugin
  that reads devCsp from tauri.conf.json, so a release build carries one policy, the strict one
  (S7039); checked in the dev server, where React Refresh's preamble still runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 00:55:27 +07:00

144 lines
5.1 KiB
Python

"""SP-3 — PEP 691 index economics.
`docs/ROADMAP.md` SP-3 asks: how big is the full PyPI name index, how long does it take to fetch
on VN broadband, how long to ingest into SQLite, and can fuzzy search stay under the 50 ms
per-keystroke budget over ~600 k names? If the answer is no, ARCHITECTURE §5 needs a delta or
compression fallback.
This measures the fetch and the SQLite side in Python. The fuzzy-search half is measured in Rust
with the real `nucleo` matcher (`spikes/sp3_search.rs` is not a thing yet — see the README), so
what is timed here is the SQLite scan that feeds the matcher, which is the part that would sink
the budget if the schema were wrong.
Usage:
py -3.14 spikes/sp3_index.py --out spikes/out
"""
from __future__ import annotations
import argparse
import gzip
import io
import json
import sqlite3
import time
import urllib.request
from pathlib import Path
SIMPLE_INDEX = "https://pypi.org/simple/"
PEP691_ACCEPT = "application/vnd.pypi.simple.v1+json"
def normalize(name: str) -> str:
"""PEP 503 normalization; must match `PkgName::parse` in the Rust core."""
out: list[str] = []
prev_sep = False
for ch in name:
if ch in "-_.":
if not prev_sep:
out.append("-")
prev_sep = True
else:
out.append(ch.lower())
prev_sep = False
return "".join(out)
def fetch(url: str) -> tuple[bytes, float, int]:
"""Fetch the index, returning (body, seconds, wire_bytes)."""
req = urllib.request.Request(
url, headers={"Accept": PEP691_ACCEPT, "Accept-Encoding": "gzip"}
)
start = time.perf_counter()
with urllib.request.urlopen(req, timeout=300) as resp:
raw = resp.read()
encoding = resp.headers.get("Content-Encoding", "")
elapsed = time.perf_counter() - start
wire = len(raw)
if encoding == "gzip":
raw = gzip.GzipFile(fileobj=io.BytesIO(raw)).read()
return raw, elapsed, wire
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--out", type=Path, default=Path("spikes/out"))
args = parser.parse_args()
# NOSONAR (S8707): a developer's measurement script writing where its developer says.
args.out.mkdir(parents=True, exist_ok=True) # NOSONAR
print(f"fetching {SIMPLE_INDEX} …", flush=True)
body, fetch_s, wire_bytes = fetch(SIMPLE_INDEX)
t0 = time.perf_counter()
doc = json.loads(body)
parse_s = time.perf_counter() - t0
names = [p["name"] for p in doc["projects"]]
t0 = time.perf_counter()
rows = [(n, normalize(n)) for n in names]
normalize_s = time.perf_counter() - t0
db_path = args.out / "index.db"
db_path.unlink(missing_ok=True)
conn = sqlite3.connect(db_path)
t0 = time.perf_counter()
conn.executescript(
"""
PRAGMA journal_mode = OFF;
PRAGMA synchronous = OFF;
CREATE TABLE names (name TEXT NOT NULL, normalized TEXT NOT NULL);
"""
)
conn.executemany("INSERT INTO names (name, normalized) VALUES (?, ?)", rows)
conn.execute("CREATE INDEX idx_names_normalized ON names (normalized)")
conn.commit()
ingest_s = time.perf_counter() - t0
# The search path PipDock actually needs: pull the normalized column into memory once, then
# let the fuzzy matcher work over it. Timing the scan separately shows whether SQLite is the
# bottleneck or the matcher is.
t0 = time.perf_counter()
loaded = [r[0] for r in conn.execute("SELECT normalized FROM names")]
scan_s = time.perf_counter() - t0
# A prefix query is the cheapest possible search; it bounds how fast any index-backed lookup
# can be, and shows whether the btree index is doing its job.
t0 = time.perf_counter()
for q in ("req", "num", "pand", "flask", "zope"):
conn.execute(
"SELECT name FROM names WHERE normalized GLOB ? LIMIT 50", (f"{q}*",)
).fetchall()
prefix_s = (time.perf_counter() - t0) / 5
conn.close()
db_bytes = db_path.stat().st_size
result = {
"projects": len(names),
"wire_bytes": wire_bytes,
"wire_mib": round(wire_bytes / 1024 / 1024, 2),
"decompressed_mib": round(len(body) / 1024 / 1024, 2),
"fetch_seconds": round(fetch_s, 2),
"json_parse_seconds": round(parse_s, 2),
"normalize_seconds": round(normalize_s, 2),
"sqlite_ingest_seconds": round(ingest_s, 2),
"sqlite_bytes": db_bytes,
"sqlite_mib": round(db_bytes / 1024 / 1024, 2),
"full_scan_ms": round(scan_s * 1000, 1),
"prefix_query_ms": round(prefix_s * 1000, 2),
"cold_total_seconds": round(fetch_s + parse_s + normalize_s + ingest_s, 2),
"budget_cold_refresh_seconds": 60,
"budget_search_ms": 50,
}
(args.out / "sp3.json").write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8")
print(json.dumps(result, indent=2))
verdict = "PASS" if result["cold_total_seconds"] < 60 else "FAIL"
print(f"\ncold refresh: {result['cold_total_seconds']}s vs 60s budget -> {verdict}")
return 0
if __name__ == "__main__":
raise SystemExit(main())