ROADMAP Phase 2 gains a Stage 1 table — what landed, and the four bugs that were not in the plan, all of which came from running things rather than reading them. It also names the three deferrals explicitly, with the slice each belongs to, because "we'll get to it" is how the plan-progress payload stayed unimplementable for a whole milestone. CLAUDE.md's "current state" was still describing the bridge as names-only. It now says the app runs, and lists the four things a next session would otherwise discover by breaking them: the bindings staleness check, the L4 goldens as an output contract, the two tests holding the wire format, and — the one that earned its place — run it, don't just test it. Three more entries under "things that look like bugs but are not", each a decision someone would reasonably try to undo: the `serde(alias)` on snapshot::Meta (remove them and every existing snapshot stops parsing), the deliberate split between take_snapshot() and execute(), and the fixture redaction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
22 KiB
PipDock — Roadmap
Version 0.1 · 2026-07-17
Phase 0 — Spike week (before any feature code) — RUN 2026-07-27
The entire architecture stands on assumptions that must be verified on a real machine first (the FrameLedger/QuoteAtlas discipline). Each spike is a small script + captured fixtures committed to crates/pipdock-core/tests/fixtures/.
Outcomes — full write-up in spikes/README.md.
| Spike | Verdict | The one thing that changed |
|---|---|---|
| SP-1 | GO — ship both engines | Neither engine holds packages back on install -U <pkg>; both silently break installed dependents. Every plan must restate the full installed set as explicit requirements. uv's conflict attribution is better than pip's; uv writes its plan to stderr, not stdout. uv minimum pinned at 0.10.0. |
| SP-2 | Corpus captured; blocker found and fixed | pip install --dry-run --report - crashes with UnicodeEncodeError on Windows/cp1252 (pip 25 and 26). Every pip invocation must set PYTHONIOENCODING=utf-8 + PYTHONUTF8=1. Also: yanked releases exit 0 (a preview warning, not a failure), and pip/uv disagree on Requires-Python enforcement. |
| SP-3 | PASS | 858 k projects; cold refresh 2.95 s (budget 60 s); worst keystroke 42.1 ms (budget 50 ms) — but only in memory: scanning SQLite per keystroke costs 218 ms. Margin is thin on fast hardware, and raw nucleo ranking puts requests-ntlm above requests. |
| SP-4 | Answered | --path cannot be combined with --no-deps; freeze-file mode (-r <freeze> --no-deps -f json) is the only option. SECURITY §6 confirmed. Findings arrive with duplicate advisory ids. |
| SP-5 | Answered by SP-1 | No engine reports held-back items at all. Resolved version, latest version and blocker must all be derived by PipDock — for pip as much as for uv, contrary to ARCHITECTURE §3's implication. |
| SP-6 | Answered | uv python list returns downloadable entries and duplicate shims; env_hash must lowercase the interpreter path on Windows or pins and snapshots split in two. probe.py -I hides 24 user-site packages on a system Python — open decision. Microsoft Store Python still unverified. |
Both adapter-behaviour questions were decided 2026-07-27 and are implemented: PipDock enforces Requires-Python itself so the preview is engine-independent (crates/pipdock-core/src/compat.rs, ARCHITECTURE §3), and probe.py keeps -I while reporting hidden_user_site so the Installed screen can disclose the gap accurately (SECURITY §2). Two non-blocking questions remain at the end of spikes/README.md: search ranking scope, and how to obtain the five error fixtures this machine cannot produce.
| # | Spike | Question to answer | Exit criterion |
|---|---|---|---|
| SP-1 | uv dry-run shape | Exact stdout of uv pip install -U --dry-run (current + previous uv) across: clean upgrade, held-back, impossible. Is held-back attribution derivable, or does uv only show the resolved set? |
Fixture set committed; adapter parsing strategy written; uv minimum version pinned. Go/no-go for launching with both engines — if uv's plan output is too lossy, v1.0 ships pip-primary with uv behind a "beta engine" flag. |
| SP-2 | pip report & stderr corpus | Capture --dry-run --report JSON + real stderr for every ERROR-CATALOG build/network/permission case (broken sdist, MSVC-missing, PEP 668, locked file, yanked, hash mismatch). |
≥ 1 fixture per launch catalog code. |
| SP-3 | PEP 691 index economics | Full name-index size, download time on VN broadband, SQLite ingest time, fuzzy-search latency on ~600 k names. | < 60 s cold refresh, < 50 ms search on the i7-14700KF and on a low-end reference VM; else design delta/compression fallback. |
| SP-4 | pip-audit foreign-env mode | Correct invocation for auditing an env the tool isn't installed in (freeze-file --no-deps vs newer options). |
Command line + JSON fixture pinned (P1 feature, but flags decided now). |
| SP-5 | Held-back attribution accuracy | On a real numpy/scipy/pandas pin tangle: does graph-based blocker attribution match reality? | Preview sentences verified by hand against the tangle; ambiguity fallback path exercised. |
| SP-6 | Windows env discovery sweep | PEP 514 registry + py -0p + Microsoft Store Python + uv-managed on a real machine; Store-Python aliasing quirks. |
Discovery module design note updated; Store Python either supported or explicitly detected-and-explained. |
Phase 1 — M1 "Core + CLI" — substantially complete 2026-07-27
Engine trait + both adapters (per SP-1 outcome), probe.py, discovery, index cache + search, plan/resolve/execute two-phase, snapshots, pins, guard graph, error catalog, CLI over all of it. Exit: TESTING L1+L2+L4 green in CI; pipdock update --all survives the SP-5 tangle env with correct summary; dogfood on Kokone's own bot venvs begins.
What works
Every P0 CLI command except health: env list|use, list [--outdated], search, info, install, update, uninstall, pin add|remove|list, snapshot list|create|diff|rollback, doctor, pip-upgrade, engine [pip|uv], index refresh, schema, self report-bug. 173 tests; cargo fmt, clippy -D warnings and npm audit clean.
Verified against real environments, not only unit tests:
- the held-back conflict from SP-1 is reported and attributed —
httpcore 0.15.0 (latest 1.0.9)/httpx 0.23.0 requires httpcore <0.16.0,>=0.15.0— which no engine produces; update --allresolves the same tangle to latest, applies it, and leavespip checkclean;snapshot rollbackrestores an environment exactly (PRD §6's "env diff vs snapshot = ∅");- the uninstall guard refuses to break
requestsand exits non-zero, per CLI-SPEC §7; - exit codes match CLI-SPEC §5.
Findings that changed the design
plan_requirements |
Every resolve restates the installed set, or both engines break installed dependents at exit 0 (SP-1). |
-U in the dry-run argv |
Without it pip plans nothing and every package looks held back for no reason. The unit test had encoded the omission instead of the document. |
| Search ranking | Tiered, not score-ordered, and ranked by selection rather than a full sort: 90.5 ms → 16.5 ms against a 50 ms budget. |
SnapshotProof |
--no-snapshot is a named waiver, not an Option, so forgetting a snapshot cannot look like deliberately waiving one. |
Not yet done
health— belongs to M3 with the tools venv (CODE-HEALTH-SPEC).TESTING L2 in CI— done 2026-07-29. It had run nightly on 07-28 and 07-29 and failed both times; five causes were found and fixed (see L2's first runs below). Run30460248664onmainis green on all three jobs, including the fixture-drift re-capture that had never passed.- TESTING L4 —
assert_cmdis a dependency and the clap surface is covered, but the golden-output tests per command are not written. The SP-5 tangle env— done 2026-07-29. A 38-package numpy/scipy/pandas/matplotlib/scikit-learn/statsmodels environment on Python 3.12, with scipy pinned so numpy had to be held back.update --allreported 15 successful, 0 failed, 0 skipped, exited 0, and leftpip checkclean; numpy stayed at 1.26.4, the scipy pin held (invariant 5), and the snapshot was written before any mutation (invariant 2). See What the dogfood found below — the answer to SP-5's question was "yes, but the explanation was wrong".- Settings beyond engine choice — locale, thresholds and the PEP 668 override land with the GUI in M2.
L2's first runs — what actually failed (2026-07-29)
The job ran and failed three times. No cause was a product bug — all five were in the test harness, and all are fixed. Worth reading before touching capture.py or the workflow, because most of them look like the product misbehaving.
1. The rollback assertion targeted the wrong snapshot. The step did snapshot create → update --all → snapshot rollback latest → snapshot diff latest, asserting the prose matches snapshot. But latest moves twice during that flow: update --all writes a Trigger::Plan snapshot before mutating, and snapshot rollback writes a Trigger::Rollback one before restoring (DATA-FLOW §8 — rollback is itself reversible). So the final diff latest compared the restored environment against the pre-rollback state and correctly reported every package as changed. The rollback itself had worked. Fixed by pinning the id from snapshot list --json immediately after create, and asserting structurally on diff --json rather than on prose.
2. fixture-drift could never have gone green. spikes/capture.py built each scenario in a mkdtemp directory whose name carries a random suffix, and redact() replaced only the temp root — so the suffix survived into every committed sidecar, and every re-capture differed. Four further sources of churn were found underneath it: pip's [notice] To update, run: <venv>\Scripts\python.exe (which also leaked the capturing user's home directory into a public repo), CPython object addresses in urllib3 retry warnings, pip's download progress bar with its transfer rate, and uv's per-phase timings (Resolved 3 packages in 775ms). Fixed by redacting all of them in capture.py, splitting the sidecar into meta.json (contract, gated) and capture-provenance.json (versions and argv, excluded from the gate), and capturing cache-free so a warm dev machine and a cold runner agree.
3. The guard step failed on the outcome it was asserting. Fixing the rollback let the job reach the uninstall-guard step, which requires pipdock uninstall urllib3 to exit non-zero (CLI-SPEC §7). The guard did exactly that — but the runner ends a pwsh step with exit $LASTEXITCODE, and nothing had reset it. The other seven end-to-end steps were audited; each ends on a command that legitimately exits 0.
4. The redaction regex ate prose. With the drift diff finally readable, it showed Some things HTTP Core does d<PATH> — pip's --report embeds package descriptions, and does do:\n\n* Sending… reads as drive o: followed by \n path segments. A lookbehind requiring a non-alphanumeric before the drive letter is what distinguishes a path from prose.
5. report-encoding-crash cannot be re-captured on CI. The SP-2 UnicodeEncodeError needs a cp1252 console and a runner's is already UTF-8, so re-capturing there recorded an ordinary successful report and destroyed the evidence. Scenarios now carry a reproducible flag; host-dependent ones are skipped in bulk runs and stay capturable with --only. pip's report also echoes the PEP 508 environment markers, which carry the host OS — normalized, while the interpreter version markers are deliberately left alone, since those can change a resolution.
Verified two ways: locally, both engines' captures run twice back to back differ in 0 of 96 files; on CI, run 30460248664 re-captured against the newest pip and uv and matched the committed fixtures byte for byte.
Two things worth keeping in mind, both recorded in .gitattributes and CLAUDE.md: -text is what preserves the CRLF bytes the fixtures need, and -diff — which was also set — only suppressed the textual diff, so the drift job's own diagnostics could report nothing but "Binary files differ". Faults 3–5 were all found because that was fixed. And the weekly drift job's if: matched the nightly cron, so the PyPI-heavy job the comment says would be "rude" to run daily was running daily; it now has its own '17 4 * * 1' entry.
What the dogfood found (2026-07-29)
SP-5 asked whether graph-based blocker attribution matches reality on a real tangle. It does — and at scale it was also saying things that were not true. The held-back numpy came with eight constraints, four of which do not apply to Python 3.12:
numpy 1.26.4 (latest 2.5.1)
pandas 2.1.4 requires numpy <2,>=1.22.4 <- python_version < "3.11"
pandas 2.1.4 requires numpy <2,>=1.23.2 <- python_version == "3.11"
pandas 2.1.4 requires numpy <2,>=1.26.0 <- the only pandas branch in force
statsmodels 0.14.1 requires numpy <2,>=1.22.3 <- python_version == "3.10" and Windows
Requirement has always carried its marker, but only extra == was honoured, so every marker-gated branch of a dependency was reported as though it were in force. That is not just noise — telling someone on 3.12 that "pandas requires numpy >=1.22.4" is false, and ERROR-CATALOG's premise is that what PipDock says about a failure can be trusted. The same bug sat under the uninstall guard, which would refuse a removal on the strength of a dependent that only needs the package on another Python.
Fixed in graph/markers.rs: python_version and python_full_version are evaluated, with and/or/parentheses. Platform markers are left alone — they are constants for a Windows-only v1 and reading them would mean plumbing more out of probe.py for no present gain. An unrecognised marker keeps the requirement, because over-reporting a constraint is noise while dropping one hides the reason a package is stuck. The same environment now reports five constraints, one per package, each genuinely in force.
One thing deliberately not changed: blockers are still derived from the installed set, so a package being upgraded by the very same plan can still be named. Here that meant matplotlib, pandas, scikit-learn and statsmodels appeared even though the plan moves all four to numpy-2-compatible versions, leaving pinned scipy as the only real blocker. Narrowing that needs post-plan metadata the preview does not have, and it makes the sentence over-broad rather than false. Worth revisiting when PdConflictRow is built.
Where to pick up
Pushed to main and green: CI / Rust, CI / Node and CodeQL all pass on the published commit. Three configuration bugs were fixed getting there, each recorded in its own commit message: caller workflows must grant at least what the callee declares (a startup_failure with no logs), cargo audit was pinned to a toolchain too old to build itself, and its informational advisories needed scoping rather than silencing.
Done 2026-07-29, closing out M1's CI debt:
Dependabot triage— six PRs resolved. #3 (TypeScript 7) and #4 (rusqlite 0.40) closed per ARCHITECTURE §10; #1, #2, #5, #6 merged. No PRs are open.L2 green— all three jobs, both matrix legs, onmain.TESTING L4—tests/golden.rs, 46 snapshots. See What L4 covers and what it does not below.
Immediate, in rough order:
- Repo settings — branch protection, the Discord webhook secrets, and CodeQL/Discussions, per RELEASE-CI §5. None of these are committable and all are owner-only. The updater keypair that used to be on this list is gone: PipDock no longer updates itself (SECURITY §5).
M1 is otherwise closed. Everything else on this list is done, and Stage 1 of M2 — the IPC bridge — is the next piece of work.
What L4 covers and what it does not
TESTING §2 asks for goldens "against a mocked core". There is no seam to mock at — run.rs builds its engine from the global options and drives a real interpreter, and print_preview / print_summary are private functions that println! rather than return. So tests/golden.rs covers what is deterministic without an environment: the clap surface, the exit-code table, the --json error envelope, and schema <T> for all fifteen SCHEMA_TYPES. Preview and summary rendering stays with L2.
Those schema snapshots are also the review surface for M2's wire-format change. They pin two contract bugs as they stand today:
ExecutionSummaryserializesplan_id/stderr_tail, where DATA-FLOW §6 documentsplanId/stderrTailCodeserializes as its Rust variant name ("EnvInterpreterMissing"), where ERROR-CATALOG documentsPD-ENV-001
The error envelope, which main.rs hand-builds rather than deriving, already has the documented shape — so the binary currently disagrees with itself across its two --json paths. Fixing that is Stage 1 work, and the goldens make it a reviewable diff.
When the flow refactor moves rendering out of the CLI ("the flow never prints"), the preview and summary renderers become pure functions of core types and should gain their own snapshots there.
Then M2. The core carries everything the GUI needs; ui/src/ipc fixes the command names, the design tokens and EN/VI catalogs are scaffolded, and the shell renders.
Phase 2 — M2 "GUI shell + core flows" (~7–8 weeks)
Tauri app, design tokens, Environments/Installed/Updates/Search/Pins screens, preview + 3-way conflict UX, console drawer, summary sheet, snapshots UI, legal gate, EN/VI catalogs, settings. Exit: all UI-SPEC click budgets met by manual count; L3 green; VI sweep clean.
The estimate was 3–4 weeks until 2026-07-29, when surveying the code for M2 showed the bridge was scaffolding rather than substance: 26 command names with zero wrappers, one registered Tauri command, 2 of 16 Pd* components, 1 of 5 stores. Scope is unchanged; the estimate moved to match it.
Stage 1 — the IPC bridge — done 2026-07-30
Ten commits, 0ccee04..a553bf7. Everything M2's screens stand on now exists, and there is a running app.
| What landed | |
|---|---|
| Wire format | #[serde(rename)] on all 30 Code variants so they serialize as PD-*, and rename_all = "camelCase" on every IPC-crossing struct. --json had never matched DATA-FLOW §6 or ERROR-CATALOG §3 — and the two --json paths in the binary disagreed with each other. Held by a Code::ALL round-trip test and a test that walks every SCHEMA_TYPES schema rejecting any property containing _. |
| Codegen | ui/src/ipc/generated.ts from schemars via cargo run -p xtask -- bindings. Deviates from ARCHITECTURE §9's specta; reasoning and the revisit condition are in crates/pipdock-core/src/bindings.rs. Drift is a test failure, so it fires locally as well as in CI. |
core::flow |
UpdateFlow, UninstallFlow, RollbackFlow — resumable state machines, because the GUI's two decision points are IPC round trips with a render in between. plan::execute_rollback is new: core had rollback_plan() and nothing to run it, so the CLI hand-assembled an ExecutionSummary. The flow never prints. |
| Cancellation | CancellationToken through exec.rs, plus kill_on_drop(true) — the 600 s watchdog had never killed anything. A cancelled Phase A does not fall through to Phase B, and a step killed by us is Skipped, not Failed. |
| Progress | ProgressSink carries step/total; scan-progress finally has a producer and a payload (ScanProgress). |
| Vertical slice | Legal gate, Environments, Settings. core::settings, docsHash computed in build.rs, capabilities split and scoped. |
Four bugs turned up that were not in the plan, all found by running things rather than reading them: the watchdog leak; --json unparseable because snapshot … written went to stdout; blocker attribution naming constraints from other Pythons (the SP-5 dogfood); and updater:default granted to a plugin whose pubkey was a placeholder.
Stage 2 onward — where to pick up
Slices, sizing and exit checks are in the plan; the order is S2 → S3 → S4 → S5 → S6 → S7. S2 (Installed + Updates, read-only) is next and needs a virtualization library — @tanstack/react-virtual is the recommendation, and any addition must survive npm audit --audit-level=high.
Three things are deliberately deferred and should be picked up with the slice that can verify them, not before:
- The
plan-progresslifecycle enum (StepStarted/Line/StepFinished) and Tauri-side event coalescing. Belongs with the console drawer in S3, where a running UI can show whether the per-package markers actually work. Changing the payload shape amends ARCHITECTURE §7. - A Windows Job Object (
JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE) so cancellation kills the whole process tree. Today it reaches the direct child only; the measured consequence is recorded onexec::Command::build.python -m pipspawns build backends, so this matters for S3. ExecutionSummary.cancelledcopy. Nothing specifies what to tell a user whose install was interrupted, and killing pip mid-install can leave site-packages partially written. The summary should say so and point at the snapshot.
Phase 3 — M3 "Health + polish" (~2 weeks)
Tools venv, deptry/vulture/ruff runners + report UI + gated fix, pip upkeep, bug-report deep link, offline states, keyboard map, icon/branding pass. Exit: CODE-HEALTH flows pass on two real projects (one pyproject, one requirements-only).
Phase 4 — RC → v1.0 (~1–2 weeks)
Release pipeline live (bundling, checksums), manual charter executed, docs/README screenshots, legal files public, a clean install of the RC verified on a machine that has never run PipDock. Exit: RELEASE-CI §5 checklist fully checked; v1.0.0 published + notify fan-out.
Post-1.0 (P1 wave)
Security tab (pip-audit), pin auto-suggest, requirements/constraints export-import, cache manager, command palette, dependency graph view, scheduled check. Then P2 candidates by demand: macOS/Linux, per-env engine, elevation broker, winget, JP locale.
Standing risks
| Risk | Mitigation |
|---|---|
| uv output churn between releases | weekly latest-engine parser CI job (TESTING L1) + PD-ENG-003 graceful degrade + "switch to pip" escape hatch |
| pip UX flags deprecations (e.g. report format changes) | same job covers pip; adapter version-gates per engine version |
| Index size growth | SP-3 fallback design kept in the drawer |
| Scope creep from Health into a linter IDE | CODE-HEALTH §7 non-goals are contractual |