DATA-FLOW §8 has always listed "dry-run resolve the restore set" between the plan and the snapshot, and **neither head has ever done it**. S6 did not add it, so §8 now says which part is real: `RollbackFlow` diffs, plans, and reports the entries no index can supply — but it does not ask the engine whether each restore spec is still fetchable, so a release yanked since the snapshot surfaces at execute time. Adding it is a network round trip on an interaction budgeted as instant, and it would resolve against the *pre-removal* environment, reporting conflicts the two-phase execution never hits. It needs its own design and belongs with M3's engine work. Until then the preview's promise is precise rather than broad: it lists what *cannot* be restored, not everything that might fail. UI-SPEC §4 records why the env detail is a mode of the Environments tab rather than a ninth sidebar entry (§8 fixes `Ctrl+1..8`), why the trigger label is load-bearing rather than decoration, and why the diff copy names a direction — "added" is ambiguous the moment you read it from the snapshot's side. §5's rollback row gains the real path and the 2-click route from a summary. §6 goes to 14 of 16, `PdSnapshotTimeline` having arrived; `PdEnvSwitcher` is now the only one absent that is not M3's, and nothing needs it. TESTING §2's L1 list gains the partition property, which is the one a rollback preview actually stands on. ROADMAP's Stage 6 table records the four bugs, three of which no test written for them would have found: the uncancellable restore loop, the freeze lines both parsers dropped, `snapshot_list` firing twice, and the rollback that left the timeline empty. CLAUDE.md: Stages 1–6, goldens 52 → 54, and the rule S6 leaves behind — two slices that key on different things need two resets. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4.4 KiB
PipDock — Testing Strategy
Version 0.1 · 2026-07-17
1. What must never regress
- Adapter parsing of pip/uv output (the product stands on it).
- The data-flow invariants (DATA-FLOW §9): no mutation without accepted plan + snapshot.
- Conflict presentation: held-back attribution sentences must match fixture ground truth.
- Skip-and-continue semantics and summary counts.
- Reverse-dependency guard correctness — including that it is enforced, not merely reported: a removal the user never accepted must be refused (
GuardAck, PD-RES-004), and widening a removal must re-run the guard rather than proceed.
2. Layers
L1 — Core unit tests (Rust, cargo test)
- Parser fixtures: captured real outputs under
tests/fixtures/{pip,uv}/{list,outdated,report,errors}/…(seeded by spikes SP-1/SP-2, grown from every new engine release and every triaged bug).instasnapshot tests assert the normalizedResolutionReport. A CI job runs the capture script against the latest engine versions weekly and fails on unrecognized shapes → PD-ENG-003 before users hit it. - Error classifiers: every catalog code has ≥ 1 stderr fixture; a test enforces "no code without fixture."
- Graph: reverse-dep construction and blocker attribution over synthetic metadata sets, including cycles and extras (
pkg[extra]) markers. - Snapshot diff/rollback planner: property-style tests —
apply(plan(diff(a,b)), b) == aover generated freeze pairs, and the partition: every significant freeze line is either parsed into a pin or reported unrestorable, never neither. The second is the one a rollback preview stands on — a line both functions dropped disappeared silently, and the preview then promised a restore it could not perform. - Name/spec validation: PEP 508/440 accept/reject tables.
L2 — Engine integration (disposable venvs, Windows CI runner)
Each test creates a throwaway venv (python -m venv and uv venv), then exercises real engines:
| Scenario | Assertion |
|---|---|
| Fresh venv + install 3 small pure wheels | summary 3/0; check clean |
Seed known conflict (install A==old requiring B<x, then plan B latest) |
plan shows held-back with correct blocker; Compatible path succeeds; Force path warns and check reports breakage |
| Batch with 1 deliberately broken sdist | Phase B isolates it; counts n-1 ok / 1 failed with PD-BLD-*; env passes check |
| Uninstall a depended-on package | guard lists dependents; --force proceeds |
| Snapshot → mutate → rollback | env freeze equals snapshot |
| PEP 668 marker file planted | PD-ENV-002 block; override path passes flag |
Fixture packages are tiny pinned PyPI wheels (documented list in tests/integration/pins.toml) so CI is deterministic; network-touching tests are tagged and can run against a local pypiserver mirror when offline determinism is needed.
L3 — Frontend (Vitest + Testing Library)
Component tests for PdConflictRow (3-way state), PdPreviewDiff grouping, PdSummarySheet counts, dimming/badging rules on PdPackageTable, and the design-token contrast test (all text/surface pairs ≥ WCAG AA computed at build). IPC mocked at the typed wrapper layer with recorded core payloads — the same JSON as L1 fixtures, guaranteeing UI and core agree on shapes (types generated via tauri-specta make drift a compile error).
L4 — CLI (assert_cmd + insta)
Golden-output tests per command against a mocked core; TTY vs non-TTY prompt behavior; exit-code table verified exhaustively; --json payloads validated against pipdock schema output.
L5 — E2E smoke (tauri-driver, P1 gate for release)
One scripted run on a Windows runner: launch → legal gate → select seeded venv → update-all happy path → summary visible → rollback. Kept minimal; breadth lives in L2.
3. Coverage & gates
Core target ≥ 85 % line coverage (cargo llvm-cov), classifiers and plan module ≥ 95 %. PRs blocked on: fmt, clippy -D warnings, L1, L3, L4; L2 runs on PRs touching engine|plan|snapshot|graph paths and nightly otherwise. The weekly latest-engine parser job files an issue automatically on failure (feeds PD-ENG-003 early warning).
4. Manual test charter (per release)
A short exploratory pass on a real machine: dirty 150+ package env, corporate-proxy simulation (PD-NET-002 copy), non-ASCII Windows username paths, VI locale sweep of all mutation dialogs, cancel mid-Phase-B, and SmartScreen/first-run installer experience.