Files
pipdock/docs/TESTING.md
T
poli0981andClaude Opus 5.5 2f6ad04967 fix: the 16 Rust findings and the coverage gap from the first CI analysis of main (#91)
The first CI analysis of main failed its gate: 16 Rust bugs and new-code
coverage 78.3 % against 80 %, over all of 1.4.0 (sending a version did not
move the previous-version baseline).

- rust:S6677 x3: a failure's message is logged as `error`, not as a second
  `message` field (CLI, update flow, app startup); golden.rs reads the
  CLI's log to hold it, and failed first.
- rust:S7493 x13: std::fs inside async fns goes through tokio::fs (tools
  venv sync, deptry's report, export/requirements/report commands).
- Tests for 1.4.0 code nothing drove: rollback end to end, forcing a
  latest this Python cannot run, PEP 691 parsing (L55), user_file_error
  (U25), a plan verified in another environment, PyPiReleases' cache, and
  the plan/env stores' success and late-error paths. InstallSet::only had
  no caller and is removed.
- ROADMAP, TESTING and CLAUDE.md record what the analysis found.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 07:21:45 +07:00

17 KiB
Raw Blame History

PipDock — Testing Strategy

Version 0.1 · 2026-07-17

1. What must never regress

  1. Adapter parsing of pip/uv output (the product stands on it).
  2. The data-flow invariants (DATA-FLOW §9): no mutation without accepted plan + snapshot.
  3. Conflict presentation: held-back attribution sentences must match fixture ground truth.
  4. Skip-and-continue semantics and summary counts.
  5. Reverse-dependency guard correctness — including that it is enforced, not merely reported: a removal the user never accepted must be refused (GuardAck, PD-RES-004), and widening a removal must re-run the guard rather than proceed.

A fix lands with a test that failed without it. 1.4.0 held every Critical and High audit finding to that rule: each fix's test was run against the code it replaced and seen to fail first (ROADMAP's 1.4.0 record). A test that has never failed has not shown that it can.

2. Layers

L1 — Core unit and corpus tests (Rust, cargo test)

  • Engine captures: real pip and uv output under crates/pipdock-core/tests/fixtures/, one directory per scenario holding stdout.txt, stderr.txt, meta.json (the exit code, and the catalog code the scenario must produce) and capture-provenance.json (engine, interpreter and argv), written by spikes/capture.py and seeded by spikes SP-1/SP-2. Ten scenarios per engine — a clean upgrade, held back, held back by a constraint, impossible, not found, yanked, Requires-Python, a build-backend failure, an unreachable network, the cp1252 report crash — in three corpora, plus one of synthetic stand-ins:

    • pip/ and uv/ — the current engines (pip 26.2.1 and uv 0.13.1 at 1.4.0), and the bytes the weekly drift job re-captures and compares (L2).
    • uv-0.11/ — frozen: a byte-identical copy of uv 0.11.32's captures, the last miette-style rendering (×, ╰─▶, "Caused by:"). uv 0.13.0 rewrote every failure as error: / cause:, and UV_MIN_VERSION is still 0.10, so users run both. both_uv_renderings_classify_identically (classifier_corpus.rs) holds every scenario to one catalog code under either rendering, and uv_impossibility_reads_the_same_in_both_renderings (plan_parsers.rs) holds the explanation to the same words.
    • synthetic/ — the eight conditions a capture machine cannot produce (a full disk, a locked file, a hash mismatch…), each flagged, saying why it is synthetic and what would replace it; a test refuses one that does not.

    The captures are redacted rather than literal — absolute paths, addresses, progress bars, timings, and data that moves with PyPI rather than with an engine (pip's "from versions" list, its release notice) — so they compare across machines and weeks. Line endings stay byte-exact (.gitattributes' -text), and a test fails if no fixture keeps a CRLF.

  • Parsers against the corpora (plan_parsers.rs): every capture is replayed through the function each resolve returns through — pip_resolution / uv_resolution — and asserted on meaning: a plan is recognised, an upgrade knows what it is upgrading from, an impossibility is a report. Deliberately not snapshotted, because a snapshot would fail on every cosmetic engine change and teach re-blessing without reading; this bullet said insta snapshots asserted the normalized ResolutionReport, and none does. every_failed_capture_reaches_the_user_as_the_code_it_was_captured_for is 1.4.0's: the classifier corpus had proved each stderr classifies correctly while uv's adapter never asked the classifier, so a failed build previewed as "no changes".

  • Error classifiers (classifier_corpus.rs): every catalog code is covered by a fixture that names it or classifies to it, or is listed as raised by PipDock itself, with the module that raises it — a watchdog (PD-ENG-004) or a stale preview (PD-RES-002) has no stderr to capture. A code with neither fails CI: "no code without a fixture". This line said every code has a stderr fixture; the table of codes PipDock raises itself has sat beside the captures since the gate was written.

  • Flows, end to end against a scripted engine (tests/flows.rs, 1.4.0). The engine owns a small mutable world — what is installed, what the index offers, whether the interpreter is externally managed — and installs really change it, so a test can move the environment between two calls the way another terminal would; Engine::probe and FlowOptions (the cancellation token, a clock) are the seams. It exists because everything a flow decides before plan::execute — the PEP 668 gate, the drift basis, what reaches the resolver, what a forced package releases — sat behind a free-function probe, so a stale-preview check that could never fire, Force latest forcing nothing and a cross-engine rollback removing pip were found by no test. A flow decision without a test here is a regression. execution.rs does the same for plan::execute: skip-and-continue (thirteen of fifteen apply), the counts against the rows, Phase B isolation, and a stop, a watchdog or an engine that will not start reported per row rather than raised.

  • Graph: reverse-dep construction and blocker attribution over synthetic metadata sets, including cycles and extras (pkg[extra]) markers — and, since probe schema 3, platform markers, so ipython's pexpect; sys_platform != "win32" no longer reads as a missing dependency on Windows.

  • Snapshot diff/rollback planner: the round trip apply(plan(diff(a,b)), b) == a over a constructed pair, and, over 400 generated freeze texts, the partition: every significant freeze line is either parsed into a pin or reported unrestorable, never neither. The second is the one a rollback preview stands on — a line both functions dropped disappeared silently, and the preview then promised a restore it could not perform. (This bullet said the round trip ran over generated pairs; only the partition does.)

  • Name/spec validation: PEP 508/440 accept/reject tables — names in model.rs, and since 1.4.0 versions and specifier sets in pep440.rs, the boundary where text becomes argv (SECURITY §2).

  • What the frontend and the shell depend on fails cargo test too: generated.ts and the L3 IPC fixtures when stale (bindings.rs, fixtures.rs); the webview's exact grants and the opener's four hosts (src-tauri/src/lib.rs, SECURITY §4) — a grant is a contract no frontend test can see, and Decline on the legal gate did nothing in the installed app while its component test, which mocks close(), passed; and the legal-documents hash (crates/pipdock-core/src/legal_hash.rs), which build.rs includes by #[path] and the library compiles only under cfg(test), so the tests run the function that runs at build time. A CRLF document hashes like its LF twin — which failed against the code that took the length before normalising — a changed or moved document changes the hash, and the hand-rolled SHA-256 agrees with sha2 across block boundaries.

  • Six tests are #[ignore]d, each needing what cargo test must not: real PyPI (the tools-venv bootstrap and its repair), a synced audit venv, a real interpreter (the graph), and a populated index in a release build (search_latency.rs, which refuses to run in debug). They run by hand with --ignored; no CI job runs them (ROADMAP's Audit backlog). L2 drives the bootstrap through the CLI instead.

L2 — Engine integration (disposable venvs, Windows CI runner)

ci-integration.yml's Real engines, throwaway venvs job, on Python 3.10 and 3.14 — the floor and the newest PipDock claims. It runs cargo test, builds the CLI and drives it against venvs it creates with python -m venv, real pip and real uv on PATH. (This line said uv venv as well; nothing creates one.)

Step Assertion
Discovery and inspection env list, doctor and list --json succeed on a fresh venv
The Code Health tools venv tools sync reaches fresh with all three tools and without pip-audit; a warm re-sync short-circuits (under 10 s); a deleted ruff.exe reads toolMissing and a re-sync repairs it; offline, PD-NET-011, exit 6, and no manifest left behind
pip upkeep a venv seeded with pip 22.1.2 is upgraded by pip-upgrade and its pip runs afterwards — on 3.14 only through the ensurepip fallback
PEP 668 with the marker planted in the base interpreter, a venv built from it is not externally managed and the base is
The SP-1 conflict over httpx==0.23.0, update httpcore --dry-run reports httpcore held back and attributes it to httpx
Update → snapshot → rollback after update --all and snapshot rollback <id>, snapshot diff <id> --json reports no drift
The uninstall guard removing urllib3 from under requests exits non-zero
Exit codes a missing environment exits 3; doctor on a healthy one exits 0
JSON contracts pipdock schema for four types, and list --json, parse
An editable install listed once, and with no size

What this table used to claim and L2 does not do: install three wheels and check the summary; take a conflict down the Force path and see check report the breakage; isolate a broken sdist in Phase B; --force past the guard; and refuse with PD-ENV-002, then pass the flag under the override, against a real engine. Those are proved against scripted engines in L1 — the batch semantics in execution.rs, Force latest and the PEP 668 gate and override in flows.rs, the guard's acknowledgement in flow's unit tests, the flag itself in the adapters' argv tests — and 1.4.0 ran Force latest and the override by hand against pip 26.2.1 and uv 0.13.1 (ROADMAP).

The packages are real PyPI packages, and only some are pinned — httpx==0.23.0, pip==22.1.2, the shipped tool pins — while requests and update --all take whatever is newest that day. This paragraph described a pinned package list in tests/integration/pins.toml and a local pypiserver mirror for offline runs; neither has ever existed. L2 is network-bound, and only as deterministic as PyPI.

The drift job (Re-capture against the latest engines, Mondays and on dispatch) re-captures pip/ and uv/ with the newest pip and uv on Python 3.12.10 — an exact patch, because pip's report and uv's output both record it — uploads what it captured whether or not it passes (captured-fixtures: the fix for a real drift is to review and commit exactly those bytes), fails if any fixture but capture-provenance.json moved, and runs cargo test over the new captures. It failed seven Mondays running before 1.4.0, mostly for reasons that were not engines — idna's newest release and scipy's growing version list, now pinned and redacted in capture.py — and it was blind where it mattered: the capture venv never upgraded pip, so the job meant to see a pip change ran the CPython-bundled 25.0.1 throughout. The one real change, uv 0.13's new rendering, was answered by freezing the old captures as uv-0.11/.

L3 — Frontend (Vitest + Testing Library)

Component tests for PdConflictRow (3-way state), PdPreviewDiff grouping, PdSummarySheet counts, dimming/badging rules on PdPackageTable, and the design-token contrast test — every text/surface pair ≥ WCAG AA, and since 1.4.0 every badge tint composited over every surface too; it is a test, so it gates every PR. Store tests include ui/src/stores/races.test.ts (1.4.0): each holds a command open, moves on — another environment, another timeline entry, Back out of a plan — and only then lets it answer, because five stores wrote whatever came back without asking whether the question was still the one being asked. Six of its seven tests fail on the stores before 1.4.0. i18n.test.ts fails on a catalog code that either language has no copy for.

IPC is mocked at the typed wrapper layer (ui/src/test/ipc.ts, a factory each test file passes to its own vi.mock) with payloads serialized from the real Rust types by cargo run -p xtask -- ipc-fixtures into ui/src/test/fixtures/, and cargo test fails when the committed files are stale — which is what guarantees UI and core agree on shapes. Where production computes a value, the fixture is computed by the same function: the guard report and the dependency graph by the real graph over the scenario's rows, the preview's blockers by blockers_for, and since 1.4.0 flow_step_forced.json by plan::derive_forced and release::mismatch. A hand-written blocker list that matched the docs while the code had drifted is how the preview shipped a garbled sentence from S3 to 1.1.0 with every suite green. fixtures.rs is compiled only under cfg(test) or the fixtures feature xtask enables, never into a release build. Types are generated by cargo run -p xtask -- bindings over schemars, not by tauri-specta as this line once said, so drift is a test failure rather than a compile error — which means it fires at cargo test, locally as well as in CI. See crates/pipdock-core/src/bindings.rs.

L4 — CLI (assert_cmd + insta)

Goldens over the real binary. There is no mocked core — this line promised one, and the CLI has no seam to mock at (golden.rs says why) — so what is goldened is everything deterministic without an interpreter: every command's and subcommand's help, which is the flag contract scripts read; the JSON and human error envelope; and pipdock schema <T> for every type in SCHEMA_TYPES, plus the unknown-type list — 68 snapshots at 1.4.0. The exit-code table is asserted for the rows reachable without an interpreter (0, 2 and 3); L2 asserts the ones it can reach. A golden diff is a behaviour change: re-bless deliberately, never reflexively.

Not covered by a committed test, though this line claimed both: TTY versus non-TTY prompting, and --json payloads checked against pipdock schema's output. CLI-SPEC §6's one-document --json rule was verified for 1.4.0 by driving the real binary through every path (ROADMAP), not by a test that stays.

L5 — E2E smoke — not built

The plan was one scripted tauri-driver run on a Windows runner — launch → legal gate → select seeded venv → update-all happy path → summary visible → rollback — as a P1 gate for release. Nothing implements it: no tauri-driver, no WebDriver, no job. What stands in is a person building, installing and driving each release's installer by hand (§4, and ROADMAP Phase 4's "building and installing are their own test tier"). An installer smoke test is in ROADMAP's Audit backlog.

3. Coverage & gates

Coverage is measured since 2026-10-25, and gated only on new code. sonar.yml runs cargo llvm-cov --workspace and npm run test:coverage (vitest's v8 provider) on Windows and hands both LCOV files to SonarQube Cloud, whose quality gate asks 80 % of new lines covered — new meaning since the previous version's period began, which at the switch was still the first analysis (2026-09-27), so all of 1.4.0 counted. The first measurement, of main at 1.4.0: Rust 71.6 % of 15 364 lines, TypeScript 68.4 % of 1 569. The two large holes are honest rather than unmeasured: pipdock-cli/src/run.rs (4 %) is the CLI driving a real interpreter, which L2 exercises and L1 cannot, and src-tauri/src/commands.rs (0 %) is the IPC surface, reached only by a running app. sonar-project.properties excludes from coverage only what no test here can execute (tooling, spikes, probe.py, generated bindings, the two entry points) and says why for each.

This section once set targets — core ≥ 85 % line coverage, classifiers and plan module ≥ 95 % — that nothing measured until then. They are not the gate; the gate is new code, and §1's rule — a test that failed first — is still the guarantee a percentage cannot give. By file, errors/catalog.rs is at 99.5 % and plan/ at 85–99 %.

Required on every PR (RELEASE-CI §2 has the exact payload): fmt · clippy · test — fmt, clippy -D warnings, and cargo test --workspace, which is L1, L4 and the Tauri crate's tests; lint · typecheck · test · build / build — ESLint, tsc, L3 and vite build; cargo audit; npm audit; and CodeQL's JavaScript/TypeScript analysis. cargo deny, tauri lockstep and CodeQL's Rust and Actions analyses run on every PR and are not required.

L2 runs nightly, on dispatch, and on PRs that touch what its end-to-end steps run through — the path list in ci-integration.yml, which 1.4.0 widened to the CLI, exec.rs, the audit, release, error, store and index modules, Cargo.lock and rust-toolchain.toml — each of them something the end-to-end steps run through, whose changes used to reach main without this job seeing them. (This line named engine|plan|snapshot|graph; the filter had outgrown that before 1.4.0.) It is not a required check: a path-filtered check that never starts stays pending forever.

The weekly drift job reports a failure to Discord through notify-ci-failure.yml, like any failure on main. It does not file an issue, which this section said it did.

4. Manual test charter (per release)

A short exploratory pass on a real machine: dirty 150+ package env, corporate-proxy simulation (PD-NET-002 copy), non-ASCII Windows username paths, VI locale sweep of all mutation dialogs, cancel mid-Phase-B, and SmartScreen/first-run installer experience.