main
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
5023e39485 |
feat(core): the pip-audit parser, and SP-4's fixture at last
SP-4's exit criterion was "command line + JSON fixture pinned". The command line was pinned in prose in Phase 0; the fixture never was, so every claim about pip-audit's output in PRD, SECURITY and CODE-HEALTH-SPEC has been unchecked since. It is captured now, from the pinned 2.10.1 against the spike's own `urllib3==2.0.0` environment, and `crates/pipdock-core/src/audit.rs` is written against it rather than against the documents. Four things running it settled, none of which reading could: - **No severity exists, under either service.** The vuln object is exactly `id`, `fix_versions`, `aliases`, `description`; pip-audit's service model has no severity field at all. PRD P1-1's "severity-sorted" and SECURITY §6's "(CVE/GHSA id, severity, fixed-in)" were both written against a field that has never existed. The next commit corrects them; `sort_advisories` orders by package, then by whether anything *fixes* it, then by id — the one distinction that changes what a user can do. Deriving a severity from the id's year or the prose would be a number PipDock invented, shown beside real ones. - **The ids are `PYSEC-*`, so the default service is PyPI**, and the CVE is an alias. Leaving that default is what keeps `legal/PRIVACY-POLICY.md` §3's "exactly one destination" true — and it costs nothing: the same environment under `--vulnerability-service osv` returns the *same eight advisories*, set difference empty in both directions. - **Ids repeat**, so dedup is load-bearing rather than tidy. Ten rows for eight advisories under PyPI; sixteen for the same eight under OSV. Keyed on `(pkg, id)`, because one advisory can legitimately apply to two installed packages. - **stderr carries a `pip-compile` advisory on every run**, so stdout is parsed first and failure classified only if that fails — SP-1's rule for uv, which writes its plan to stderr for the same reason. `advisory_url` builds the OSV link from a **validated** id rather than taking a URL out of the tool's JSON. That is the difference between this and `PdHealthReport`'s `finding.url`, which `capabilities/external-links.json` calls out as the only URL in the application derived from a third party's output: a whitelist of ASCII alphanumerics and `-` cannot produce a `/`, `?`, `#`, `:` or `%`, so nothing that passes can be anything but a path segment under one host. Ten hostile ids are asserted to get no link at all. No wire registration yet — the types derive `JsonSchema` but stay out of `SCHEMA_TYPES` until a command returns one, exactly as P2 held `ToolsManifest` back until P3 had a use for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
93285ad262 |
feat(core): three parsers, against captured output rather than a guess
The commit that earns the slice. Parsing is where P3's risk lives — three formats PipDock does not control, bumped by Dependabot on its own schedule — and all of it is testable without ever spawning a subprocess. Runners come next; if these are wrong nothing after them can be right. CODE-HEALTH-SPEC §5 is amended on three points, each because the tools were run: deptry emits a **flat list**, one object per (code, module, location), keyed by `module`. §5 sketched a per-dependency object with a `locations` array; that shape does not exist. Grouping is therefore PipDock's job, done on (code, module) rather than module alone — one dependency can be both DEP001 and DEP003, and merging those yields a row whose message describes only one of them. `dep` is a **module** name: yaml is PyYAML, cv2 is opencv-python, so anything handing it to a package operation must reconcile it first. vulture has no machine-readable output at all. Its eighth message shape, `unreachable code after '<token>'`, quotes the token the code follows rather than a dead identifier — so `name` is optional and quoting alone cannot decide it. The line number is found by splitting from the right, because `C:\proj\a.py:12:` has three colons and a left-to-right split would ship broken on every Windows machine. ruff's docs URL is keyed by rule **name**: I001 lives at .../rules/unsorted-imports, so §6's .../rules/<code> 404s. Carried, never constructed. A syntax error arrives as `invalid-syntax` with a null url — not a null code, which is what §6 assumed. The exit-code table is the sharpest thing here. All three exit non-zero on findings and they disagree about how — deptry 1, ruff 1, **vulture 3**, with vulture's 1 and 2 being real failures. A plain !out.ok() reports every successful run over a real project as PD-HLT-002, and a suite that only runs the tools over a clean directory agrees with it. Fixtures are captured from the pinned versions and redacted: ruff's `filename` is absolute and carried this machine's home directory, which is the exact failure CLAUDE.md records pip shipping to a public repo. They are also `-text` in .gitattributes, as an exception to the rule above them — under health/ the JSON *is* the tool's output, not a sidecar of ours, and normalising a captured document's line endings makes it something no tool ever emitted. One of these fixtures was captured with `2>&1` first and every parse failed: ruff writes findings to stdout and its warnings to stderr. A test now says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
4066aa63d4 |
fix(spikes): make fixture captures reproducible, and stop leaking $HOME
The weekly fixture-drift job has never been able to pass, and could not have.
Every capture built its venv in a `mkdtemp` directory whose name carries a random
suffix, and `redact()` replaced only the temp *root*, so the suffix survived into
every committed sidecar. A 100% false-positive rate, independent of any engine
release.
Four more sources of churn were underneath it, found by running the captures twice
back to back rather than by reasoning about them:
- pip's "[notice] To update, run: <venv>\Scripts\python.exe" embeds the venv
path -- which also shipped the capturing user's home directory to a public repo
- CPython object addresses in urllib3 retry warnings, moved by ASLR
("<HTTPConnection object at 0x000001A92FBCFBC0>")
- pip's download progress bar, which carries a transfer rate and an ETA
- uv's per-phase timings ("Resolved 3 packages in 775ms")
Plus two absolute interpreter paths that can never match a re-capture on CI, where
Python lives in hostedtoolcache rather than under the user profile, and pip's cache
state, which turns "Downloading numpy-1.19.5.zip (7.3 MB)" into "Using cached ..."
depending on what the machine happened to have. A runner is always cold; a dev
machine is usually warm.
So: redact all of it, capture cache-free so both sides take the download path, and
split the sidecar by what each half is for. meta.json is the contract the gate
watches; capture-provenance.json records engine and interpreter versions and argv,
churns on every release, and is excluded from the gate by pathspec -- still
committed, so `git diff` always tells you which engine produced the current bytes.
This is the one place fixtures are not literal engine bytes, against the letter of
TESTING section 1. It is safe because nothing reads a path or a duration out of
them: every CLASSIFIERS entry in errors/catalog.rs keys off message text, and the
uv adapter only tests contains("Resolved "). Line endings are untouched -- `-text`
is what protects those. Verified: 173 tests green, and two full capture passes over
both engines differ in 0 of 96 files.
Two smaller things in the same area, both of which hid the problem:
- .gitattributes set `-diff` as well as `-text` on the fixture tree. `-text` is
what preserves the CRLF bytes the comment argues for; `-diff` only suppresses
the textual diff, so the drift job's own diagnostics could report nothing but
"Binary files differ". Scope byte-exactness to the .txt captures and let the
JSON sidecars be diffable.
- the "weekly, daily would be rude" drift job matched the nightly cron, the only
one in the file, so it ran nightly. Give it its own '17 4 * * 1', and drop the
run_attempt guard -- re-running is how you reproduce a drift failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
7f152200ac |
fix: preserve engine fixture bytes exactly
Git normalised CRLF to LF when the fixtures were committed: the blob for uv/yanked/stderr.txt held 0 carriage returns where the real engine output has 6. pip and uv emit CRLF on Windows, so the adapter parsers would have been tested against line endings that never occur in production — a parser mishandling a trailing \r would pass CI and fail on every real Windows machine. TESTING section 1 makes adapter parsing the thing that must never regress, so .gitattributes marks the fixture tree -text -diff and the blobs are re-added with their true bytes. The override is listed last because later gitattributes rules win; placed before `* text=auto` it has no effect. Verified with git check-attr and by comparing carriage returns in the staged blob against the working file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |