4 Commits
Author SHA1 Message Date
poli0981andClaude Opus 5 5023e39485 feat(core): the pip-audit parser, and SP-4's fixture at last
SP-4's exit criterion was "command line + JSON fixture pinned". The command line was
pinned in prose in Phase 0; the fixture never was, so every claim about pip-audit's
output in PRD, SECURITY and CODE-HEALTH-SPEC has been unchecked since. It is captured
now, from the pinned 2.10.1 against the spike's own `urllib3==2.0.0` environment, and
`crates/pipdock-core/src/audit.rs` is written against it rather than against the
documents.

Four things running it settled, none of which reading could:

- **No severity exists, under either service.** The vuln object is exactly `id`,
  `fix_versions`, `aliases`, `description`; pip-audit's service model has no severity
  field at all. PRD P1-1's "severity-sorted" and SECURITY §6's "(CVE/GHSA id,
  severity, fixed-in)" were both written against a field that has never existed. The
  next commit corrects them; `sort_advisories` orders by package, then by whether
  anything *fixes* it, then by id — the one distinction that changes what a user can
  do. Deriving a severity from the id's year or the prose would be a number PipDock
  invented, shown beside real ones.
- **The ids are `PYSEC-*`, so the default service is PyPI**, and the CVE is an alias.
  Leaving that default is what keeps `legal/PRIVACY-POLICY.md` §3's "exactly one
  destination" true — and it costs nothing: the same environment under
  `--vulnerability-service osv` returns the *same eight advisories*, set difference
  empty in both directions.
- **Ids repeat**, so dedup is load-bearing rather than tidy. Ten rows for eight
  advisories under PyPI; sixteen for the same eight under OSV. Keyed on `(pkg, id)`,
  because one advisory can legitimately apply to two installed packages.
- **stderr carries a `pip-compile` advisory on every run**, so stdout is parsed first
  and failure classified only if that fails — SP-1's rule for uv, which writes its
  plan to stderr for the same reason.

`advisory_url` builds the OSV link from a **validated** id rather than taking a URL
out of the tool's JSON. That is the difference between this and `PdHealthReport`'s
`finding.url`, which `capabilities/external-links.json` calls out as the only URL in
the application derived from a third party's output: a whitelist of ASCII alphanumerics
and `-` cannot produce a `/`, `?`, `#`, `:` or `%`, so nothing that passes can be
anything but a path segment under one host. Ten hostile ids are asserted to get no
link at all.

No wire registration yet — the types derive `JsonSchema` but stay out of
`SCHEMA_TYPES` until a command returns one, exactly as P2 held `ToolsManifest` back
until P3 had a use for it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 10:55:11 +07:00
poli0981andClaude Opus 5 93285ad262 feat(core): three parsers, against captured output rather than a guess
The commit that earns the slice. Parsing is where P3's risk lives — three formats PipDock
does not control, bumped by Dependabot on its own schedule — and all of it is testable
without ever spawning a subprocess. Runners come next; if these are wrong nothing after
them can be right.

CODE-HEALTH-SPEC §5 is amended on three points, each because the tools were run:

deptry emits a **flat list**, one object per (code, module, location), keyed by `module`.
§5 sketched a per-dependency object with a `locations` array; that shape does not exist.
Grouping is therefore PipDock's job, done on (code, module) rather than module alone — one
dependency can be both DEP001 and DEP003, and merging those yields a row whose message
describes only one of them. `dep` is a **module** name: yaml is PyYAML, cv2 is
opencv-python, so anything handing it to a package operation must reconcile it first.

vulture has no machine-readable output at all. Its eighth message shape, `unreachable code
after '<token>'`, quotes the token the code follows rather than a dead identifier — so
`name` is optional and quoting alone cannot decide it. The line number is found by
splitting from the right, because `C:\proj\a.py:12:` has three colons and a left-to-right
split would ship broken on every Windows machine.

ruff's docs URL is keyed by rule **name**: I001 lives at .../rules/unsorted-imports, so
§6's .../rules/<code> 404s. Carried, never constructed. A syntax error arrives as
`invalid-syntax` with a null url — not a null code, which is what §6 assumed.

The exit-code table is the sharpest thing here. All three exit non-zero on findings and
they disagree about how — deptry 1, ruff 1, **vulture 3**, with vulture's 1 and 2 being
real failures. A plain !out.ok() reports every successful run over a real project as
PD-HLT-002, and a suite that only runs the tools over a clean directory agrees with it.

Fixtures are captured from the pinned versions and redacted: ruff's `filename` is absolute
and carried this machine's home directory, which is the exact failure CLAUDE.md records pip
shipping to a public repo. They are also `-text` in .gitattributes, as an exception to the
rule above them — under health/ the JSON *is* the tool's output, not a sidecar of ours, and
normalising a captured document's line endings makes it something no tool ever emitted.

One of these fixtures was captured with `2>&1` first and every parse failed: ruff writes
findings to stdout and its warnings to stderr. A test now says so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 23:31:37 +07:00
poli0981andClaude Opus 5 4066aa63d4 fix(spikes): make fixture captures reproducible, and stop leaking $HOME
The weekly fixture-drift job has never been able to pass, and could not have.
Every capture built its venv in a `mkdtemp` directory whose name carries a random
suffix, and `redact()` replaced only the temp *root*, so the suffix survived into
every committed sidecar. A 100% false-positive rate, independent of any engine
release.

Four more sources of churn were underneath it, found by running the captures twice
back to back rather than by reasoning about them:

  - pip's "[notice] To update, run: <venv>\Scripts\python.exe" embeds the venv
    path -- which also shipped the capturing user's home directory to a public repo
  - CPython object addresses in urllib3 retry warnings, moved by ASLR
    ("<HTTPConnection object at 0x000001A92FBCFBC0>")
  - pip's download progress bar, which carries a transfer rate and an ETA
  - uv's per-phase timings ("Resolved 3 packages in 775ms")

Plus two absolute interpreter paths that can never match a re-capture on CI, where
Python lives in hostedtoolcache rather than under the user profile, and pip's cache
state, which turns "Downloading numpy-1.19.5.zip (7.3 MB)" into "Using cached ..."
depending on what the machine happened to have. A runner is always cold; a dev
machine is usually warm.

So: redact all of it, capture cache-free so both sides take the download path, and
split the sidecar by what each half is for. meta.json is the contract the gate
watches; capture-provenance.json records engine and interpreter versions and argv,
churns on every release, and is excluded from the gate by pathspec -- still
committed, so `git diff` always tells you which engine produced the current bytes.

This is the one place fixtures are not literal engine bytes, against the letter of
TESTING section 1. It is safe because nothing reads a path or a duration out of
them: every CLASSIFIERS entry in errors/catalog.rs keys off message text, and the
uv adapter only tests contains("Resolved "). Line endings are untouched -- `-text`
is what protects those. Verified: 173 tests green, and two full capture passes over
both engines differ in 0 of 96 files.

Two smaller things in the same area, both of which hid the problem:

  - .gitattributes set `-diff` as well as `-text` on the fixture tree. `-text` is
    what preserves the CRLF bytes the comment argues for; `-diff` only suppresses
    the textual diff, so the drift job's own diagnostics could report nothing but
    "Binary files differ". Scope byte-exactness to the .txt captures and let the
    JSON sidecars be diffable.
  - the "weekly, daily would be rude" drift job matched the nightly cron, the only
    one in the file, so it ran nightly. Give it its own '17 4 * * 1', and drop the
    run_attempt guard -- re-running is how you reproduce a drift failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 20:26:01 +07:00
poli0981andClaude Opus 5 7f152200ac fix: preserve engine fixture bytes exactly
Git normalised CRLF to LF when the fixtures were committed: the blob for
uv/yanked/stderr.txt held 0 carriage returns where the real engine output
has 6. pip and uv emit CRLF on Windows, so the adapter parsers would have
been tested against line endings that never occur in production — a
parser mishandling a trailing \r would pass CI and fail on every real
Windows machine.

TESTING section 1 makes adapter parsing the thing that must never
regress, so .gitattributes marks the fixture tree -text -diff and the
blobs are re-added with their true bytes. The override is listed last
because later gitattributes rules win; placed before `* text=auto` it has
no effect.

Verified with git check-attr and by comparing carriage returns in the
staged blob against the working file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-27 18:06:34 +07:00