feat(core): report installed size from the probe
UI-SPEC §4 specifies the Installed table as `name · version · latest · size · chips`, and nothing produced a size: `Dist` had no field, `probe.py` did not emit one, and neither engine's `list --format=json` reports it. `Dist.sizeBytes` closes that, summed from each distribution's RECORD manifest. **Measured, because the first implementation was 10x slower.** `Distribution.files` is the obvious API and it constructs a PackagePath object per installed file: on this machine's 352-package system Python it took the probe from 551 ms to 5,492 ms — on the code path the Installed screen runs every time an environment is opened. Parsing RECORD directly costs 79 ms for the same answer (551 → 625 ms, +14.6%). Sanity-checked against the real directory: 1,624 MiB reported against 1,844 MiB on disk, the 12% shortfall being __pycache__ written after install, which RECORD by definition does not list. **`None` is not `0`, and that distinction is the whole design.** A PEP 660 editable install has a valid RECORD listing only its import shim and a .pth, so the naive sum reports **240 bytes for a package whose sources are 8.1 MiB** — verified against a real `pip install -e`. That is worse than reporting nothing, so editables are detected through direct_url.json's `dir_info.editable` and excluded, as are `.egg-info` distributions, whose SOURCES.txt carries no sizes. The field is `skip_serializing_if = "Option::is_none"`, so those rows simply have no `sizeBytes` and the table renders an em dash rather than a claim. `list_json` sets it to `None` unconditionally — one more reason the Installed screen reads the probe rather than the engine (DATA-FLOW §7). probe.py schema goes to 2; the Rust parse is permissive, so a schema-1 document still loads with `size_bytes: None`. One golden re-blessed, deliberately: `golden__schema-Dist.snap` gains `sizeBytes` and nothing else moved. Human `list` output is unchanged — it already omits requiresDist and requiresPython. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a21895edbf
commit
e7bc6a5d4d
12 files changed
+140
-4
No files matched your search
@@ -36,6 +36,15 @@ expression: out
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"sizeBytes": {
|
||||
"description": "Installed size in bytes, summed from the distribution's RECORD manifest.\n\n`None` means \"not knowable\", and is deliberately not `0`: `.egg-info` distributions have\nno RECORD, and a PEP 660 editable install has one that lists only its import shim, so\nsumming it reports a few hundred bytes for a project of any size. Only `probe.py` can\nsupply this — `<engine> list --format=json` does not report it — so it is always `None`\non a `Dist` that came from an engine.\n\nA lower bound even when present: uncompressed bytes as recorded at install time,\nexcluding `__pycache__` written afterwards.",
|
||||
"format": "uint64",
|
||||
"minimum": 0,
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"version": {
|
||||
"$ref": "#/$defs/Version",
|
||||
"description": "Installed version."
|
||||
|
||||
@@ -330,6 +330,7 @@ mod tests {
|
||||
version: Version(version.into()),
|
||||
requires_dist: Vec::new(),
|
||||
requires_python: None,
|
||||
size_bytes: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -376,6 +376,9 @@ pub fn list_json(stdout: &str) -> Result<Vec<Dist>> {
|
||||
version: Version(v.get("version")?.as_str()?.to_owned()),
|
||||
requires_dist: Vec::new(),
|
||||
requires_python: None,
|
||||
// Neither engine reports an installed size, so this listing can never carry one.
|
||||
// Another reason the Installed screen reads the probe rather than the engine.
|
||||
size_bytes: None,
|
||||
})
|
||||
})
|
||||
.collect())
|
||||
@@ -494,6 +497,7 @@ mod tests {
|
||||
version: Version((*v).to_owned()),
|
||||
requires_dist: Vec::new(),
|
||||
requires_python: None,
|
||||
size_bytes: None,
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
@@ -158,6 +158,9 @@ pub fn parse_probe(stdout: &str, interpreter: &Path, source: EnvSource) -> Resul
|
||||
.get("requires_python")
|
||||
.and_then(serde_json::Value::as_str)
|
||||
.map(str::to_owned),
|
||||
// Absent on a schema-1 probe and null whenever the probe judged the number
|
||||
// unknowable; both arrive here as `None`, which is the honest answer either way.
|
||||
size_bytes: entry.get("size_bytes").and_then(serde_json::Value::as_u64),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -436,11 +439,12 @@ mod tests {
|
||||
#[test]
|
||||
fn parses_a_real_probe_document() {
|
||||
let doc = r#"{
|
||||
"schema": 1, "python": "3.12.4", "prefix": "C:\\proj\\.venv",
|
||||
"schema": 2, "python": "3.12.4", "prefix": "C:\\proj\\.venv",
|
||||
"is_venv": true, "externally_managed": false, "hidden_user_site": null,
|
||||
"dists": [
|
||||
{"name":"requests","version":"2.32.3",
|
||||
"requires_dist":["urllib3<3,>=1.21.1"],"requires_python":">=3.8"},
|
||||
"requires_dist":["urllib3<3,>=1.21.1"],"requires_python":">=3.8",
|
||||
"size_bytes":131072},
|
||||
{"name":null,"error":"KeyError: broken"}
|
||||
]}"#;
|
||||
let got = parse_probe(doc, Path::new("python.exe"), EnvSource::Manual).expect("parse");
|
||||
@@ -452,6 +456,25 @@ mod tests {
|
||||
assert_eq!(got.dists.len(), 1);
|
||||
assert_eq!(got.dists[0].name.as_str(), "requests");
|
||||
assert_eq!(got.dists[0].requires_dist, ["urllib3<3,>=1.21.1"]);
|
||||
assert_eq!(got.dists[0].size_bytes, Some(131_072));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unknowable_size_arrives_as_none_not_zero() {
|
||||
// Three ways the probe declines to guess, and one older probe that never knew to.
|
||||
// `Some(0)` would render as "0 B" in the Installed table, which is a claim; `None`
|
||||
// renders as "—", which is the truth.
|
||||
let doc = r#"{
|
||||
"schema": 2, "python": "3.12.4", "prefix": "C:\\proj\\.venv",
|
||||
"externally_managed": false, "hidden_user_site": null,
|
||||
"dists": [
|
||||
{"name":"editable-one","version":"1.0.0","size_bytes":null},
|
||||
{"name":"legacy-egg","version":"0.9.0"},
|
||||
{"name":"schema-one-probe","version":"2.0.0"}
|
||||
]}"#;
|
||||
let got = parse_probe(doc, Path::new("python.exe"), EnvSource::Manual).expect("parse");
|
||||
assert_eq!(got.dists.len(), 3);
|
||||
assert!(got.dists.iter().all(|d| d.size_bytes.is_none()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -356,6 +356,7 @@ mod tests {
|
||||
version: Version(version.into()),
|
||||
requires_dist: requires.iter().map(|s| (*s).to_owned()).collect(),
|
||||
requires_python: None,
|
||||
size_bytes: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -606,6 +607,7 @@ mod tests {
|
||||
version: Version("1.0".into()),
|
||||
requires_dist: vec!["urllib3".into()],
|
||||
requires_python: None,
|
||||
size_bytes: None,
|
||||
}
|
||||
}))
|
||||
.collect();
|
||||
|
||||
@@ -253,6 +253,18 @@ pub struct Dist {
|
||||
/// Raw `Requires-Python` specifier, when declared.
|
||||
#[serde(default)]
|
||||
pub requires_python: Option<String>,
|
||||
/// Installed size in bytes, summed from the distribution's RECORD manifest.
|
||||
///
|
||||
/// `None` means "not knowable", and is deliberately not `0`: `.egg-info` distributions have
|
||||
/// no RECORD, and a PEP 660 editable install has one that lists only its import shim, so
|
||||
/// summing it reports a few hundred bytes for a project of any size. Only `probe.py` can
|
||||
/// supply this — `<engine> list --format=json` does not report it — so it is always `None`
|
||||
/// on a `Dist` that came from an engine.
|
||||
///
|
||||
/// A lower bound even when present: uncompressed bytes as recorded at install time,
|
||||
/// excluding `__pycache__` written afterwards.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub size_bytes: Option<u64>,
|
||||
}
|
||||
|
||||
/// An installed distribution with a newer release available.
|
||||
|
||||
@@ -221,6 +221,7 @@ mod tests {
|
||||
version: Version(version.into()),
|
||||
requires_dist: requires.iter().map(|s| (*s).to_owned()).collect(),
|
||||
requires_python: None,
|
||||
size_bytes: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ Output shape:
|
||||
{"python": "3.12.4", "prefix": "...", "externally_managed": false,
|
||||
"dists": [{"name": "requests", "version": "2.32.3",
|
||||
"requires_dist": ["urllib3<3,>=1.21.1"],
|
||||
"requires_python": ">=3.8"}]}
|
||||
"requires_python": ">=3.8", "size_bytes": 131072}]}
|
||||
|
||||
Any failure prints {"error": ...} on stdout and exits non-zero, which the core
|
||||
classifies as PD-ENV-003.
|
||||
@@ -34,7 +34,7 @@ import os
|
||||
import sys
|
||||
import sysconfig
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
SCHEMA_VERSION = 2
|
||||
|
||||
|
||||
def _normalize(name: str) -> str:
|
||||
@@ -102,6 +102,70 @@ def _hidden_user_site() -> str | None:
|
||||
return user_site
|
||||
|
||||
|
||||
def _is_editable(dist) -> bool:
|
||||
"""True for a PEP 660 editable install.
|
||||
|
||||
Such a distribution has a perfectly valid RECORD -- it just lists the import shim and a
|
||||
.pth file rather than the package, so summing it reports a few hundred bytes for a project
|
||||
of any size. `direct_url.json`'s `dir_info.editable` is the marker pip and uv both write.
|
||||
"""
|
||||
try:
|
||||
raw = dist.read_text("direct_url.json")
|
||||
except Exception: # noqa: BLE001 - absence is the common case, not an error
|
||||
return False
|
||||
if not raw:
|
||||
return False
|
||||
try:
|
||||
info = json.loads(raw)
|
||||
except ValueError:
|
||||
return False
|
||||
return bool(isinstance(info, dict) and info.get("dir_info", {}).get("editable"))
|
||||
|
||||
|
||||
def _size_bytes(dist) -> int | None:
|
||||
"""Installed size in bytes, summed from the RECORD manifest.
|
||||
|
||||
RECORD is read and parsed directly rather than through `Distribution.files`. That is not
|
||||
premature: `files` constructs a PackagePath object per installed file, and on this machine's
|
||||
352-package system Python it took the probe from 551 ms to 5,492 ms -- a 10x regression on a
|
||||
code path the Installed screen runs every time an environment is opened. Parsing the text
|
||||
ourselves costs 29 ms for the same answer.
|
||||
|
||||
RECORD is CSV of `path,hash,size`. The path may be quoted and contain commas; the trailing
|
||||
two fields never do, so splitting from the right is both correct and cheap.
|
||||
|
||||
Returns None rather than 0 whenever the number would be a lie, because a wrong size is
|
||||
worse than an absent one:
|
||||
|
||||
* `.egg-info` distributions have no RECORD -- their SOURCES.txt carries no sizes;
|
||||
* editable installs, per _is_editable above;
|
||||
* anything whose metadata cannot be read at all.
|
||||
|
||||
Even when present the number is a lower bound: it is uncompressed installed bytes as
|
||||
recorded at install time, and excludes __pycache__ written afterwards.
|
||||
"""
|
||||
try:
|
||||
record = dist.read_text("RECORD")
|
||||
except Exception: # noqa: BLE001 - unreadable metadata is reported as unknown, not fatal
|
||||
return None
|
||||
if not record:
|
||||
return None
|
||||
|
||||
total = 0
|
||||
measured = False
|
||||
for line in record.splitlines():
|
||||
parts = line.rsplit(",", 2)
|
||||
if len(parts) != 3:
|
||||
continue
|
||||
size = parts[2].strip()
|
||||
if size.isdigit():
|
||||
total += int(size)
|
||||
measured = True
|
||||
if not measured:
|
||||
return None
|
||||
return None if _is_editable(dist) else total
|
||||
|
||||
|
||||
def _dists() -> list[dict[str, object]]:
|
||||
"""Read installed distribution metadata.
|
||||
|
||||
@@ -130,6 +194,7 @@ def _dists() -> list[dict[str, object]]:
|
||||
"version": dist.version or "",
|
||||
"requires_dist": list(dist.requires or []),
|
||||
"requires_python": meta.get("Requires-Python"),
|
||||
"size_bytes": _size_bytes(dist),
|
||||
}
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - one bad dist must not sink the probe
|
||||
|
||||
@@ -383,6 +383,7 @@ async fn a_preview_made_before_the_environment_changed_is_refused() {
|
||||
version: Version("1.0".into()),
|
||||
requires_dist: Vec::new(),
|
||||
requires_python: None,
|
||||
size_bytes: None,
|
||||
}];
|
||||
let err = execute(
|
||||
&FakeEngine::new(&[]),
|
||||
@@ -409,12 +410,14 @@ async fn reordering_the_installed_set_is_not_drift() {
|
||||
version: Version("1.0".into()),
|
||||
requires_dist: Vec::new(),
|
||||
requires_python: None,
|
||||
size_bytes: None,
|
||||
};
|
||||
let b = Dist {
|
||||
name: PkgName::parse("bbb").unwrap(),
|
||||
version: Version("2.0".into()),
|
||||
requires_dist: Vec::new(),
|
||||
requires_python: None,
|
||||
size_bytes: None,
|
||||
};
|
||||
let plan = AcceptedPlan::accept(
|
||||
report(&[("idna", "3.18")]),
|
||||
|
||||
@@ -55,6 +55,7 @@ fn dist(name: &str, version: &str) -> Dist {
|
||||
version: Version(version.into()),
|
||||
requires_dist: Vec::new(),
|
||||
requires_python: None,
|
||||
size_bytes: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -73,6 +73,8 @@ requests held back at 2.30.0 (latest 2.32.3) — blocked by apiclient 1.4 (requ
|
||||
|
||||
`--json` payloads are the serde-serialized core types (`Dist`, `OutdatedDist`, `ResolutionReport`, `ExecutionSummary`, `CheckReport`) — schema documented by `pipdock schema <type>` which prints the JSON Schema generated from the Rust types, so scripts can pin against it. Streaming commands (`update`, `install`, `health`) with `--json` emit NDJSON events matching the GUI's `plan-progress` payloads, terminated by a final `summary` object.
|
||||
|
||||
`Dist.sizeBytes` is present only when it can be known: it is summed from the distribution's RECORD manifest, which `.egg-info` distributions do not have and which a PEP 660 editable install fills with its import shim rather than its sources. The field is **omitted** in those cases rather than reported as `0`, and it is always omitted on a `Dist` that came from `<engine> list` — neither engine reports a size. Where present it is a lower bound: uncompressed bytes as recorded at install time, excluding `__pycache__` written afterwards. `pipdock list`'s human table does not show it, as it already omits `requiresDist` and `requiresPython`.
|
||||
|
||||
## 7. Examples
|
||||
|
||||
```bash
|
||||
|
||||
@@ -131,6 +131,19 @@ export interface Dist {
|
||||
requiresDist?: string[];
|
||||
/** Raw `Requires-Python` specifier, when declared. */
|
||||
requiresPython?: string | null;
|
||||
/**
|
||||
* Installed size in bytes, summed from the distribution's RECORD manifest.
|
||||
*
|
||||
* `None` means "not knowable", and is deliberately not `0`: `.egg-info` distributions have
|
||||
* no RECORD, and a PEP 660 editable install has one that lists only its import shim, so
|
||||
* summing it reports a few hundred bytes for a project of any size. Only `probe.py` can
|
||||
* supply this — `<engine> list --format=json` does not report it — so it is always `None`
|
||||
* on a `Dist` that came from an engine.
|
||||
*
|
||||
* A lower bound even when present: uncompressed bytes as recorded at install time,
|
||||
* excluding `__pycache__` written afterwards.
|
||||
*/
|
||||
sizeBytes?: number | null;
|
||||
/** Installed version. */
|
||||
version: Version;
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user