Files
frameledger/docs/14_TESTING.md
T
poli0981andClaude Opus 5 81011315ab feat(tier2): PresentMon is dropped, and Tier 2 keeps its place while losing its mechanism (#96)
Owner decision 2026-08-27. §S31 retired PresentMon as a measurement ORACLE (row
P2); this removes it as an IMPLEMENTATION. It is not bundled, not fetched, not
used, and tools/frametype-oracle.ps1 -- the parser written for its output -- is
deleted with it, along with its build.ps1 gate.

THE DISTINCTION THIS COMMIT IS BUILT ON: dropping PresentMon is not dropping
Tier 2. The owner's earlier instruction was explicit that a release "will use a
different approach", so Tier 2 keeps its place in the design and loses its named
tool. Nothing here rewrites the tier out of the product; everything here stops
naming an implementation nobody chose.

TWELVE DOCUMENTS NAMED IT AS THE MECHANISM AND EVERY ONE IS CORRECTED:
CLAUDE.md's pinned-stack row; README's capture-tier table AND its safety row;
01_ARCHITECTURE's tier table and ADR-3; 03_METRICS rung 2; 04_CAPTURE's source
table and tier selection; 05_DETECTION's two Tier-2 rows; 07_IPC's
PresentMonMissing error code; 12_BUILD; 14_TESTING's CSV-parser row;
15_ROADMAP's P2 phase and v2 backlog; and legal/THIRD_PARTY_NOTICES.md plus
legal/licenses/README.md. Struck in place rather than deleted wherever the
reasoning is what a later reader needs.

THE NOTICES ROW AND license-check's CLAIM HAD TO GO TOGETHER, AND THAT IS THE
GATE WORKING. tools/license-check.ps1 bound `Intel PresentMon` to
assets/native/PresentMon.exe BIDIRECTIONALLY, so removing the row alone fails
with "no table row mentions Intel PresentMon, so its bundling claim cannot be
checked". Removing a claim is a deliberate act and is recorded as one, in the
notices file itself. legal/licenses/README.md also listed a presentmon-MIT.txt
that never existed -- a row pointing at a file nobody had shipped.

§M1, §M2 AND §M6 CLOSE BY DECISION, NOT BY MEASUREMENT, and each row says which.
They were questions about a tool that no longer has a subject. §M2's answers are
kept in full -- the binary's identity, its refusal to run unelevated, the 24-column
set -- because every one of them cost a measurement, and a closed row that deletes
its evidence is how the next person re-runs it.

§G IS NOT CLOSED. IT GETS BIGGER. "Is the binary committed or fetched?" becomes
"by what mechanism does a shipped build reach Tier 2 at all?", and nothing in
this repository names one. The row now carries its own constraints: README's
capture-tier table and its SAFETY row, legal/DISCLAIMER.md §73 and
legal/EULA.md §33 all describe a no-injection mode as THE DEFAULT for uncertain
cases; ETW needs elevation regardless of tool (§M6); and 15_ROADMAP has no v2
fallback any more either.

WHAT THIS COSTS USERS, PUT WHERE THEY WILL READ IT. README's safety row promised
"always a way out". It now carries, in that row, that the mode is unbuilt and has
no chosen mechanism, and that A REFUSED OR UNHOOKABLE TITLE GETS TIER 3 --
session duration and hardware telemetry. Said in the safety table rather than in
a footnote, because a way out that does not exist is the wrong thing to be
reassured by.

THE TWO LEGAL DOCUMENTS ARE DELIBERATELY NOT TOUCHED. DISCLAIMER §73 and EULA §33
still call the no-injection mode "the default" without the qualification the
README now carries. That is a real gap, §G names both files, and changing a legal
document is not a cleanup -- it is an owner decision with its own review.

AND THE DISCRIMINATOR RUN PRODUCED NOTHING, so it settled nothing. The command I
supplied targeted explorer.exe, which may present nothing PresentMon tracks, so
"no CSV" and "no FrameType column" are indistinguishable -- the §Traps entry
about a canary that dies before reaching the gate, in a command I wrote. A target
known to present would have settled it. Recorded in §S31 and spike-notes §11 so
nobody re-runs it believing it is untried, and moot either way now.

Also corrected while in the file: CLAUDE.md's tools list omitted
hookinventory-check. Same class as §R10 and found by counting the directory
rather than by reading the line.

./build.ps1 check with no switches: PASSED, one skipped gate (resx-audit, which
does not exist). license-check green -- the row/claim pair was removed
consistently, which is the only way that gate can stay green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>


---

docs: two pointers to a script that was never landed, left by closing #94

`spike-notes` §11 and `HANDOFF` item 3 both credited the §S31 run to
`tools/s31-capture.ps1`. That script lived in #94, which was closed unmerged
because it drives PresentMon and PresentMon was dropped the same day. So two
documents on `main` name a file the repository does not contain -- a reader
follows the pointer and finds nothing.

Caught by grepping `main` for the name after the close, not by reading the
diff: the reference was correct when it was written and became wrong when a
DIFFERENT pull request was closed. That is the "a document can go stale by NOT
being touched" entry arriving from the one direction §Traps does not describe --
staleness caused by a decision taken elsewhere.

Both now describe the run without naming an artifact, and `spike-notes` says
explicitly that the script is not in this repository so nobody goes looking.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 09:57:36 +07:00

11 KiB
Raw Blame History

14 — Testing

Managed: xUnit + FluentAssertions + NSubstitute. Native: Catch2. Coverage goal ≥ 80% on Domain + Application; Domain metric calculators ≥ 95% or the PR fails.

Safety-guard tests (highest priority — these protect users, not code quality)

The anti-cheat guard is the one component where a bug can cost someone an account. It gets the most rigorous treatment in the codebase:

These are Catch2 tests, not xUnit. 20_OPEN_QUESTIONS §S13(a) settled the guard into the C++ FrameLedger.Injector, so the fail-closed matrix below lives in the native suite and the guard needs seams — injectable enumerator function pointers — before any of these failures can be forced at all. Landing Catch2 is a prerequisite of the guard, not a later nicety (§S15).

  • Blocklist matching: exact, case-insensitive, prefix rules; every family in 19_SAFETY §Blocklist seed has a fixture. Two families (Activision Ricochet, Valve VAC) have no data yet, so their fixtures must assert that absence explicitly rather than quietly passing on an empty rule set.
  • Fail-closed proofs: malformed detection-rules.json, missing anticheat block, unreadable target process, EnumProcessModulesEx failure, partial module list → all must refuse injection, never allow. Each is a named test.
  • Measured error paths that must be named tests (spike-notes.md §1): ERROR_PARTIAL_COPY (299) from a suspended target, ERROR_ACCESS_DENIED (5) from a protected one, a driver-list parse whose paths are not native paths, and a service query returning ACCESS_DENIED — the last of which cannot be produced on real services by a standard user, so it exists only as a fake. Every one means REFUSE.
  • Mid-session detection: simulated late-loading anti-cheat module → unhookRequested set, hooks disabled within one frame, session finalized unhooked_safety. Cover the driver scan too, not only modules: a machine-wide driver can start after injection.
  • Absence-of-override test: no code path may reach the injection primitive without a passing guard result. The mechanism previously described here — a sealed token type "only the guard can produce" — does not work and was disproved by compiling (20_OPEN_QUESTIONS §S8: C# accessibility flows inward, CS0122). The replacement is structural: the guard owns the chokepoint and calls the primitive itself, the primitive has internal linkage in the guard's own translation unit so no other TU has a symbol to call, and a build-time check asserts that nothing else references it. A token that escapes can be ignored; a symbol that does not exist cannot.
  • Static pre-scan: a game directory containing EasyAntiCheat/ renders hook_enabled un-settable; API-level attempt to set it is rejected — including via SetWatchlist, which no longer carries hooking state at all (07_IPC §The pipe is not a trust boundary). The toggle half needs persistence and is not built yet; the scan itself is ctest fl_prescan, whose matrix is mostly fail-closed: a listing that FAILED, one that was TRUNCATED, a directory that could not be established, unusable rules, and a missing evidence source must each be undetermined, never clean. Two cases exist only to stop the rest passing vacuously — a genuinely clean directory must be allowed, and a name matching the wrong group (a directory named like a files entry) must not fire. One case removes a family from the rules and asserts the hit disappears, which is what proves there is a single matcher rather than a second one that happens to agree.

Native unit tests (Catch2)

  • Ring buffer: SPSC correctness under a hammering writer thread; seqlock torn-read detection; wrap and overwrite-oldest semantics; power-of-two capacity assumptions. Include the one-full-lap case: a reader stalled exactly capacity records must not validate a different frame as unchanged (this is what the never-reset seq counter defends, 07_IPC §Protocol rules).
  • Drop accounting: the reader-side computation (writeIndex - readIndex > capacity) reports the right count and resumes at the right index. The writer has no drop counter to test — by design.
  • Torn records produce gap markers, not silently shortened streams; a golden test asserts the gap does not appear as a stutter.
  • Record and header layout: static_assert(sizeof(...) == 64) and offsetof assertions for FlFrameRecord and all three header structs (FlShmHandshake, FlWriterState, FlControlBlock) — the Agent reads across the process boundary in every one of them. Both halves exist as of 2026-08-05: the native side is layout_test.cpp plus the static_asserts (ctest fl_shm_layout), and the runtime offset dump from tools/fl-layout-dump is consumed by ShmLayoutMirrorTests, which walks the field list in both directions and additionally asserts blittability — Marshal.SizeOf and every offset stay correct for a mirror that uses [MarshalAs(ByValTStr)] string, while Unsafe.SizeOf drops from 64 to 40 and the struct can no longer be read from a mapped view.
  • Fault policy: injected SEH faults in a fake hook body → counter increments, original still called, self-disable at 3.
  • Hook index verification, by behaviour. A vtable slot carries no identity, so "check that slot 8 is Present" is not a question the runtime can answer — the earlier wording here required something unimplementable. What is answerable: patch the slot, call the method, see whether the detour ran (17_HOOK_ENGINE §Getting vtable addresses; proved for slots 8, 13 and 22 in spike-notes.md §H4). Runs headless on CI via WARP and CreateSwapChainForComposition. A deliberate mismatch aborts installation instead of patching.
  • Every probe must be shown to fail. A probe that ends in an unconditional assertion is green by construction and its ctest is decorative — fl_proxy_swapchain shipped in exactly that state. Before a probe counts as a regression net, break the thing it watches and watch it go red.

Managed unit tests

Golden metric tests (write first)

  • Hand-computed fixtures: time-based avg (not mean-of-FPS), median, p99/p99.9 with linear interpolation (include cases where interpolation and nearest-rank differ), min/max, σ, stutter rule, sufficiency guards at 999/1,000 and 9,999/10,000 frames.
  • FG ladder: records with API-sourced FG → fg_source = api; present-count delta only → presentdelta; ETW FrameType → etw; cadence ≥ 1.5 → cadence; ambiguous → none. Factor always = Displayed/Native regardless of rung.
  • Upscaling: exact ratio from render/output pairs; segment splitting when resolution changes mid-stream; settings_changed_midsession flag; dominant-segment selection.
  • RT: AS-builds-only stream (inline RayQuery case) → rt = Yes — the specific regression the old design got wrong; dispatch-only → Yes; RT-capable device with neither → No; D3D11 → N/A. PT confidence scoring never yields Yes.
  • Tri-state precedence: manual > measured > inherited > N/A.
  • Tier gating: a Tier-2 record stream must produce N/A for upscaler/RT/VRAM/latency — never a fabricated value.

Parsers & infrastructure

  • Shm reader against a synthetic mapping: version mismatch → refuse; torn records skipped; dropped counter propagated.
  • PresentMon CSV (Tier 2): header-map building with shuffled/unknown/missing columns, malformed lines, invariant decimal. Fixtures recorded from real runs. Removed 2026-08-27 with the tool. Whatever Tier 2 ends up parsing needs its own row, written against the format it actually reads — and the lesson from the one parser that did get written is worth carrying: its CSV's first column was named Application while the value it counted was also Application, so resolving by NAME rather than position was load-bearing (spike-notes §11).
  • Steam ACF (KeyValues), GOG .info, Epic .item, itch receipt — real-world fixtures.
  • RuleEvaluator: fixture directory trees per engine, including the "Unity markers present but UE structure too" ordering case.
  • Blob codecs round-trip for every series in frame_blobs (NaN forbidden — assert), including the two-pair render_res encoding and the three-bit rt_flags byte. SQLite migrations apply cleanly from an empty file to the current schema, and re-applying is a no-op. (There is no v1→v2 upgrade to test — 06_DATA_MODEL §Migrations.)
  • IPC pipe: framing, split reads, oversize rejection, unknown fields/types ignored, protocol mismatch.

Integration tests (CI-runnable, no game, no anti-cheat surface)

hook-harness is what makes this architecture testable without touching a real title:

  • Harness presents 120 s at a controlled cadence → Overlay injected → ring drained → session written → assert every aggregate against expected values.
  • Harness simulates: resolution change mid-run (segments), stub upscaler exports (upscaler detection), RT PSO + dispatch (RT flags), AS builds without dispatch (inline RayQuery path), PSO compile spikes (stutter attribution), and a fault-triggering hook path (self-disable).
  • .partial recovery: kill the Agent mid-capture, restart, assert interrupted session recovered.
  • Guard integration: harness loads a dummy DLL named like an anti-cheat module → injection refused pre-launch; loaded late → safety unhook. (A renamed harmless DLL, not real anti-cheat software.)

Hook overhead measurement (NFR-1, per release)

  1. hook-harness in a fixed workload, present-rate uncapped: 3 runs hooked vs 3 unhooked → mean present-call cost delta must be ≤ 1 µs (measured with QPC around the call site in an instrumented harness build).
  2. Real game, fixed 10-minute scene, capture ON vs OFF, 3 runs each: game's own Avg FPS delta ≤ 0.5%, Agent CPU ≤ 1% of a core, Agent RSS ≤ 150 MB, Overlay resident ≤ 8 MB.
  3. Results recorded in the release notes ("measured overhead this release: …").

Tier cross-validation (accuracy regression net)

Run the same game session with Tier 1 and Tier 2 simultaneously where possible (ETW does not conflict with hooking) and compare frametime distributions: Avg/1% Low must agree within 1%. Divergence beyond that means one of the two paths is wrong and blocks the release. This is also the check that would have caught the original detection-accuracy problem.

Manual test matrix (per release)

Axis Values
OS Win 10 22H2 VM · Win 11 dev machine
GPU vendor NVIDIA (primary) · AMD or Intel if available, else document as untested
API D3D11 · D3D12 · Vulkan (layer path) · OpenGL · a 32-bit title, asserting it is correctly refused and routed to Tier 2
Upscaler DLSS SR · DLSS-G · DLSS-RR · FSR2/3 · XeSS · none
RT DXR 1.0 dispatch title · inline RayQuery title · non-RT title
Mode launch mode · attach mode · mid-session settings change
Safety game with anti-cheat → toggle disabled · simulated late AC load → unhook · double-crash → auto-disable
Tier forced Tier 2 · Tier 1 → Tier 2 degradation notice
Update Velopack delta; update deferred while a game is hooked (FR-12)

Release smoke

Clean-VM install → Legal Gate → Agent setup (unelevated) → add a game → consent dialog → hooked 2-min session → charts render → CSV opens in Excel → Vulkan layer registered/unregistered cleanly → uninstall removes layer registration, task, and asks about data.