Files
frameledger/tests/FrameLedger.Agent.Tests/AgentEndToEndTests.cs
T
poli0981andClaude Opus 5.5 24a87159f4 fix: the owner's eight sessions of 2026-10-20 - a x2.52 state, the driver's DLSS word lost at exit, Start directly losing the game Steam starts, two summary windows, display sizes, a bare DLSS line; the Steam note; the two load-flaky tests (beta.27 PR-1) (#330)
The owner ran the installed 0.1.0-beta.26 (sessions 64-71) and asked for the results to be checked,
the two flaky tests fixed, bugs fixed, and a release if there were bugs. Read from a backup copy of
the ledger and the logs (docs/spike-notes/23-owner-runs-2026-10-20.md).

- FgWindow.IsNearWholeMultiplier: a window is a state's only within 10 % of a whole multiplier of two
  or more (FgSlices.Slice.IsActive); FgMixRecord.Parse reads stored rows by the same rule. Silent Hill
  2 under Dynamic MFG listed x2.52 for 12.8 s - windows where the multiplier stepped; it now reads
  x2.0, x3.0, none and 7.36 % not classified.
- NgxDriverState / CaptureSession: the driver is not asked after the game has gone, and CREATED |
  EVALUATE stand once a reading carried them. The last NGX reading, taken after the target exited,
  replaced every in-game one: none of the 14 answered sessions kept the two bits, so DLSS
  (driver-reported) and the NVIDIA App's mode (D63) never showed on an NGX-direct title.
- CaptureOrchestrator: a launch says it ended on its own (Recorded) with the pids it recorded
  (Targets), so a process turned away while it ran - the game Steam starts after Lies of P hands
  itself over - is recorded when it ends; the launch's own process never is.
- SessionSummaryOpener: one window per session; opening it again brings it forward.
- DisplayTally: the sizes are the mode's last sample that saw the window (a game leaving exclusive
  fullscreen closes it first; five sessions stored none).
- DlssText.Line: no line that says only "DLSS".
- Tests: the guard's scan set includes the test host above the harness, which under a full gate
  releases modules while it starts its other classes - ModuleScanFailed (reproduced 1 run in 12 of
  the assembly with coverage; 15 of 15 green after), ProcessUnreadable on 2026-10-19.
  ShmDrainIntegrationTests.InjectAsync asks again on those two reasons only (4 times at most, 250 ms
  apart); the pause case waits for READY; every injection suite waits for the harness's flushed
  "presenting" line instead of 800 ms (tests/Shared/HarnessReady.cs).
- Docs: 03_METRICS (the mix, the driver-reported rung, display sizes), 04_CAPTURE, 06_DATA_MODEL,
  08_UI, 14_TESTING, 18_GPU_VENDOR_APIS, 19_SAFETY (direct starts counted by the crash safety -
  measured, the policy unchanged), LIMITATIONS, guide (troubleshooting, results), spike-notes section
  23, HANDOFF items 20 and 21, CHANGELOG.

Then, on the owner's word about Steam games (Play > Start directly says a Steam game may hand itself to Steam - the notice, the tooltip and Settings (en/vi/ja); why a note rather than a fixed Steam rule (beta.27 PR-1)):

The owner, 2026-10-20: Steam games hand themselves to Steam; a note in the UI, or a fixed setting
that Steam's games always go through the Steam client - the choice left open. The note:

- GamePlayer: PlayResult.SteamGame - the game the agent starts (Start directly, or an App running as
  administrator, or a Steam record with no link) is one Steam's own records name; the notice then
  reads Play_Started_Steam_Format / Play_Elevated_Steam_Format. Store records are read once a press.
- Strings: Play_ToolTip_Direct and Settings_PlayMode_Body say a Steam game may hand itself to Steam
  and the game Steam then starts is recorded; Settings adds that a few will not start outside Steam
  at all. en/vi/ja, ja marked review; Strings.Designer regenerated.
- Why not the fixed rule: three of the five Steam games started directly on 2026-10-20 ran and were
  measured from their first frame, and a Vulkan game is measured only when FrameLedger starts it
  (Everhex Demo) - recorded in 08_UI.
- Docs: 08_UI, guide/08-settings, LIMITATIONS, CHANGELOG.
- Tests: GamePlayerTests (Steam started directly and elevated say it; a loose or GOG game does not).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 10:31:47 +07:00

312 lines
15 KiB
C#

// SPDX-License-Identifier: GPL-3.0-only
// Copyright (C) 2026 poli0981 - additional terms under GPLv3 section 7: see NOTICE
using System.Diagnostics;
using System.IO.MemoryMappedFiles;
using FluentAssertions;
using FrameLedger.Application.Capture;
using FrameLedger.Application.Consent;
using FrameLedger.Application.Persistence;
using FrameLedger.Domain.Consent;
using FrameLedger.Domain.Sessions;
using FrameLedger.Infrastructure.Persistence;
using FrameLedger.Infrastructure.Settings;
using FrameLedger.Testing;
namespace FrameLedger.Agent.Tests;
/// <summary>
/// The shipped Agent as a separate process (P2 PR-F): <c>--console capture</c> against our own harness, with a
/// scratch ledger under <c>--data-dir</c>, ends as ONE <c>sessions</c> row — the milestone's technical half,
/// from outside the process.
/// </summary>
/// <remarks>
/// <para>
/// <b>hook-harness is the only thing this suite ever injects into</b>, and
/// <c>TheTargetAndTheConsentRecordAreBothOurOwnHarness</c> asserts that on the suite itself, exactly as the
/// capture host's suite does. The consent record is written through the real store rather than through
/// <c>consent grant</c>, which refuses redirected stdin by design; the test is standing in for a human, so the
/// self-constraint pins WHICH executable it may stand in for.
/// </para>
/// <para>
/// <b>The scratch ledger is this test's, never the profile's.</b> <c>--data-dir</c> is the whole reason decision
/// D6 exists; every Agent this class starts is given one, and the directory is deleted with the test.
/// </para>
/// </remarks>
// ONE COLLECTION FOR EVERY CLASS THAT STARTS hook-harness, so they never run at the same time. xUnit
// parallelises across classes but never within a collection, and two harnesses of the same image are exactly
// the ambiguity TargetResolver refuses on ("2 processes share the image and none can be singled out") —
// measured 2026-09-10, when this class and its sibling first ran concurrently. Refusing was correct; the
// suites were wrong to be concurrent.
[Trait("Category", "Integration")]
[Collection("agent-harness")]
public sealed class AgentEndToEndTests : IDisposable
{
private static string Harness => Path.Combine(AppContext.BaseDirectory, "hook-harness.exe");
private static string Agent => Path.Combine(AppContext.BaseDirectory, "FrameLedger.Agent.exe");
private readonly string _dataDir = Path.Combine(Path.GetTempPath(), "fl-agent-e2e-" + Guid.NewGuid().ToString("N"));
/// <summary>
/// The ONE executable this suite may write a consent record for: our harness, copied under a run-unique name
/// (beta.16). The Agent resolves its target BY NAME (<c>TargetResolver</c>), and <c>build.ps1</c> runs the test projects as
/// parallel processes: a <c>hook-harness.exe</c> another project was still starting - its module list a partial copy -
/// is an unreadable process of the same name, which the resolver must count, and it refused <c>TargetAmbiguous</c>
/// (#269's first CI run). No other test starts a process of this name.
/// </summary>
private string ConsentedExecutable => Path.Combine(_dataDir, "hook-harness-" + Path.GetFileName(_dataDir) + ".exe");
private string Ledger => Path.Combine(_dataDir, LedgerPaths.DatabaseFileName);
public void Dispose()
{
try
{
Directory.Delete(_dataDir, recursive: true);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
}
}
[Fact]
public void TheTargetAndTheConsentRecordAreBothOurOwnHarness()
{
StageTarget();
Path.GetFileName(Harness).Should().Be("hook-harness.exe");
Path.GetDirectoryName(Harness).Should().Be(AppContext.BaseDirectory.TrimEnd(Path.DirectorySeparatorChar));
Path.GetFileName(ConsentedExecutable).Should().StartWith("hook-harness-");
Path.GetDirectoryName(ConsentedExecutable).Should().Be(_dataDir, "the copy lives in this run's scratch directory and nowhere else");
File.ReadAllBytes(ConsentedExecutable).Should().Equal(File.ReadAllBytes(Harness), "the target IS hook-harness, byte for byte, under another name");
Path.GetFileName(Agent).Should().Be("FrameLedger.Agent.exe");
}
/// <summary>This run's copy of hook-harness, once per test; its overlay logs are this assembly's to remove (<c>17_HOOK_ENGINE</c> §Native logging).</summary>
private void StageTarget()
{
if (File.Exists(ConsentedExecutable))
{
return;
}
File.Exists(Harness).Should().BeTrue(
"hook-harness.exe must be staged beside the test binary (FrameLedger.DrainFixtures.targets). "
+ "This FAILS rather than skipping: an integration test that quietly does nothing when its fixture "
+ "is absent is a gate that cannot fail.");
Directory.CreateDirectory(_dataDir);
HarnessOverlayLogSweep.Own(_dataDir);
File.Copy(Harness, ConsentedExecutable);
}
private async Task<ConsentWriteOutcome> GrantConsentAsync()
{
StageTarget();
ExecutableFingerprint fingerprint = FrameLedger.Infrastructure.Io.ExecutableIdentity.Read(ConsentedExecutable)!.Value;
Directory.CreateDirectory(_dataDir);
LedgerDatabase db = await LedgerDatabase.OpenAsync(Ledger, ct: TestContext.Current.CancellationToken).ConfigureAwait(false);
await using (db.ConfigureAwait(false))
{
return await new SqliteGameConsentStore(db).RecordOperatorAcknowledgementAsync(new OperatorAcknowledgement
{
Fingerprint = fingerprint,
DisclosureVersion = OperatorDisclosure.AgentConsoleVersion,
AcknowledgedAt = DateTimeOffset.UtcNow,
Provenance = ConsentProvenance.AgentConsoleOperator,
}, TestContext.Current.CancellationToken).ConfigureAwait(false);
}
}
private async Task SetKillSwitchAsync(bool engaged)
{
Directory.CreateDirectory(_dataDir);
LedgerDatabase db = await LedgerDatabase.OpenAsync(Ledger, ct: TestContext.Current.CancellationToken).ConfigureAwait(false);
await using (db.ConfigureAwait(false))
{
await new SettingsKillSwitch(new SqliteSettingsStore(db)).SetAsync(engaged, TestContext.Current.CancellationToken).ConfigureAwait(false);
}
}
private async Task<Process> StartHarnessAsync(string arguments)
{
StageTarget();
File.Exists(Agent).Should().BeTrue("the Agent must be built and copied beside this test");
var p = Process.Start(new ProcessStartInfo(ConsentedExecutable, arguments) { UseShellExecute = false, CreateNoWindow = true, RedirectStandardOutput = true })!;
bool presenting = await HarnessReady.WaitUntilPresentingAsync(p, TestContext.Current.CancellationToken).ConfigureAwait(false);
if (!presenting && !p.HasExited)
{
p.Kill(entireProcessTree: true);
}
presenting.Should().BeTrue("the harness must reach its present loop before anything looks for it or is injected into it (HarnessReady)");
p.HasExited.Should().BeFalse("the harness must be running before the Agent looks for it");
return p;
}
private Process StartAgent(params string[] consoleArgs)
{
var psi = new ProcessStartInfo(Agent) { UseShellExecute = false, CreateNoWindow = true, RedirectStandardOutput = true, RedirectStandardError = true };
psi.ArgumentList.Add("--console");
psi.ArgumentList.Add("--data-dir");
psi.ArgumentList.Add(_dataDir);
foreach (string a in consoleArgs)
{
psi.ArgumentList.Add(a);
}
return Process.Start(psi)!;
}
private static async Task<(string Stdout, int Exit)> FinishAsync(Process p)
{
Task<string> err = p.StandardError.ReadToEndAsync(TestContext.Current.CancellationToken);
string stdout = await p.StandardOutput.ReadToEndAsync(TestContext.Current.CancellationToken).ConfigureAwait(false);
await p.WaitForExitAsync(TestContext.Current.CancellationToken).ConfigureAwait(false);
return (stdout + "\n---- stderr ----\n" + await err.ConfigureAwait(false) + $"\n---- exit {p.ExitCode} ----\n", p.ExitCode);
}
private static void Kill(Process p)
{
try
{
if (!p.HasExited)
{
p.Kill(entireProcessTree: true);
}
}
catch (InvalidOperationException)
{
}
p.Dispose();
}
[Fact]
public async Task AConsentedBoundedCaptureLandsAsOneHookedRowInTheScratchLedger()
{
// THE MILESTONE'S TECHNICAL HALF, from outside the process: the shipped Agent, our harness, one row.
(await GrantConsentAsync()).Should().Be(ConsentWriteOutcome.Written);
Process harness = await StartHarnessAsync("--real --hold-presenting 40");
try
{
using Process agent = StartAgent("capture", "--exe", ConsentedExecutable, "--seconds", "8");
(string output, int exit) = await FinishAsync(agent);
exit.Should().Be(0, output);
output.Should().Contain("ledger: session ").And.Contain("SAVED as sessions.id=", output);
output.Should().Contain("telemetry=l1", output);
LedgerDatabase db = await LedgerDatabase.OpenAsync(Ledger, ct: TestContext.Current.CancellationToken).ConfigureAwait(true);
await using (db.ConfigureAwait(true))
{
var sessions = new SqliteSessionRepository(db);
SessionRow row = (await sessions.ListRecentAsync(1, TestContext.Current.CancellationToken).ConfigureAwait(true)).Should().ContainSingle().Subject;
row.Tier.Should().Be(CaptureTier.Hooked);
row.Mode.Should().Be(CaptureMode.Attach);
row.ExitStatus.Should().Be(ExitStatus.Normal, "the harness was still presenting when the bounded capture ended");
row.FrameCount.Should().BeGreaterThan(100, "8 s at ~120/s");
row.Api.Should().Be("d3d11");
row.GuardTicksPublished.Should().BeGreaterThanOrEqualTo(1);
row.OverlayBuildId.Should().NotBeNullOrEmpty();
row.TelemetrySource.Should().Contain("l1", "the Agent composes L1 + L2 + L3 for every session");
FrameBlobs? frames = await sessions.FindFramesAsync(row.Id, TestContext.Current.CancellationToken).ConfigureAwait(true);
frames.Should().NotBeNull();
frames!.SampleCount.Should().Be(row.FrameCount);
}
Directory.EnumerateFiles(Path.Combine(_dataDir, "tmp"), "*.partial").Should().BeEmpty("finalize deletes the .partial");
}
finally
{
Kill(harness);
}
}
[Fact]
public async Task WithTheKillSwitchOnAConsentedGameIsRefusedAndNothingIsEverInjected()
{
// FR-2.4 from outside the process (decision D7): consent is present and enabled, the switch is on, the
// gate refuses with its own reason, and no ring ever exists in the target.
(await GrantConsentAsync()).Should().Be(ConsentWriteOutcome.Written);
await SetKillSwitchAsync(engaged: true);
Process harness = await StartHarnessAsync("--real --hold-presenting 20");
try
{
using Process agent = StartAgent("capture", "--exe", ConsentedExecutable, "--seconds", "3");
(string output, int exit) = await FinishAsync(agent);
output.Should().Contain(nameof(SessionEndReason.RefusedKillSwitch), output);
exit.Should().NotBe(0, output);
harness.HasExited.Should().BeFalse();
Action open = () => MemoryMappedFile.OpenExisting($@"Local\FrameLedger.Ring.{harness.Id}", MemoryMappedFileRights.Read).Dispose();
open.Should().Throw<FileNotFoundException>("no ring means no Overlay means no injection");
}
finally
{
Kill(harness);
}
}
/// <summary>
/// beta.16: the collision #269's first CI run hit. Another hook-harness.exe running beside this suite's run - another
/// test project's, or one left over - shares neither the copy's name nor its path, so the Agent's resolver never counts
/// it. Until beta.16 this suite consented to the staged hook-harness.exe itself, and a second one was a second match:
/// TargetAmbiguous, exit 6.
/// </summary>
[Fact]
public async Task AnotherHookHarnessRunningBesideTheSuiteDoesNotMakeTheTargetAmbiguous()
{
(await GrantConsentAsync()).Should().Be(ConsentWriteOutcome.Written);
Process stranger = Process.Start(new ProcessStartInfo(Harness, "--real --hold-presenting 30") { UseShellExecute = false, CreateNoWindow = true, RedirectStandardOutput = true })!;
Process harness = await StartHarnessAsync("--real --hold-presenting 30");
try
{
using Process agent = StartAgent("capture", "--exe", ConsentedExecutable, "--seconds", "3");
(string output, int exit) = await FinishAsync(agent);
exit.Should().Be(0, output);
output.Should().Contain("SAVED as sessions.id=", output);
output.Should().NotContain("TargetAmbiguous", output);
}
finally
{
Kill(harness);
Kill(stranger);
}
}
[Fact]
public async Task ConsentGrantRefusesRedirectedStdinAndWritesNothing()
{
StageTarget();
using Process agent = StartAgent("consent", "grant", "--exe", ConsentedExecutable);
(string output, int exit) = await FinishAsync(agent);
exit.Should().Be(2, output);
output.Should().Contain("stdin is redirected", output);
output.Should().Contain("OPERATOR SURFACE", "the Agent's disclosure names its own surface first");
using Process list = StartAgent("consent", "list");
(string listed, int listExit) = await FinishAsync(list);
listExit.Should().Be(0, listed);
listed.Should().Contain("nothing is enabled", listed);
}
[Fact]
public async Task TheAppsDiagFlagAnswersNotThisBinarysWithExitTwo()
{
// The four maintenance flags are real since P3 PR-8b and run against the PRODUCT directory (a registration,
// a scheduled task), so an end-to-end test never invokes them; --diag is the App's and still exits 2 here.
var psi = new ProcessStartInfo(Agent) { UseShellExecute = false, CreateNoWindow = true, RedirectStandardOutput = true, RedirectStandardError = true };
psi.ArgumentList.Add("--diag");
using Process agent = Process.Start(psi)!;
(string output, int exit) = await FinishAsync(agent);
exit.Should().Be(2, output);
output.Should().Contain("not this binary", output);
}
}