Skip to content

test: derive load-sensitive harness budgets from named constants - #419

Draft
roodboi wants to merge 15 commits into
nextfrom
wf/g52-load-sensitive-test-budgets
Draft

roodboi wants to merge 15 commits into
nextfrom
wf/g52-load-sensitive-test-budgets

Conversation

@roodboi

@roodboi roodboi commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Refs HACK-1367 (unit G52). Related: HACK-1458, which owns the product clocks this PR leaves alone. Test-only change: no product code, product default or product bound changes. The branch also merges next (latest: 191382d).

Round 3 summary. The table under "Tests and evidence" records round 1 at 7f76651. Rounds 2 and 3 replace parts of it; where they disagree, the later round wins. See "Round 3" at the end.

Root cause

Each case below passes alone but failed under host load (M5 at load 15-40, or higher; M3 while another suite runs), because a fixed harness budget was spent on fixture work whose wall time follows host load, or because the harness raced a kill:

  1. runtime-core native runtime tests (provider::graph::native::runtime::tests, for example two_owned_bridges_keep_per_network_members_and_aliases_through_cleanup and two_bridge_cleanup_retries_only_the_retained_second_bridge_after_first_delete). Fake-backend execution::run calls had 1-5 s budgets. The fake is ready at once, but on a loaded host the journal writes took longer than 2 s, so the run failed with graph_timeout.
  2. tests/native-compose-adoption-generation.test.ts. Every engine probe spawns the Bun stub engine, and one retained dispatch issues several hundred probes. On the loaded M5 one start took 18-24 s, which is past the 20 s caller deadlines and the 20-30 s case timeouts. Baseline at origin/next on the loaded M5: 8 failures (v3, v5 and v7 retained dispatch cases) before I stopped the run, all E_LEGACY_ADOPTION_DEADLINE or case timeouts.
  3. tests/e2e/native-process-policy-initial-replay.ts. Each replay child cold-starts Bun, transpiles the ownership policy and spawns one stub engine per recorded reply. Its default was 5 s, so under load the child timed out and the harness threw Initial process-policy replay unavailable.
  4. tests/native-authored-project-start.test.ts.
    • The named stage runtime failure, and the module-level unconfirmed latch that then failed every later case, were already fixed on next by test(native): isolate startup settlement guard per case #388 (HACK-1299). It added per-case guards (tests/helpers/native-fixture-cases.ts), so a failed case reports its own failure once, and it replaced the owner's fixed 30 ms ready window with a release handshake.
    • A loaded M5 run on this branch found a second harness race. In cancel during the runtime status check refuses before the startup intent, the stand-in status request appended its PID with appendFile, and the case cancelled as soon as the request appeared in the call log. The product's SIGKILL could land after the PID file was created but before the PID was written. Settlement reads an empty PID file as a write still in progress, so it reported settlement unconfirmed after 15 s even though the case's assertions passed. The retained root had an empty status.pid.

The witnessed enrollment Fixture::new temp-dir collision (persistent_data/enrollment/witnessed/tests.rs) is fixed by open draft #399 (G40/HACK-1348, next_test_scratch()). This PR does not duplicate it.

Fix

  • New tests/helpers/test-budget.ts holds one scale, TEST_BUDGET_SCALE = 4, and testBudgetMs(unloadedMs). Each call site names the time its fixture work needs on an unloaded host, and the helper scales it.
  • Adoption generation: case timeouts, and the caller deadlines of cases that do not test the deadline (OPERATION_DEADLINE_MS), now come from testBudgetMs. The product bounds under test stay literal: the 5 s deadline-expiry cases, the deadline + 1500 promptness check, and the 4 s cancellation reap.
  • Replay harness: the child budget (REPLAY_CHILD_BUDGET_MS), the summary budget, and the replay cases' timeouts in tests/native-process-policy-initial-trace.test.ts now come from testBudgetMs. The ownership policy's own probe bounds still run unchanged inside the child.
  • runtime-core: one RUN_BUDGET (120 s, under the 600 s product cap of execution::run) for every fake-backend run, selection, exec, run and log read in runtime/{tests,exec_tests,run_tests}.rs. Passing calls never wait on it. The log-expiry case keeps its own 1 s deadline, because the fake sleeps past it on purpose.
  • Authored start:
    • The stand-in status request stages its PID and renames it into place, so the record is either absent or complete.
    • The case now waits for the published PID (awaitPidFile) before it cancels. It no longer waits on the call log.
    • The file's settlement, startup and stop harness budgets now derive from testBudgetMs (20 s, 60 s and 60 s). The 50 ms deadline cases and their 1 s wall bound stay literal.

Tests and evidence

  • Toolchain: Bun 1.4.2 (mise x bun@1.4.2). Rust used cargo +1.97.1 with its own CARGO_TARGET_DIR and debug info off.

  • Full suite on the M3 at the pushed head 7f76651 (bun run test): 7817 pass, 126 skip, 1 fail. The one failure is repository catalog > the checked-in .hack/tasks.json recipes are all reviewed, which refuses because the M3 CI worktree is a detached checkout (detached_checkout). It is unrelated to this change: the same failure is in the G36, G41, G49 and G51 M3 logs, and the test passes locally on a branch checkout. See the finding in the unit report.

  • Loaded repetitions on the M3 at 7f76651 (round 1, TEST_BUDGET_SCALE = 4). Superseded for the authored start and the adoption retained dispatch. Round 2 could not repeat these 10/10 results on the M5 under heavier concurrent load, and its "Not met" section stands. The runtime-core and replay rows still hold. Load came from 16 spin-and-spawn shell workers on the 16-core host, stopped in a shell trap. The 1-minute load average during the runs is shown.

    Target Command Consecutive passes 1-min load during runs Per run
    runtime-core runtime tests cargo +1.97.1 test --features native-config-plan --lib -- provider::graph::native::runtime::tests 10/10 (51 pass, 1 ignored each) 47-79 6-9 s
    Process-policy replay bun test tests/native-process-policy-initial-trace.test.ts 10/10 (10 pass each) 67-92 19-21 s
    Authored start bun test tests/native-authored-project-start.test.ts (whole file) 10/10 (52 pass each) 74-159 167-186 s
    Adoption generation, retained dispatch bun test tests/native-compose-adoption-generation.test.ts -t '^(v[0-9]+ |rolled-back v|public |retained command dispatch)' 10/10 (55 pass each) 107-181 1172-1268 s

    The baseline (the 616407a5 versions of the Rust and replay tests, under the same churn) passed 3/3 for both targets on the M3. This churn did not reproduce those two failures on the M3, so the before/after contrast for them comes from the M5 runs below.

  • Local runs (M5, already loaded, no added churn):

    • Adoption generation: 115 pass, 0 fail (948 s); at origin/next the same host failed 8 cases.
    • Replay trace: 10 consecutive passes at load 64-176.
    • runtime-core runtime tests: 10 consecutive passes at load 76-153.
    • Authored start, before the status-PID fix: 2 of 4 runs failed at load 80-138. One was the status-PID race. The other is the product diagnostic finding below.
    • Authored start, after the fix: 52 pass, 0 fail.
  • Mutations:

    • With only RUN_BUDGET set to 1 ns, both named Rust tests fail with graph_timeout at their execution::run line.
    • Status-PID race, three runs on the authored case:
      • (A) Base harness, with the stand-in pausing 1.5 s between creating and writing its PID record: fails with settlement unconfirmed ... stand-in hanging runtime status request. This reproduces the loaded failure deterministically.
      • (B) Fixed harness, with the same 1.5 s pause between staging and publishing: passes.
      • (D) Fixed harness, with the published PID belonging to a process that survives the cancel: fails with settlement unconfirmed, so the case still detects a surviving request.
    • All mutations were restored afterwards.
  • Gates: root bun run check (includes privacy:check) and bun run --cwd packages/cli typecheck passed with 0 errors. Biome found nothing on the changed test files. cargo +1.97.1 fmt --check and clippy -D warnings passed for the Rust change. gitleaks 8.30.1 over origin/next..HEAD found no leaks.

Gaps

  • The 5 s deadline-expiry adoption cases still check product bounds against the wall clock. Fixed in round 3: these cases now move Date.now instead of waiting.
  • A static7 case that times out latches publicStatic7Settlement, and every later case then fails. Fixed in round 3 by the per-case drain guard. The file still shares module-level fixture globals.
  • The authored stand-in owner still appends its PID with appendFile. A SIGTERM that lands before its handler is registered could leave an empty owner.pid in the same way. This was not seen in any run.
  • cancel during the runtime status check assumes the status request is still in flight when it cancels. If a Bun child takes longer than the product's 5 s status timeout to start, the case can still fail. The product has no seam for that timeout.
  • Product finding, not fixed here. On the M5 at load 120-150, startup keeps the native failure when missing receipt inspection also refuses failed once. The start reported native_graph_receipt from the follow-up inspection instead of the serve failure graph_network_intent. The serve child had written its code to stderr before exiting. The most likely cause, which I have not confirmed, is that serveGraphProcess stops reading stderr 250 ms after the child exits (src/backends/native-project-process.ts), so a reader that falls behind on a loaded host drops the code. The case passed 10/10 on the M3.
  • Loaded-run proof is from the M3 with added churn and from the loaded M5. CI runs neither.

Round 2: verifier rework (M5 under concurrent load)

The verifier found that items (2), (3) and (4) still failed on the M5 at head 7f76651. This round merges next at 6e4bcf2 and adds four commits: 1686c1d, f0352ce, fa5b36d and 073041c. It is still test-only. The Fix section above describes round 1. Where it gives TEST_BUDGET_SCALE = 4 and fixed 5 s expiry deadlines, this section replaces it.

What changed

  • One scale, now 20. TEST_BUDGET_SCALE went from 4 to 10, then to 20. These fixtures are spawn-bound: one retained adoption dispatch runs about 750 stub-engine processes. Their wall time therefore grows faster than the load average.
    • With a second suite plus churn workers (1-min load 28-91), a scale of 4 still expired the Source5 authored starts and the v7 dispatches.
    • At scale 10, one public retained up outlasted a 200 s deadline.
    • Every budget that testBudgetMs derives is a ceiling. A passing case does not wait on it.
  • Authored start (tests/native-authored-project-start.test.ts):
    • The startup deadline is min(testBudgetMs(15 s), 300 s). 300 s is the largest value serveNativeAuthoredProject accepts. The stop deadline uses the same value.
    • The planner-refusal case used a literal 10 s deadline. It now uses STARTUP_TIMEOUT_MS.
    • known failed down.after ... runs two full cycles, so it gets two case budgets. The stand-in owner's cap was raised so it outlasts that case.
  • Adoption generation (tests/native-compose-adoption-generation.test.ts):
    • Public dispatch takes its caller deadline from the Compose startup budget, which defaults to 90 s. beforeEach now injects the harness operation deadline through HACK_COMPOSE_STARTUP_TIMEOUT_MS, and afterEach restores it.
    • Stores the cases open now use a probe-phase bound of 60 s (timeoutMs, the largest the product accepts) unless the case names one. One store probe carries a single 15 s clock across a whole resource inspection, and under load it expired as E_NATIVE_COMPOSE_PROBE_TIMEOUT, which surfaces as E_LEGACY_ADOPTION_STATE.
    • The two expiry cases used a 5 s caller deadline, which expired before the callback was reached at load 30-45. They now use 45 s. That is still under the 60 s phase bound, so the caller deadline, not a probe's own clock, ends the held probe. The 1.5 s promptness assertion is unchanged.
  • Replay (tests/native-process-policy-initial-trace.test.ts, tests/e2e/native-process-policy-initial-replay.ts):
    • Cases that name no time now get Bun's 5 s default scaled by testBudgetMs. Lines 258, 302 and 749 had kept an unscaled 5 s.
    • The multicall and UTF-8 cases used a literal 3 s for exec and probe. They now use the forwarded-exec bound.
    • The replayed ownership policy runs with its largest probe bound (60 s), unless the recorded selection names one.

Mutations (each restored afterwards)

  • Replay wrapper default. The UTF-8 case first sleeps 6 s. With the new wrapper it passes. With the round-1 wrapper it fails with timed out after 5000ms.
  • Dispatch deadline injection. With HACK_COMPOSE_STARTUP_TIMEOUT_MS injected as 1000, public retained up ... "no" fails 5.5 s after start, with E_CONFIG_INVALID. This shows the injected value reaches the CLI's deadline.
  • Store probe-phase bound. With PROBE_PHASE_BUDGET_MS = 1, v7 start preserves ... fails in 1.2 s with E_LEGACY_ADOPTION_STATE.
  • Replay probe bound. With REPLAY_PROBE_BUDGET_MS = 1, HACK-1279 case 1 replays E_NATIVE_COMPOSE_PROBE_TIMEOUT instead of code: null.

Evidence (M5, Bun 1.4.2, private HOME, HACK_HOME and DOCKER_CONFIG, unreachable DOCKER_HOST, keychain fallback disabled)

Load came from bash spin and /bin/sh spawn workers, stopped in an EXIT/INT/TERM/HUP trap. Other agents on the host added their own load, at 1-min load 20-140 even with none of mine running. Loop logs were kept outside the checkout. The checkout-write guard from #405 fails a whole file when any new file appears in the checkout during the run.

Target Load setup Result
Replay trace file, at 073041c 36 workers 10/10 consecutive (10 pass each), 1-min load 11-164, 234-420 s per run
Replay trace file, at fa5b36d (before the replay probe bound) 36 workers, load 54-122 4 passes, then run 5 failed HACK-1279 case 1 with E_NATIVE_COMPOSE_PROBE_TIMEOUT inside the replay child
Authored file, scale 10 (1686c1d) 8 workers plus the adoption subset running at the same time, load 44-91 52/52 in one full run (1463 s)
Authored file, scale 20 (fa5b36d) same, load 23-138 run 1: 50/52; run 2: 48/52; run 3: at least 5 failures before I stopped it
Adoption retained-dispatch subset, fa5b36d same run 1 had 6 failures before I stopped it

Not met: 10 consecutive loaded passes for items (2) and (4). None of the remaining failures is a harness budget expiring. They come from product bounds that a test cannot set, and from one hang:

  • Authored:
    • E_COMPILER_TIMEOUT at stage runtime or hook-up-before. Each compiler invocation has a fixed 10 s default (DEFAULT_TIMEOUT_MS in src/lib/native-config-compiler.ts). The authored start passes no compiler timeout, even when its own deadline has minutes left.
    • The product promptness bound under test: a 50 ms deadline must return within 1 s. It returned in 2.6 s.
    • In one run, Source5 cancellation after readiness ... hung for its whole 600 s budget. See the findings below.
  • Adoption:
    • Public dispatch opens its own store without timeoutMs, so its 15 s probe-phase clock stays fixed. It expires under load and is reported as E_CONFIG_INVALID ("generation state is invalid, unsafe or changed").
    • After a public case fails, later cases' store open/prepare refuse with E_LEGACY_ADOPTION_STATE within 4-5 s. translate() hides the original error, so I did not find the root cause.

Gates

  • Root bun run check (includes privacy:check): passed at every pushed head.
  • bun run --cwd packages/cli typecheck: 0 errors at every pushed head.
  • Biome: nothing found on the changed files.
  • gitleaks 8.30.1 dir, over a git log -p origin/next..HEAD export: no leaks.
  • The CLI surface and agent guidance did not change, so docs:cli-reference and plugin regeneration do not apply.

Hosted CI

At 073041c, run 40115797226 (pull_request, its headSha matches) passed every check. That includes test shards 1-4, docker-e2e shards 1-4, Candidate core (macOS and ubuntu) and secret-scan. This is the full-suite evidence. It ran against the PR merge ref with next as of that time.

At fa5b36d, test shard 3/4 failed in tests/project-env-layer-lock.test.ts > concurrent legacy dotenv sets and removals of different keys all apply, with a private-file "not a regular file" refusal. That test is not in this diff and not on the known-flake list, so I did not rerun it. It passed at the next head.

Round 2 findings (not fixed here)

  1. Public adoption dispatch has a fixed 15 s inspection clock. tryLegacyComposeAdoptedCommand opens its store without timeoutMs, so each inspection phase gets ctx.timeoutMs ?? 15_000 across dozens of engine calls. HACK_COMPOSE_STARTUP_TIMEOUT_MS does not affect it. A slow engine therefore turns hack up on an adopted project into E_CONFIG_INVALID "state is invalid, unsafe or changed", not a timeout code. Medium.
  2. The authored start's compiler calls ignore the remaining startup deadline. Each call is bounded only by the compiler's 10 s default, so a loaded host fails starts with E_COMPILER_TIMEOUT while startupTimeoutMs still has time left. Medium-low.
  3. Source5 cancellation after readiness can hang.
    • What happened: 8 churn workers plus the adoption subset, load 23-60. After onReady aborted, the runtime issued graph native control --action status twice. It never terminated the frontend-serve owner and never ran the down hooks. The owner and the persistent host process group were still alive 13 minutes later.
    • Expected: cancellation settles the host processes and retires the hooks, or refuses within its budget.
    • Medium-high: possibly a product hang in cancellation after publish.
  4. Failed authored cases leave lifecycle process groups running. The persistent host process and the mux bun index.ts stay up after the run. Settlement reports them as unconfirmed but does not end them. I ended the groups that my runs left, after confirming ownership from my temp roots. Medium.

Round 3: verifier rework (items 1-4)

This round merges next (7dac4a0, cd35034, then 191382d) and adds four commits: 7fbeb50, 5afe58a, 6e3ec5e and b18202c. It is still test-only. Where this section disagrees with rounds 1 and 2 above, this section wins.

Scope. The product clocks stay out of scope here. They are tracked in HACK-1458: the fixed 15 s public dispatch inspection clock (E_CONFIG_INVALID), the authored compiler's fixed 10 s timeout, the Source5 cancellation hang, and lifecycle process groups left after failed authored cases. Because of them, HACK-1367's "10 consecutive loaded passes" for the adoption and authored files is still not met, and this PR does not claim it.

What changed

  1. Witnessed enrollment deadlines (item 1).
    • The runtime tests' 120 s budget moved to a crate-level TEST_RUN_BUDGET in packages/runtime-core/src/test_budget.rs. It is gated on test and native-config-plan, the only feature its users build under.
    • persistent_data/enrollment/witnessed/tests.rs (lines 196 and 206) now uses it in place of a literal 5 s. So do the sibling enrollment/tests.rs (literal 10 s) and engine/tests.rs (literal 5 s).
    • runtime/tests.rs keeps its local RUN_BUDGET name, now set to that constant.
  2. No more file latch in the adoption file (item 2).
    • publicStatic7Settlement, publicStatic7Commands and the publicStatic7Case wrapper are gone.
    • Every case now runs under createCaseDrain (tests/helpers/case-drain.ts). Each case body has its own budget, and an overrun fails that case once, by name, with E_TEST_CASE_BUDGET.
    • The overrunning body then gets a drain of 85 s (the 80 s operation deadline plus 5 s) inside the same case's Bun timeout. If it settles, the next case starts normally.
    • If the body is still running, it keeps the file's roots and environment. The next case's beforeEach waits for it for up to one more case ceiling (240 s). This wait runs outside that case's own timeout, and the case starts normally once the body settles.
    • Only a body that outlasts the budget, the drain and that wait makes later cases refuse with E_TEST_CASE_RESERVED, which names the case still running. They refuse immediately, without waiting again, until it settles.
    • Each case restores the environment as it was when the file loaded, not a per-case "prior" value that a late body could have left behind.
  3. Budgets stay inside the hosted per-file limit (item 3).
    • testBudgetMs now stops at CASE_TIMEOUT_CEILING_MS. That is a third of scripts/test-shard.ts's DEFAULT_FILE_TIMEOUT_SECONDS (720 s), so 240 s.
    • withinCaseCeilingMs checks each file's composed case timeout when the file loads.
    • Adoption file:
      • Body budgets are at most 150 s, so 150 + 85 + 5 = 240 s.
      • OPERATION_DEADLINE_MS is 80 s, down from 400 s.
      • The 60/90/240 s "unloaded" literals all clamp to the 150 s body ceiling.
    • Authored file:
      • The case budget is 240 - 30 (settlement) - 5 = 205 s, down from 600 s. The two-cycle case uses the same budget, down from 1200 s.
      • Startup and stop deadlines are min(300 s, 102 s), down from 300 s.
      • The stand-in owner cap and the hanging stand-in status request are both 240 s, down from 1800 s and 1200 s. Neither outlives the 720 s file any more.
    • Replay/trace cases clamp to 240 s, down from 600 s and 1200 s.
    • The trade-off: above the ceiling, the scale no longer gives headroom, so a host loaded past it fails cases, by name. That is the point of item 3, and the evidence below shows it.
  4. The expiry cases use a controllable clock (item 4).
    • The product has no clock seam. It reads the caller deadline as a Date.now timestamp and derives each probe's timeout from the time left. Bun's setSystemTime freezes Date.now, so the product's Date.now() >= deadline check after a killed probe would never pass. I tried it and did not use it.
    • Instead, shiftProductClock spies on Date.now for the rest of the case. It moves the clock forward and lets it keep advancing at the real rate. It is restored in finally.
    • Once the callback runs:
      • v5 expired aggregate deadline after callback moves the clock 5 ms past the deadline instead of sleeping until then.
      • The two v5 remaining aggregate clock ... cases move the clock to a lead before the deadline: 5 s for per-effect freshness and 20 s for the final observation. The held probe still ends by the timeout the product derives from the time left. The 1.5 s promptness assertion, the PID-killed check and pending retention are unchanged.
    • The leads come from measurements on the M5 at 1-min load 45, from the callback to the held stub recording its PID: 0.3 s for per-effect freshness and 5.7 s for the final observation (it first runs other final probes).
    • Each shift is max(0, ...), so the clock never moves back.
    • The v3 linked saved proof case (49 s on hosted CI) waits on no clock. It is compiler and probe work, and it is unchanged.

New tests and mutations (each restored afterwards)

  • tests/case-drain.test.ts (8 cases) and tests/test-budget.test.ts (4 cases). Mutations run against copies of the helper:

    • Never clearing the reservation fails 4 cases.
    • Dropping the refusal fails 1 case.
    • Skipping the drain fails 1 case.
    • Skipping the next-case wait fails 2 cases.
    • Waiting again after the wait has already run out fails 1 case.
    • One mutation passes: always reserving, even after a settled drain. The reservation then clears in the same microtask turn, so this mutation does not change behavior.
  • Rust: with TEST_RUN_BUDGET set to 1 ns, 16 of 44 persistent_data tests fail, including 6 of the 7 in enrollment::witnessed::tests.

  • Injected overrun in the adoption file, like the verifier's reproduction. A case with a 2 s budget was placed before the first real case, then the file ran with a filter for it and three fast cases. The host was at 1-min load 185-215, from other agents plus these three runs in parallel.

    Injected body Result
    Ends after 5 s (inside the drain) Injected case fails once (E_TEST_CASE_BUDGET ... settled within the 85000 ms drain). The two light real cases pass. v3 raw managed drift failed its own 150 s budget by name at that load.
    Ends after 97 s (past budget and drain) Injected case fails once. The next case waits for it, then passes. v3 raw managed drift failed its own budget by name, as above.
    Never ends Injected case fails once. The next case waits 240 s, then refuses with E_TEST_CASE_RESERVED naming it, and the remaining cases refuse at once. This is the one case in which later cases still fail. Their reason names the case that is still running, and they start again if that case settles.

    At the original head, the verifier's single hanging case produced 0 pass and 116 fail.

Evidence (M5, Bun 1.4.2, private HOME, HACK_HOME and DOCKER_CONFIG, unreachable DOCKER_HOST, keychain fallback disabled)

Other agents kept the host at 1-min load 55-215 throughout, with their own churn workers. I added no churn of my own.

  • Rust, persistent_data module, cargo +1.97.1 --all-features: 10/10 consecutive runs, 44 pass each, at load 56-59. Also fmt --check and clippy -D warnings, both default and --all-features, and the runtime module (52 pass, 1 ignored).
  • Adoption expiry cases (-t aggregate, 5 cases): 5/5 pass at load 56-66, after the merge too. There is no 45 s wait any more. Hosted per-case times are in the CI section below.
  • Adoption file, whole, at load 100-120. This ran before b18202c added the next-case wait.
    • 15 cases failed their own budget, by name. 14 of their bodies settled within the drain.
    • One body (rolled-back v5 can reprepare ... directory) still had several owner operations to run. The 92 cases after it refused at once, so the reservation never got time to clear.
    • That run is what led to b18202c: the next case now waits for such a body.
    • I did not repeat the whole file at that load after the change.
  • Authored file, plus the trace and test-shard files, at load 110-120: 96 pass, 4 fail, each failure reported by name with no cascade.
    • Source5 supervised ... failed with hook-up-before.
    • Source5 cancellation after readiness ... failed with hook-up-before (E_COMPILER_CANCELLED). Both of these are compiler-side failures (HACK-1458 items 2-3).
    • The two-cycle known failed down.after ... exceeded its 205 s budget. This is the ceiling trade-off.
    • The fourth failure was the checkout write guard, because I edited files during that run.

Gates (at 0dd0d5a)

  • Root bun run check (includes privacy:check): exit 0.
  • bun run --cwd packages/cli typecheck: 0 errors.
  • Biome: nothing found on the changed files.
  • gitleaks 8.30.1 dir, over a git log -p next..HEAD export: no leaks.
  • The CLI surface and agent guidance did not change.

Hosted CI

At 0dd0d5a, run 40138406767 (pull_request, headSha 0dd0d5a) passed every check: test shards 1-4, docker-e2e shards 1-4, Candidate core (macOS and ubuntu), Config compiler (all four platforms), Runtime state models, linux-process-lifetime, runtime-images and secret-scan. This is the full-suite evidence. It ran on the PR merge ref with next at 191382d.

Per-file times from the shard logs:

File Round 2 (073041c, run 40115797226) Round 3 (0dd0d5a)
native-compose-adoption-generation.test.ts 6m57s (417 s of 720 s) 5m20s (320 s)
v5 expired aggregate deadline after callback ... 46.2 s 1.7 s
v5 remaining aggregate clock ... per-effect freshness 46.2 s 6.8 s
v5 remaining aggregate clock ... final observation 46.2 s 21.8 s
v3 linked saved proof ... (no clock wait) 49.3 s 53.3 s
native-authored-project-start.test.ts - 31.6 s
native-process-policy-initial-trace.test.ts - 2.6 s
case-drain.test.ts / test-budget.test.ts new 1.6 s / 0.5 s

Round 3 findings (not fixed here)

  1. The ceiling and loaded-host headroom conflict.
    • What happens: hosted CI runs each file under a 720 s limit, so a case cannot take more than about 240 s and still be named. At 1-min load 100+, adoption public dispatch and authored two-cycle cases need more than that.
    • Option: scripts/test-shard.ts could pass its effective --file-timeout-seconds to each file, so direct bun test runs keep the larger 20x budgets and only shard runs clamp.
    • Severity: low-medium. It decides how HACK-1367's loaded-run acceptance can be proven.
  2. A truly hung adoption body still blocks the rest of its file, after one 240 s wait.
    • Why: the cases share module globals and process.env, so later cases cannot safely reuse them while that body may still resume.
    • Option: per-case state via AsyncLocalStorage, as native-fixture-cases.ts does. That is a rewrite of a 5,000-line file.
    • Severity: low, since it takes a body that never settles.

🤖 Generated with Claude Code

Fixed harness budgets failed under host load (M5 at load 15-40, M3 while
another suite runs) although each case passes alone:

- runtime-core native runtime tests gave fake-backend graph runs 1-5 s;
  journal writes on a loaded host exceeded that with graph_timeout. They now
  take RUN_BUDGET (120 s, under the product's 600 s cap). Passing runs never
  wait on it. The log-expiry case keeps its own 1 s deadline.
- native-compose-adoption-generation cases spawn the Bun stub engine for
  several hundred probes per retained dispatch; 20 s caller deadlines and
  20-30 s case timeouts expired. Case timeouts and non-deadline caller
  deadlines now derive from testBudgetMs. The 5 s expiry cases and their
  promptness assertions are product bounds and stay literal.
- the process-policy replay child defaulted to 5 s and refused replays
  ("Initial process-policy replay unavailable"); its child and summary
  budgets, and the replay cases' timeouts, now derive from testBudgetMs.

tests/helpers/test-budget.ts holds the one scale (TEST_BUDGET_SCALE).

Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 22, 2026

Copy link
Copy Markdown

HACK-1367

roodboi and others added 7 commits October 22, 2026 16:59
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hored budgets

Under host load, "cancel during the runtime status check refuses before the
startup intent" failed settlement after its assertions passed. The stand-in
status request appended its PID with appendFile, and the case cancelled as
soon as the request was logged. The product's SIGKILL could land after the
file was created but before the PID was written. Settlement reads an empty
PID file as a write still in progress, so it waited out its budget.

- The stand-in now stages its PID and renames it into place, so the record
  is either absent or complete.
- The case waits for the published PID (awaitPidFile) before it cancels,
  instead of the call log.
- The file's settlement, startup and stop harness budgets now derive from
  testBudgetMs. The 50 ms deadline cases and their wall bound stay literal.

Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ch and replay cases

Raise TEST_BUDGET_SCALE from 4 to 10. With a second suite plus churn
workers on the M5 (1-min load 28-66), Source5 authored starts expired the
60 s startup deadline and v7 retained dispatches expired their 80 s
caller deadline; a v7 dispatch case took 146 s at load 22-50.

- Adoption: public dispatch takes its caller deadline from the Compose
  startup budget (90 s default). Inject the harness operation deadline
  through HACK_COMPOSE_STARTUP_TIMEOUT_MS in beforeEach and restore it.
- Authored: the planner-refusal case uses STARTUP_TIMEOUT_MS instead of a
  literal 10 s; the two-cycle down.after case gets two case budgets, and
  the stand-in owner cap outlasts it.
- Replay trace: cases that name no time now get Bun's 5 s default scaled
  by testBudgetMs, and the multicall/UTF-8 cases use the forwarded-exec
  bound instead of literal 3 s exec and probe deadlines.

Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the product ceiling

At scale 10 the adoption retained-dispatch subset still expired its
200 s caller deadlines on the M5 while the authored suite and 8 churn
workers ran (1-min load 44-91): one public retained up outlasted 200 s
and v7 start/restart refused with E_LEGACY_ADOPTION_DEADLINE. Raise
TEST_BUDGET_SCALE to 20.

The authored startup deadline is capped at 300 s, the largest value
serveNativeAuthoredProject accepts, and the stop deadline uses the same
value.

Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One store probe carries a single aggregate 15 s clock across a whole
resource inspection, dozens of stub-engine processes. Under load that
expired as E_NATIVE_COMPOSE_PROBE_TIMEOUT, reported as
E_LEGACY_ADOPTION_STATE. Stores the cases open now pass timeoutMs 60 s,
the largest the product accepts, unless a case names its own.

The two expiry cases used a 5 s caller deadline that expired before the
callback at load 30-45. They now use 45 s, under the 60 s phase bound so
the caller deadline still ends the held probe; the 1.5 s promptness
assertion is unchanged.

Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ound

The replay child runs the real ownership policy against recorded
replies, one stub engine per reply under a single 15 s probe clock.
Under a 36-worker churn (load 100-120) that clock expired with
E_NATIVE_COMPOSE_PROBE_TIMEOUT and changed the replayed outcome of
'trace replay keeps a member whose placeholder first appears in the
recheck scan unready'. The replay classifies replies and never tests
that clock, so the child now passes timeoutMs 60 s, the largest the
policy accepts, unless the recorded selection names one. The child
budget still outlasts it.

Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
roodboi and others added 7 commits October 24, 2026 16:44
Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e test budget

The witnessed, enrollment and engine persistent-data tests passed literal
5-10 s deadlines to fake-backend enrollments and retained reads. Move the
runtime tests' 120 s RUN_BUDGET to a crate-level TEST_RUN_BUDGET and use it
for those deadlines too. Passing calls never wait on it; expiry cases keep
their own zero deadline.

Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
testBudgetMs now stops at CASE_TIMEOUT_CEILING_MS, a third of the shard's
720 s per-file limit, and withinCaseCeilingMs checks composed case timeouts
when a file loads. A hung case then times out by name before the shard
stops its file. The authored start file derives its case budget from the
ceiling minus settlement, caps the startup and stop deadlines at half of
it, and bounds its stand-in owner and hanging status request at the
ceiling instead of 1800 s and 1200 s.

Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uard

Bun's timeout does not cancel a case body, so the adoption file latched
after one abandoned public static7 case and failed every later case. Every
case now runs under createCaseDrain: an overrun fails that case by name,
the body gets a drain (the operation deadline plus slack) to settle, and
only a body still running after that reserves the shared roots and env
until it settles. Case budgets, the operation deadline and the drain fit
the case ceiling.

The three expiry cases no longer wait out a 45 s deadline: once the
callback runs they move Date.now (the product's only clock) to just
before or past the deadline, so the held probe still ends by the timeout
the product derives from the remaining time.

Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xt one

A loaded run (1-min load 100-120) showed a real case body outlasting its
150 s budget and 85 s drain because several owner operations were still
ahead of it. The next 92 cases then refused at once, so the reservation
never had time to clear. The next case's beforeEach now waits for such a
body (one more case ceiling, outside that case's own timeout) and starts
normally once it settles. Only a body that outlasts that wait too makes
later cases refuse, without waiting again.

Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs HACK-1367

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
roodboi added a commit that referenced this pull request Oct 25, 2026
…reap failed relay starts (#508)

## Summary

Fix unit G112 for HACK-1342, items 1, 2 and 4. Test-harness only: one file changed
(`packages/runtime-core/tests/stream_relay_contract.rs`). The relay (`guest/stream-relay.c`) is
not changed, because the evidence shows it behaves correctly.

**Item 3 (the 2 s deadlines in `provider/graph/native/runtime/tests.rs`) is left to #419
(HACK-1367).** This PR does not touch `runtime/tests.rs`, `exec_tests.rs`, `run_tests.rs` or
`test_budget.rs`.

| Item | Root cause | Change | |---|---|---| | 2. `Relay::connect` EINVAL (CI run 40119166731,
`stream_relay_contract.rs:108:14`) | The relay refuses an excess connection by accepting it and
calling `close()` (`stream-relay.c:489`). For an unavailable target it calls `release()` after
`connect_target` fails. Both can happen before the client's next syscall. Once the peer has
closed a stream socket, macOS (XNU `sosetopt`) rejects `SO_RCVTIMEO`/`SO_SNDTIMEO` with EINVAL,
because both directions are shut. `Relay::connect` unwraps `set_read_timeout` (line 105) and
`set_write_timeout` (line 108), so it panics when the refusal arrives first. The hypothesis is
**confirmed**. The relay is correct. | The two refusal tests now call
`Relay::connect_refused()`. It sets no socket timeouts. It waits up to `REFUSAL_DEADLINE` (10 s,
as before) with `poll` and then still requires EOF or a reset (`assert_refused`). A peer that
stays open fails the check (timeout), and so does a peer that sends data. `Relay::connect` still
unwraps the timeout calls for streams the relay must keep, so EINVAL there is still a real
failure. | | 4. Orphaned `stream-relay-host` on a readiness panic | `start_options` spawned the
child and ran its readiness assertions before any `Relay` existed, so a panic dropped the
`Child` without killing it. | The readiness checks moved into `await_bare_ready`. Until
readiness passes, the existing `PublisherStartup` guard owns the child. Its `Drop` kills and
reaps the child, and removes a fresh scratch root (never a root the caller passed in). | | 1.
`provider/publisher.rs` probe-port race | Already fixed on `next` by #399 (674a0a4): the
fixture now runs `--publish 0` and reads `ready <port>`. No probe listener is left. | No change.
It was re-verified under load (below). |

## Evidence

### Root cause (macOS 26.5.1, Darwin 25.5.0)
A standalone Rust repro (not committed):
- socketpair, peer still open: `set_read_timeout` = `Ok(())`
- socketpair, peer dropped: `set_read_timeout` and `set_write_timeout` = `Os { code: 22,
  InvalidInput }`; `read` = `Ok(0)`
- Unix listener, server accepts and then closes, as the relay does for an excess connection:
  **200/200** `set_write_timeout` calls return EINVAL
- peer shut down only its write side: `set_read_timeout` = `Ok(())`. The error needs both
  directions to be shut, which matches XNU's check.

### Parallel load on this macOS host (`cargo +1.97.1`, Zig 0.15.2 via mise, 8 concurrent processes × `--test-threads 16`, host load average ~105–116 on 18 cores)
| Run | Before (origin/next) | After |
|---|---|---|
| Full `stream_relay_contract` binary | 104/104 pass | 104/104 pass |
| Focused (`connection_limit`, `occupied_socket`) | **1994/2000 pass**: 6 failures, all
  `connection_limit_refuses_excess...` with `Os { code: 22 }` at the `set_read_timeout` unwrap
  (`:105:14`) | **2000/2000 pass** |
| Item 1: `provider::publisher` lib tests (`--features native-stream-relay`) | n/a (already
  fixed on next) | 104/104 pass |

The before failures appeared at line 105 locally and at line 108 on CI. Both are the same race;
it depends on which `setsockopt` the relay's `close()` lands before.

### New tests, each proven by mutation
- `refusal_check_accepts_a_peer_that_closed_before_the_check`: drops the peer before the check.
  Mutation: restore the old pattern (set timeouts, then read) in `assert_refused_within`.
  Result: it fails deterministically with EINVAL.
- `refusal_check_rejects_a_live_peer` and `refusal_check_rejects_a_peer_that_sends_data`
  (`should_panic`). Mutation: make the check return early (accept anything). Result: both fail.
  The new check does not weaken the refusal proof.
- `failed_relay_readiness_kills_and_reaps_the_child`: a live child prints the wrong readiness
  line. The test then requires `waitpid(pid, WNOHANG)` = ECHILD (already killed and reaped) and
  the fresh root removed. Mutation: take the child out of the guard before the checks (the old
  ownership). Result: it fails with `E_RELAY_STARTUP_ORPHAN`, and the test kills the child
  itself.

### Gates
- `cargo +1.97.1 fmt --check`; clippy `--all-targets -D warnings` with default features
  (`check:local`) and with `--all-features`: clean
- `test:local` (`cargo +1.97.1 test --jobs 2`): all pass (lib 1065 passed, 101 ignored)
- `stream_relay_contract` with `native-stream-relay`: 29/29
- `bun run check` (includes `privacy:check`) and `bun run --cwd packages/cli typecheck`: 0
  errors
- gitleaks 8.30.1 over `origin/next..HEAD` (`gitleaks dir` on a `git log -p` export): no leaks
- Hosted CI at head 8cfc2a0 (run 40145181695): all 20 checks green, including `test` shards 1-4
  and `Candidate core (macos-latest)` (optional feature contracts). This is the full-suite
  evidence.

## Gaps
- Load proof ran on this macOS host, not the M3. The M3 is reserved; hosted CI provides the full
  suite.
- The EINVAL behavior comes from XNU's `sosetopt` check. Linux does not show it, and the new
  helper does not depend on it either way.
- Item 3 is left to #419.

### Round 2 (5752a32): stale scratch roots

The verifier hit one unrelated failure on `next` under 32-process load: `fs::create_dir` at line
52 returned EEXIST. Roots are named `/tmp/<prefix>-<pid>-<n>`, and a directory left by an
earlier run whose PID is now reused collides. The new `fresh_root(prefix)` skips an existing
name and panics on any other error. Every root site uses it, including the fixed-name `hkr-
netns`, `hkr-reservation`, `hkp-ack` and `hkp-private` roots.

- New test `fresh_root_skips_a_stale_directory_with_the_next_name`. A mutation that stops
  skipping makes it fail.
- `stream_relay_contract`: 30/30 pass. 32 full-binary runs (4 rounds of 8 processes, `--test-
  threads 16`, load 92-154) all pass. fmt and `clippy --all-targets --all-features -D warnings`
  are clean. gitleaks found no leaks.
- Not changed: the occupied-socket test still takes its "unavailable" port by binding and
  dropping a listener, so a parallel process could in theory take that port. I tried holding a
  bound socket that never listens. On macOS 26 a connect to such a port times out (ETIMEDOUT
  after about 7.8 s, with `net.inet.tcp.blackhole` 0) and is not refused, so that would turn the
  refusal into a timeout. The theoretical race is tracked separately.

Refs HACK-1342

🤖 Generated with [Claude Code](https://claude.com/claude-code)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant