Repository navigation
Conversation
Fixed harness budgets failed under host load (M5 at load 15-40, M3 while
another suite runs) although each case passes alone:
- runtime-core native runtime tests gave fake-backend graph runs 1-5 s;
journal writes on a loaded host exceeded that with graph_timeout. They now
take RUN_BUDGET (120 s, under the product's 600 s cap). Passing runs never
wait on it. The log-expiry case keeps its own 1 s deadline.
- native-compose-adoption-generation cases spawn the Bun stub engine for
several hundred probes per retained dispatch; 20 s caller deadlines and
20-30 s case timeouts expired. Case timeouts and non-deadline caller
deadlines now derive from testBudgetMs. The 5 s expiry cases and their
promptness assertions are product bounds and stay literal.
- the process-policy replay child defaulted to 5 s and refused replays
("Initial process-policy replay unavailable"); its child and summary
budgets, and the replay cases' timeouts, now derive from testBudgetMs.
tests/helpers/test-budget.ts holds the one scale (TEST_BUDGET_SCALE).
Refs HACK-1367
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hored budgets Under host load, "cancel during the runtime status check refuses before the startup intent" failed settlement after its assertions passed. The stand-in status request appended its PID with appendFile, and the case cancelled as soon as the request was logged. The product's SIGKILL could land after the file was created but before the PID was written. Settlement reads an empty PID file as a write still in progress, so it waited out its budget. - The stand-in now stages its PID and renames it into place, so the record is either absent or complete. - The case waits for the published PID (awaitPidFile) before it cancels, instead of the call log. - The file's settlement, startup and stop harness budgets now derive from testBudgetMs. The 50 ms deadline cases and their wall bound stay literal. Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ch and replay cases Raise TEST_BUDGET_SCALE from 4 to 10. With a second suite plus churn workers on the M5 (1-min load 28-66), Source5 authored starts expired the 60 s startup deadline and v7 retained dispatches expired their 80 s caller deadline; a v7 dispatch case took 146 s at load 22-50. - Adoption: public dispatch takes its caller deadline from the Compose startup budget (90 s default). Inject the harness operation deadline through HACK_COMPOSE_STARTUP_TIMEOUT_MS in beforeEach and restore it. - Authored: the planner-refusal case uses STARTUP_TIMEOUT_MS instead of a literal 10 s; the two-cycle down.after case gets two case budgets, and the stand-in owner cap outlasts it. - Replay trace: cases that name no time now get Bun's 5 s default scaled by testBudgetMs, and the multicall/UTF-8 cases use the forwarded-exec bound instead of literal 3 s exec and probe deadlines. Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the product ceiling At scale 10 the adoption retained-dispatch subset still expired its 200 s caller deadlines on the M5 while the authored suite and 8 churn workers ran (1-min load 44-91): one public retained up outlasted 200 s and v7 start/restart refused with E_LEGACY_ADOPTION_DEADLINE. Raise TEST_BUDGET_SCALE to 20. The authored startup deadline is capped at 300 s, the largest value serveNativeAuthoredProject accepts, and the stop deadline uses the same value. Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One store probe carries a single aggregate 15 s clock across a whole resource inspection, dozens of stub-engine processes. Under load that expired as E_NATIVE_COMPOSE_PROBE_TIMEOUT, reported as E_LEGACY_ADOPTION_STATE. Stores the cases open now pass timeoutMs 60 s, the largest the product accepts, unless a case names its own. The two expiry cases used a 5 s caller deadline that expired before the callback at load 30-45. They now use 45 s, under the 60 s phase bound so the caller deadline still ends the held probe; the 1.5 s promptness assertion is unchanged. Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ound The replay child runs the real ownership policy against recorded replies, one stub engine per reply under a single 15 s probe clock. Under a 36-worker churn (load 100-120) that clock expired with E_NATIVE_COMPOSE_PROBE_TIMEOUT and changed the replayed outcome of 'trace replay keeps a member whose placeholder first appears in the recheck scan unready'. The replay classifies replies and never tests that clock, so the child now passes timeoutMs 60 s, the largest the policy accepts, unless the recorded selection names one. The child budget still outlasts it. Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e test budget The witnessed, enrollment and engine persistent-data tests passed literal 5-10 s deadlines to fake-backend enrollments and retained reads. Move the runtime tests' 120 s RUN_BUDGET to a crate-level TEST_RUN_BUDGET and use it for those deadlines too. Passing calls never wait on it; expiry cases keep their own zero deadline. Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
testBudgetMs now stops at CASE_TIMEOUT_CEILING_MS, a third of the shard's 720 s per-file limit, and withinCaseCeilingMs checks composed case timeouts when a file loads. A hung case then times out by name before the shard stops its file. The authored start file derives its case budget from the ceiling minus settlement, caps the startup and stop deadlines at half of it, and bounds its stand-in owner and hanging status request at the ceiling instead of 1800 s and 1200 s. Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uard Bun's timeout does not cancel a case body, so the adoption file latched after one abandoned public static7 case and failed every later case. Every case now runs under createCaseDrain: an overrun fails that case by name, the body gets a drain (the operation deadline plus slack) to settle, and only a body still running after that reserves the shared roots and env until it settles. Case budgets, the operation deadline and the drain fit the case ceiling. The three expiry cases no longer wait out a 45 s deadline: once the callback runs they move Date.now (the product's only clock) to just before or past the deadline, so the held probe still ends by the timeout the product derives from the remaining time. Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xt one A loaded run (1-min load 100-120) showed a real case body outlasting its 150 s budget and 85 s drain because several owner operations were still ahead of it. The next 92 cases then refused at once, so the reservation never had time to clear. The next case's beforeEach now waits for such a body (one more case ceiling, outside that case's own timeout) and starts normally once it settles. Only a body that outlasts that wait too makes later cases refuse, without waiting again. Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs HACK-1367 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
roodboi
added a commit
that referenced
this pull request
Oct 25, 2026
…reap failed relay starts (#508) ## Summary Fix unit G112 for HACK-1342, items 1, 2 and 4. Test-harness only: one file changed (`packages/runtime-core/tests/stream_relay_contract.rs`). The relay (`guest/stream-relay.c`) is not changed, because the evidence shows it behaves correctly. **Item 3 (the 2 s deadlines in `provider/graph/native/runtime/tests.rs`) is left to #419 (HACK-1367).** This PR does not touch `runtime/tests.rs`, `exec_tests.rs`, `run_tests.rs` or `test_budget.rs`. | Item | Root cause | Change | |---|---|---| | 2. `Relay::connect` EINVAL (CI run 40119166731, `stream_relay_contract.rs:108:14`) | The relay refuses an excess connection by accepting it and calling `close()` (`stream-relay.c:489`). For an unavailable target it calls `release()` after `connect_target` fails. Both can happen before the client's next syscall. Once the peer has closed a stream socket, macOS (XNU `sosetopt`) rejects `SO_RCVTIMEO`/`SO_SNDTIMEO` with EINVAL, because both directions are shut. `Relay::connect` unwraps `set_read_timeout` (line 105) and `set_write_timeout` (line 108), so it panics when the refusal arrives first. The hypothesis is **confirmed**. The relay is correct. | The two refusal tests now call `Relay::connect_refused()`. It sets no socket timeouts. It waits up to `REFUSAL_DEADLINE` (10 s, as before) with `poll` and then still requires EOF or a reset (`assert_refused`). A peer that stays open fails the check (timeout), and so does a peer that sends data. `Relay::connect` still unwraps the timeout calls for streams the relay must keep, so EINVAL there is still a real failure. | | 4. Orphaned `stream-relay-host` on a readiness panic | `start_options` spawned the child and ran its readiness assertions before any `Relay` existed, so a panic dropped the `Child` without killing it. | The readiness checks moved into `await_bare_ready`. Until readiness passes, the existing `PublisherStartup` guard owns the child. Its `Drop` kills and reaps the child, and removes a fresh scratch root (never a root the caller passed in). | | 1. `provider/publisher.rs` probe-port race | Already fixed on `next` by #399 (674a0a4): the fixture now runs `--publish 0` and reads `ready <port>`. No probe listener is left. | No change. It was re-verified under load (below). | ## Evidence ### Root cause (macOS 26.5.1, Darwin 25.5.0) A standalone Rust repro (not committed): - socketpair, peer still open: `set_read_timeout` = `Ok(())` - socketpair, peer dropped: `set_read_timeout` and `set_write_timeout` = `Os { code: 22, InvalidInput }`; `read` = `Ok(0)` - Unix listener, server accepts and then closes, as the relay does for an excess connection: **200/200** `set_write_timeout` calls return EINVAL - peer shut down only its write side: `set_read_timeout` = `Ok(())`. The error needs both directions to be shut, which matches XNU's check. ### Parallel load on this macOS host (`cargo +1.97.1`, Zig 0.15.2 via mise, 8 concurrent processes × `--test-threads 16`, host load average ~105–116 on 18 cores) | Run | Before (origin/next) | After | |---|---|---| | Full `stream_relay_contract` binary | 104/104 pass | 104/104 pass | | Focused (`connection_limit`, `occupied_socket`) | **1994/2000 pass**: 6 failures, all `connection_limit_refuses_excess...` with `Os { code: 22 }` at the `set_read_timeout` unwrap (`:105:14`) | **2000/2000 pass** | | Item 1: `provider::publisher` lib tests (`--features native-stream-relay`) | n/a (already fixed on next) | 104/104 pass | The before failures appeared at line 105 locally and at line 108 on CI. Both are the same race; it depends on which `setsockopt` the relay's `close()` lands before. ### New tests, each proven by mutation - `refusal_check_accepts_a_peer_that_closed_before_the_check`: drops the peer before the check. Mutation: restore the old pattern (set timeouts, then read) in `assert_refused_within`. Result: it fails deterministically with EINVAL. - `refusal_check_rejects_a_live_peer` and `refusal_check_rejects_a_peer_that_sends_data` (`should_panic`). Mutation: make the check return early (accept anything). Result: both fail. The new check does not weaken the refusal proof. - `failed_relay_readiness_kills_and_reaps_the_child`: a live child prints the wrong readiness line. The test then requires `waitpid(pid, WNOHANG)` = ECHILD (already killed and reaped) and the fresh root removed. Mutation: take the child out of the guard before the checks (the old ownership). Result: it fails with `E_RELAY_STARTUP_ORPHAN`, and the test kills the child itself. ### Gates - `cargo +1.97.1 fmt --check`; clippy `--all-targets -D warnings` with default features (`check:local`) and with `--all-features`: clean - `test:local` (`cargo +1.97.1 test --jobs 2`): all pass (lib 1065 passed, 101 ignored) - `stream_relay_contract` with `native-stream-relay`: 29/29 - `bun run check` (includes `privacy:check`) and `bun run --cwd packages/cli typecheck`: 0 errors - gitleaks 8.30.1 over `origin/next..HEAD` (`gitleaks dir` on a `git log -p` export): no leaks - Hosted CI at head 8cfc2a0 (run 40145181695): all 20 checks green, including `test` shards 1-4 and `Candidate core (macos-latest)` (optional feature contracts). This is the full-suite evidence. ## Gaps - Load proof ran on this macOS host, not the M3. The M3 is reserved; hosted CI provides the full suite. - The EINVAL behavior comes from XNU's `sosetopt` check. Linux does not show it, and the new helper does not depend on it either way. - Item 3 is left to #419. ### Round 2 (5752a32): stale scratch roots The verifier hit one unrelated failure on `next` under 32-process load: `fs::create_dir` at line 52 returned EEXIST. Roots are named `/tmp/<prefix>-<pid>-<n>`, and a directory left by an earlier run whose PID is now reused collides. The new `fresh_root(prefix)` skips an existing name and panics on any other error. Every root site uses it, including the fixed-name `hkr- netns`, `hkr-reservation`, `hkp-ack` and `hkp-private` roots. - New test `fresh_root_skips_a_stale_directory_with_the_next_name`. A mutation that stops skipping makes it fail. - `stream_relay_contract`: 30/30 pass. 32 full-binary runs (4 rounds of 8 processes, `--test- threads 16`, load 92-154) all pass. fmt and `clippy --all-targets --all-features -D warnings` are clean. gitleaks found no leaks. - Not changed: the occupied-socket test still takes its "unavailable" port by binding and dropping a listener, so a parallel process could in theory take that port. I tried holding a bound socket that never listens. On macOS 26 a connect to such a port times out (ETIMEDOUT after about 7.8 s, with `net.inet.tcp.blackhole` 0) and is not refused, so that would turn the refusal into a timeout. The theoretical race is tracked separately. Refs HACK-1342 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs HACK-1367 (unit G52). Related: HACK-1458, which owns the product clocks this PR leaves alone. Test-only change: no product code, product default or product bound changes. The branch also merges
next(latest: 191382d).Root cause
Each case below passes alone but failed under host load (M5 at load 15-40, or higher; M3 while another suite runs), because a fixed harness budget was spent on fixture work whose wall time follows host load, or because the harness raced a kill:
provider::graph::native::runtime::tests, for exampletwo_owned_bridges_keep_per_network_members_and_aliases_through_cleanupandtwo_bridge_cleanup_retries_only_the_retained_second_bridge_after_first_delete). Fake-backendexecution::runcalls had 1-5 s budgets. The fake is ready at once, but on a loaded host the journal writes took longer than 2 s, so the run failed withgraph_timeout.tests/native-compose-adoption-generation.test.ts. Every engine probe spawns the Bun stub engine, and one retained dispatch issues several hundred probes. On the loaded M5 one start took 18-24 s, which is past the 20 s caller deadlines and the 20-30 s case timeouts. Baseline atorigin/nexton the loaded M5: 8 failures (v3, v5 and v7 retained dispatch cases) before I stopped the run, allE_LEGACY_ADOPTION_DEADLINEor case timeouts.tests/e2e/native-process-policy-initial-replay.ts. Each replay child cold-starts Bun, transpiles the ownership policy and spawns one stub engine per recorded reply. Its default was 5 s, so under load the child timed out and the harness threwInitial process-policy replay unavailable.tests/native-authored-project-start.test.ts.stage runtimefailure, and the module-levelunconfirmedlatch that then failed every later case, were already fixed onnextby test(native): isolate startup settlement guard per case #388 (HACK-1299). It added per-case guards (tests/helpers/native-fixture-cases.ts), so a failed case reports its own failure once, and it replaced the owner's fixed 30 ms ready window with a release handshake.cancel during the runtime status check refuses before the startup intent, the stand-in status request appended its PID withappendFile, and the case cancelled as soon as the request appeared in the call log. The product's SIGKILL could land after the PID file was created but before the PID was written. Settlement reads an empty PID file as a write still in progress, so it reportedsettlement unconfirmedafter 15 s even though the case's assertions passed. The retained root had an emptystatus.pid.The witnessed enrollment
Fixture::newtemp-dir collision (persistent_data/enrollment/witnessed/tests.rs) is fixed by open draft #399 (G40/HACK-1348,next_test_scratch()). This PR does not duplicate it.Fix
tests/helpers/test-budget.tsholds one scale,TEST_BUDGET_SCALE = 4, andtestBudgetMs(unloadedMs). Each call site names the time its fixture work needs on an unloaded host, and the helper scales it.OPERATION_DEADLINE_MS), now come fromtestBudgetMs. The product bounds under test stay literal: the 5 s deadline-expiry cases, thedeadline + 1500promptness check, and the 4 s cancellation reap.REPLAY_CHILD_BUDGET_MS), the summary budget, and the replay cases' timeouts intests/native-process-policy-initial-trace.test.tsnow come fromtestBudgetMs. The ownership policy's own probe bounds still run unchanged inside the child.RUN_BUDGET(120 s, under the 600 s product cap ofexecution::run) for every fake-backend run, selection, exec, run and log read inruntime/{tests,exec_tests,run_tests}.rs. Passing calls never wait on it. The log-expiry case keeps its own 1 s deadline, because the fake sleeps past it on purpose.awaitPidFile) before it cancels. It no longer waits on the call log.testBudgetMs(20 s, 60 s and 60 s). The 50 ms deadline cases and their 1 s wall bound stay literal.Tests and evidence
Toolchain: Bun 1.4.2 (
mise x bun@1.4.2). Rust used cargo +1.97.1 with its ownCARGO_TARGET_DIRand debug info off.Full suite on the M3 at the pushed head 7f76651 (
bun run test): 7817 pass, 126 skip, 1 fail. The one failure isrepository catalog > the checked-in .hack/tasks.json recipes are all reviewed, which refuses because the M3 CI worktree is a detached checkout (detached_checkout). It is unrelated to this change: the same failure is in the G36, G41, G49 and G51 M3 logs, and the test passes locally on a branch checkout. See the finding in the unit report.Loaded repetitions on the M3 at 7f76651 (round 1,
TEST_BUDGET_SCALE = 4). Superseded for the authored start and the adoption retained dispatch. Round 2 could not repeat these 10/10 results on the M5 under heavier concurrent load, and its "Not met" section stands. The runtime-core and replay rows still hold. Load came from 16 spin-and-spawn shell workers on the 16-core host, stopped in a shell trap. The 1-minute load average during the runs is shown.cargo +1.97.1 test --features native-config-plan --lib -- provider::graph::native::runtime::testsbun test tests/native-process-policy-initial-trace.test.tsbun test tests/native-authored-project-start.test.ts(whole file)bun test tests/native-compose-adoption-generation.test.ts -t '^(v[0-9]+ |rolled-back v|public |retained command dispatch)'The baseline (the
616407a5versions of the Rust and replay tests, under the same churn) passed 3/3 for both targets on the M3. This churn did not reproduce those two failures on the M3, so the before/after contrast for them comes from the M5 runs below.Local runs (M5, already loaded, no added churn):
origin/nextthe same host failed 8 cases.Mutations:
RUN_BUDGETset to 1 ns, both named Rust tests fail withgraph_timeoutat theirexecution::runline.settlement unconfirmed ... stand-in hanging runtime status request. This reproduces the loaded failure deterministically.settlement unconfirmed, so the case still detects a surviving request.Gates: root
bun run check(includesprivacy:check) andbun run --cwd packages/cli typecheckpassed with 0 errors. Biome found nothing on the changed test files.cargo +1.97.1 fmt --checkand clippy-D warningspassed for the Rust change. gitleaks 8.30.1 overorigin/next..HEADfound no leaks.Gaps
The 5 s deadline-expiry adoption cases still check product bounds against the wall clock.Fixed in round 3: these cases now moveDate.nowinstead of waiting.A static7 case that times out latchesFixed in round 3 by the per-case drain guard. The file still shares module-level fixture globals.publicStatic7Settlement, and every later case then fails.appendFile. A SIGTERM that lands before its handler is registered could leave an emptyowner.pidin the same way. This was not seen in any run.cancel during the runtime status checkassumes the status request is still in flight when it cancels. If a Bun child takes longer than the product's 5 s status timeout to start, the case can still fail. The product has no seam for that timeout.startup keeps the native failure when missing receipt inspection also refusesfailed once. The start reportednative_graph_receiptfrom the follow-up inspection instead of the serve failuregraph_network_intent. The serve child had written its code to stderr before exiting. The most likely cause, which I have not confirmed, is thatserveGraphProcessstops reading stderr 250 ms after the child exits (src/backends/native-project-process.ts), so a reader that falls behind on a loaded host drops the code. The case passed 10/10 on the M3.Round 2: verifier rework (M5 under concurrent load)
The verifier found that items (2), (3) and (4) still failed on the M5 at head 7f76651. This round merges
nextat 6e4bcf2 and adds four commits: 1686c1d, f0352ce, fa5b36d and 073041c. It is still test-only. TheFixsection above describes round 1. Where it givesTEST_BUDGET_SCALE = 4and fixed 5 s expiry deadlines, this section replaces it.What changed
TEST_BUDGET_SCALEwent from 4 to 10, then to 20. These fixtures are spawn-bound: one retained adoption dispatch runs about 750 stub-engine processes. Their wall time therefore grows faster than the load average.upoutlasted a 200 s deadline.testBudgetMsderives is a ceiling. A passing case does not wait on it.tests/native-authored-project-start.test.ts):min(testBudgetMs(15 s), 300 s). 300 s is the largest valueserveNativeAuthoredProjectaccepts. The stop deadline uses the same value.STARTUP_TIMEOUT_MS.known failed down.after ...runs two full cycles, so it gets two case budgets. The stand-in owner's cap was raised so it outlasts that case.tests/native-compose-adoption-generation.test.ts):beforeEachnow injects the harness operation deadline throughHACK_COMPOSE_STARTUP_TIMEOUT_MS, andafterEachrestores it.timeoutMs, the largest the product accepts) unless the case names one. One store probe carries a single 15 s clock across a whole resource inspection, and under load it expired asE_NATIVE_COMPOSE_PROBE_TIMEOUT, which surfaces asE_LEGACY_ADOPTION_STATE.tests/native-process-policy-initial-trace.test.ts,tests/e2e/native-process-policy-initial-replay.ts):testBudgetMs. Lines 258, 302 and 749 had kept an unscaled 5 s.Mutations (each restored afterwards)
timed out after 5000ms.HACK_COMPOSE_STARTUP_TIMEOUT_MSinjected as 1000,public retained up ... "no"fails 5.5 s afterstart, withE_CONFIG_INVALID. This shows the injected value reaches the CLI's deadline.PROBE_PHASE_BUDGET_MS = 1,v7 start preserves ...fails in 1.2 s withE_LEGACY_ADOPTION_STATE.REPLAY_PROBE_BUDGET_MS = 1,HACK-1279 case 1replaysE_NATIVE_COMPOSE_PROBE_TIMEOUTinstead ofcode: null.Evidence (M5, Bun 1.4.2, private HOME, HACK_HOME and DOCKER_CONFIG, unreachable DOCKER_HOST, keychain fallback disabled)
Load came from
bashspin and/bin/shspawn workers, stopped in an EXIT/INT/TERM/HUP trap. Other agents on the host added their own load, at 1-min load 20-140 even with none of mine running. Loop logs were kept outside the checkout. The checkout-write guard from #405 fails a whole file when any new file appears in the checkout during the run.HACK-1279 case 1withE_NATIVE_COMPOSE_PROBE_TIMEOUTinside the replay childNot met: 10 consecutive loaded passes for items (2) and (4). None of the remaining failures is a harness budget expiring. They come from product bounds that a test cannot set, and from one hang:
E_COMPILER_TIMEOUTat stageruntimeorhook-up-before. Each compiler invocation has a fixed 10 s default (DEFAULT_TIMEOUT_MSinsrc/lib/native-config-compiler.ts). The authored start passes no compiler timeout, even when its own deadline has minutes left.Source5 cancellation after readiness ...hung for its whole 600 s budget. See the findings below.timeoutMs, so its 15 s probe-phase clock stays fixed. It expires under load and is reported asE_CONFIG_INVALID("generation state is invalid, unsafe or changed").open/preparerefuse withE_LEGACY_ADOPTION_STATEwithin 4-5 s.translate()hides the original error, so I did not find the root cause.Gates
bun run check(includesprivacy:check): passed at every pushed head.bun run --cwd packages/cli typecheck: 0 errors at every pushed head.dir, over agit log -p origin/next..HEADexport: no leaks.docs:cli-referenceand plugin regeneration do not apply.Hosted CI
At 073041c, run 40115797226 (
pull_request, its headSha matches) passed every check. That includestestshards 1-4,docker-e2eshards 1-4, Candidate core (macOS and ubuntu) and secret-scan. This is the full-suite evidence. It ran against the PR merge ref withnextas of that time.At fa5b36d,
test shard 3/4failed intests/project-env-layer-lock.test.ts > concurrent legacy dotenv sets and removals of different keys all apply, with a private-file "not a regular file" refusal. That test is not in this diff and not on the known-flake list, so I did not rerun it. It passed at the next head.Round 2 findings (not fixed here)
tryLegacyComposeAdoptedCommandopens its store withouttimeoutMs, so each inspection phase getsctx.timeoutMs ?? 15_000across dozens of engine calls.HACK_COMPOSE_STARTUP_TIMEOUT_MSdoes not affect it. A slow engine therefore turnshack upon an adopted project intoE_CONFIG_INVALID"state is invalid, unsafe or changed", not a timeout code. Medium.E_COMPILER_TIMEOUTwhilestartupTimeoutMsstill has time left. Medium-low.Source5 cancellation after readinesscan hang.onReadyaborted, the runtime issuedgraph native control --action statustwice. It never terminated thefrontend-serveowner and never ran the down hooks. The owner and the persistent host process group were still alive 13 minutes later.bun index.tsstay up after the run. Settlement reports them as unconfirmed but does not end them. I ended the groups that my runs left, after confirming ownership from my temp roots. Medium.Round 3: verifier rework (items 1-4)
This round merges
next(7dac4a0, cd35034, then 191382d) and adds four commits: 7fbeb50, 5afe58a, 6e3ec5e and b18202c. It is still test-only. Where this section disagrees with rounds 1 and 2 above, this section wins.Scope. The product clocks stay out of scope here. They are tracked in HACK-1458: the fixed 15 s public dispatch inspection clock (
E_CONFIG_INVALID), the authored compiler's fixed 10 s timeout, the Source5 cancellation hang, and lifecycle process groups left after failed authored cases. Because of them, HACK-1367's "10 consecutive loaded passes" for the adoption and authored files is still not met, and this PR does not claim it.What changed
TEST_RUN_BUDGETinpackages/runtime-core/src/test_budget.rs. It is gated ontestandnative-config-plan, the only feature its users build under.persistent_data/enrollment/witnessed/tests.rs(lines 196 and 206) now uses it in place of a literal 5 s. So do the siblingenrollment/tests.rs(literal 10 s) andengine/tests.rs(literal 5 s).runtime/tests.rskeeps its localRUN_BUDGETname, now set to that constant.publicStatic7Settlement,publicStatic7Commandsand thepublicStatic7Casewrapper are gone.createCaseDrain(tests/helpers/case-drain.ts). Each case body has its own budget, and an overrun fails that case once, by name, withE_TEST_CASE_BUDGET.beforeEachwaits for it for up to one more case ceiling (240 s). This wait runs outside that case's own timeout, and the case starts normally once the body settles.E_TEST_CASE_RESERVED, which names the case still running. They refuse immediately, without waiting again, until it settles.testBudgetMsnow stops atCASE_TIMEOUT_CEILING_MS. That is a third ofscripts/test-shard.ts'sDEFAULT_FILE_TIMEOUT_SECONDS(720 s), so 240 s.withinCaseCeilingMschecks each file's composed case timeout when the file loads.OPERATION_DEADLINE_MSis 80 s, down from 400 s.Date.nowtimestamp and derives each probe's timeout from the time left. Bun'ssetSystemTimefreezesDate.now, so the product'sDate.now() >= deadlinecheck after a killed probe would never pass. I tried it and did not use it.shiftProductClockspies onDate.nowfor the rest of the case. It moves the clock forward and lets it keep advancing at the real rate. It is restored infinally.v5 expired aggregate deadline after callbackmoves the clock 5 ms past the deadline instead of sleeping until then.v5 remaining aggregate clock ...cases move the clock to a lead before the deadline: 5 s for per-effect freshness and 20 s for the final observation. The held probe still ends by the timeout the product derives from the time left. The 1.5 s promptness assertion, the PID-killed check and pending retention are unchanged.max(0, ...), so the clock never moves back.v3 linked saved proofcase (49 s on hosted CI) waits on no clock. It is compiler and probe work, and it is unchanged.New tests and mutations (each restored afterwards)
tests/case-drain.test.ts(8 cases) andtests/test-budget.test.ts(4 cases). Mutations run against copies of the helper:Rust: with
TEST_RUN_BUDGETset to 1 ns, 16 of 44persistent_datatests fail, including 6 of the 7 inenrollment::witnessed::tests.Injected overrun in the adoption file, like the verifier's reproduction. A case with a 2 s budget was placed before the first real case, then the file ran with a filter for it and three fast cases. The host was at 1-min load 185-215, from other agents plus these three runs in parallel.
E_TEST_CASE_BUDGET ... settled within the 85000 ms drain). The two light real cases pass.v3 raw managed driftfailed its own 150 s budget by name at that load.v3 raw managed driftfailed its own budget by name, as above.E_TEST_CASE_RESERVEDnaming it, and the remaining cases refuse at once. This is the one case in which later cases still fail. Their reason names the case that is still running, and they start again if that case settles.At the original head, the verifier's single hanging case produced 0 pass and 116 fail.
Evidence (M5, Bun 1.4.2, private HOME, HACK_HOME and DOCKER_CONFIG, unreachable DOCKER_HOST, keychain fallback disabled)
Other agents kept the host at 1-min load 55-215 throughout, with their own churn workers. I added no churn of my own.
persistent_datamodule, cargo +1.97.1--all-features: 10/10 consecutive runs, 44 pass each, at load 56-59. Alsofmt --checkand clippy-D warnings, both default and--all-features, and theruntimemodule (52 pass, 1 ignored).-t aggregate, 5 cases): 5/5 pass at load 56-66, after the merge too. There is no 45 s wait any more. Hosted per-case times are in the CI section below.rolled-back v5 can reprepare ... directory) still had several owner operations to run. The 92 cases after it refused at once, so the reservation never got time to clear.Source5 supervised ...failed withhook-up-before.Source5 cancellation after readiness ...failed withhook-up-before (E_COMPILER_CANCELLED). Both of these are compiler-side failures (HACK-1458 items 2-3).known failed down.after ...exceeded its 205 s budget. This is the ceiling trade-off.Gates (at 0dd0d5a)
bun run check(includesprivacy:check): exit 0.bun run --cwd packages/cli typecheck: 0 errors.dir, over agit log -p next..HEADexport: no leaks.Hosted CI
At 0dd0d5a, run 40138406767 (
pull_request, headSha 0dd0d5a) passed every check:testshards 1-4,docker-e2eshards 1-4, Candidate core (macOS and ubuntu), Config compiler (all four platforms), Runtime state models, linux-process-lifetime, runtime-images and secret-scan. This is the full-suite evidence. It ran on the PR merge ref withnextat 191382d.Per-file times from the shard logs:
native-compose-adoption-generation.test.tsv5 expired aggregate deadline after callback ...v5 remaining aggregate clock ... per-effect freshnessv5 remaining aggregate clock ... final observationv3 linked saved proof ...(no clock wait)native-authored-project-start.test.tsnative-process-policy-initial-trace.test.tscase-drain.test.ts/test-budget.test.tsRound 3 findings (not fixed here)
scripts/test-shard.tscould pass its effective--file-timeout-secondsto each file, so directbun testruns keep the larger 20x budgets and only shard runs clamp.process.env, so later cases cannot safely reuse them while that body may still resume.AsyncLocalStorage, asnative-fixture-cases.tsdoes. That is a rewrite of a 5,000-line file.🤖 Generated with Claude Code