Skip to content

test: keep the write guard, pid handshakes and store cases reliable under load (HACK-1439) - #496

Draft
roodboi wants to merge 3 commits into
nextfrom
wf/g99-test-reliability-under-load
Draft

roodboi wants to merge 3 commits into
nextfrom
wf/g99-test-reliability-under-load

Conversation

@roodboi

@roodboi roodboi commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Refs HACK-1439. Based on next after #480 (HACK-1422), which touched three of these files, landed. next was merged in again at 148ef8f, which brought in #494 (HACK-1440) and #497.

Test-only. No assertion was removed or weakened; the only changes are bounds on waits, named Bun timeouts, and fixture timing that makes a race happen on every run.

Root causes

  1. Checkout write guard. installCheckoutWriteGuard registered its end-of-run afterAll without a timeout, so it ran under Bun's 5 s default hook timeout. Under host load the synchronous git status --untracked-files=all passed 5 s. Bun then killed git mid-read ("killed 1 dangling process"), git returned non-zero, and the guard refused with "git status failed" on a run whose cases all passed.
  2. Create-then-write race. In native-runtime-client.test.ts, the "cancellation starts at most one native request and reaps only its owned child" case waited only for the pid file to exist, then aborted. writeFileSync creates the file before it writes the digits, so the abort could SIGKILL the child in between. The file then stayed "" and Number("") gave 0.
  3. 5 s defaults and 30 s literals. The cases in private-env-file.test.ts and native-compose-storage-witness-xattr-owner.test.ts do a fixed amount of store file work (env layer lock, staged write and rename; durable generation publish/enroll/verify/retire) and wait on no timer. Under load they passed Bun's 5 s default or the fixed 30_000 literals while their assertions still held.

Fix

  1. tests/helpers/checkout-write-guard.ts
    • Every guard git call is bounded by CHECKOUT_GIT_BUDGET_MS (60 s, spawnSync timeout + SIGKILL).
    • A call that reaches the bound is refused, closed, with its own message: git status did not finish within N ms. A real failure still gives git status failed. The guard never treats an unreadable checkout as clean.
    • The afterAll hook is registered with CHECKOUT_GUARD_HOOK_BUDGET_MS (status bound + 30 s). A slow status therefore ends as the guard's named refusal, not as a Bun kill.
    • checkout-write-guard.test.ts: the guarded fixtures' keep-alive cases get KEEP_ALIVE_CASE_TIMEOUT_MS. The late spawnSync blocks their event loop while a cold bun child starts, and under load that passed 5 s. This was found while running the file under load.
  2. tests/native-runtime-client.test.ts
    • A single bounded waitForFileContent helper now backs waitForPid and waitForFileText (fix(env): retry a private env read that races an atomic replace instead of refusing it as a symlink #480 added the latter for the finished file).
    • The existence-only waitForFile is removed. Each of its three call sites now waits for complete content: the cancellation case's pid before it aborts, the abort-mode keeper pid, and the late-writer finished text.
    • The cancellation fixture now publishes its pid as a torn handshake (empty file, 50 ms gap, then the digits), as the sibling cases do. An existence-only wait now fails on every run, not by chance.
    • Audited the native-runtime siblings (native-runtime-metadata, -not-running-cli, -preflight): none waits for a file.
  3. private-env-file.test.ts and native-compose-storage-witness-xattr-owner.test.ts
    • Each file sets a file-scoped default with setDefaultTimeout: ENV_STORE_CASE_BUDGET_MS (60 s) and fix(env): retry a private env read that races an atomic replace instead of refusing it as a symlink #480's STORE_CASE_BUDGET_MS. Every case and hook gets a named budget.
    • STORE_CASE_BUDGET_MS is raised from 180 s to 600 s. In one wave at a sustained load average of 110-190, every run failed the same four store cases (warm no-op up/restart, warm enrollment, adjacent verification) at 180 s while their assertions held. The slowest case had taken 63 s at a load of about 90. Those measurements predate perf(native-compose): cut redundant store rechecks per proof step #494, which cut the store's call counts. The budget is a hang guard, so it is sized with room to spare.
    • The 22 fixed 30_000 literals in the xattr file are removed so the named default applies.
    • The two other real-use carriers captured with a 60 s wall-clock deadline now use FAKE_CARRIER_LIFETIME_MS. A slow case can then only time out cleanly; it cannot fail because the carrier deadline expired. The one case that mocks Date.now keeps its own clock deadline.
    • Bun 1.4.2 was checked directly: setDefaultTimeout applies to this file's cases, test.each rows and hooks, and does not leak into a later file in the same run.

Tests and mutation proofs

  • New: "the guard kills and refuses a git status that reaches its bound instead of waiting on it". Fake git exec sleep 30, budget 200 ms.
    • Without the spawnSync timeout, the guard returns [] after 30 s (no refusal): the test fails.
    • Without the timedOut branch, the message is "git status failed": the test fails.
  • New: "an end-of-run git status slower than Bun's default hook timeout still passes a clean run". A git wrapper makes only the end-of-run status sleep 6 s, and records that it slept.
    • With the hook registered without its budget (the old code), the inner run fails with exactly the reported killed 1 dangling process + checkout write guard: git status failed.
  • Cancellation case: replacing waitForPid(pidFile) with an existence wait fails 3 of 3 runs with Expected: <pid> Received: 0, the reported failure. The fixed version passes 3 of 3.
  • Budget changes, before and after (before = fix(env): retry a private env read that races an atomic replace instead of refusing it as a symlink #480's head, at load average ~70 on 18 CPUs): the xattr file had 20 timeouts (13 at the 5 s default, 7 at 30 s), for example "resume refuses * before workload", "fresh readonly proofs tolerate …" and "warm run stays …". After: 108 of 108 pass at the same load. The slowest case took 63 s.

Evidence (local, macOS, 18 CPUs; ambient load from other agents, plus a yes load generator stopped in a trap)

Each run is a full bun test <file> with a private HOME, HACK_HOME and DOCKER_CONFIG, and the checkout write guard preloaded. The load average is the 1-minute value, sampled as each run ended.

At the exact PR head 384bf68:

File Runs passed Parallel Extra yes load Load average
private-env-file.test.ts 30 / 30 (15 cases each) 3 8 31-35
checkout-write-guard.test.ts 30 / 30 (14 cases each) 3 8 38-72
native-runtime-client.test.ts 30 / 30 (34 cases each) 3 8 70-88
native-compose-storage-witness-xattr-owner.test.ts 30 / 30 (109 cases each) 6 8 65-122; uptime read 110-290 while the fifth wave ran

Earlier rounds of the same change:

Gates: bun run check (includes privacy:check) passes. bun run typecheck and bun run --cwd packages/cli typecheck report 0 errors. ultracite check on the five changed files is clean. gitleaks 8.30.1 over origin/next..HEAD finds no leaks. The CLI surface and agent guidance did not change, so no generated docs needed regenerating.

Full suite: hosted CI run 40143709633 passed at the exact PR head 384bf68, including test shards 1-4. All 20 checks are green. The earlier head 7dc0359 was also fully green.

Gaps

  • The keep-alive budget and the two store budgets are timing fixes. No deterministic mutation reproduces them; the before/after load runs above are their evidence.
  • The local 30-run loops are macOS only. Hosted CI covers Linux, but runs each file once.
  • No budget is a guarantee under unbounded load. 600 s held at load averages up to about 290, but a host that is even more oversubscribed can still time out. HACK-1434 tracks the product cost behind the store cases.
  • Other test files outside this unit still wait for existence and then read, or use short fixed budgets. They are filed as findings, not changed here.

🤖 Generated with Claude Code

…nder load (HACK-1439)

- checkout write guard: bound each git call (CHECKOUT_GIT_BUDGET_MS, killed
  and refused closed at the bound) and register the end-of-run afterAll with
  CHECKOUT_GUARD_HOOK_BUDGET_MS instead of Bun's 5 s default, which killed
  git mid-status and failed runs whose cases all passed. Guarded fixture
  keep-alive cases get a named budget too.
- native-runtime-client: one bounded wait-for-complete-content helper backs
  waitForPid and waitForFileText; the cancellation case waits for a complete
  pid before it aborts (its fixture now publishes a torn handshake every
  run), and no case waits for existence and then reads.
- private-env-file and native-compose-storage-witness-xattr-owner: a
  file-scoped named default budget replaces Bun's 5 s default and the fixed
  30_000 literals; captured carriers outlive their case budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 24, 2026

Copy link
Copy Markdown

HACK-1439

roodboi and others added 2 commits October 24, 2026 19:34
…load (HACK-1439)

Four store cases (warm no-op up/restart, warm enrollment, adjacent
verification) passed the 180 s budget in every run of a wave at a sustained
load average of 110-190 on an 18-CPU host, while their assertions held; the
slowest took 63 s at a load of about 90. Raise STORE_CASE_BUDGET_MS to 600 s:
it is a hang guard for fixed store work, not a speed assertion. The fake
carrier lifetime stays twice the budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keeps HACK-1440's store call counts in the xattr budget comment alongside
the HACK-1439 load measurements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant