Repository navigation
Conversation
…nder load (HACK-1439) - checkout write guard: bound each git call (CHECKOUT_GIT_BUDGET_MS, killed and refused closed at the bound) and register the end-of-run afterAll with CHECKOUT_GUARD_HOOK_BUDGET_MS instead of Bun's 5 s default, which killed git mid-status and failed runs whose cases all passed. Guarded fixture keep-alive cases get a named budget too. - native-runtime-client: one bounded wait-for-complete-content helper backs waitForPid and waitForFileText; the cancellation case waits for a complete pid before it aborts (its fixture now publishes a torn handshake every run), and no case waits for existence and then reads. - private-env-file and native-compose-storage-witness-xattr-owner: a file-scoped named default budget replaces Bun's 5 s default and the fixed 30_000 literals; captured carriers outlive their case budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…load (HACK-1439) Four store cases (warm no-op up/restart, warm enrollment, adjacent verification) passed the 180 s budget in every run of a wave at a sustained load average of 110-190 on an 18-CPU host, while their assertions held; the slowest took 63 s at a load of about 90. Raise STORE_CASE_BUDGET_MS to 600 s: it is a hang guard for fixed store work, not a speed assertion. The fake carrier lifetime stays twice the budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keeps HACK-1440's store call counts in the xattr budget comment alongside the HACK-1439 load measurements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs HACK-1439. Based on
nextafter #480 (HACK-1422), which touched three of these files, landed.nextwas merged in again at 148ef8f, which brought in #494 (HACK-1440) and #497.Test-only. No assertion was removed or weakened; the only changes are bounds on waits, named Bun timeouts, and fixture timing that makes a race happen on every run.
Root causes
installCheckoutWriteGuardregistered its end-of-runafterAllwithout a timeout, so it ran under Bun's 5 s default hook timeout. Under host load the synchronousgit status --untracked-files=allpassed 5 s. Bun then killed git mid-read ("killed 1 dangling process"), git returned non-zero, and the guard refused with "git status failed" on a run whose cases all passed.native-runtime-client.test.ts, the "cancellation starts at most one native request and reaps only its owned child" case waited only for the pid file to exist, then aborted.writeFileSynccreates the file before it writes the digits, so the abort could SIGKILL the child in between. The file then stayed""andNumber("")gave0.private-env-file.test.tsandnative-compose-storage-witness-xattr-owner.test.tsdo a fixed amount of store file work (env layer lock, staged write and rename; durable generation publish/enroll/verify/retire) and wait on no timer. Under load they passed Bun's 5 s default or the fixed30_000literals while their assertions still held.Fix
tests/helpers/checkout-write-guard.tsgitcall is bounded byCHECKOUT_GIT_BUDGET_MS(60 s,spawnSynctimeout+ SIGKILL).git status did not finish within N ms. A real failure still givesgit status failed. The guard never treats an unreadable checkout as clean.afterAllhook is registered withCHECKOUT_GUARD_HOOK_BUDGET_MS(status bound + 30 s). A slow status therefore ends as the guard's named refusal, not as a Bun kill.checkout-write-guard.test.ts: the guarded fixtures' keep-alive cases getKEEP_ALIVE_CASE_TIMEOUT_MS. The latespawnSyncblocks their event loop while a coldbunchild starts, and under load that passed 5 s. This was found while running the file under load.tests/native-runtime-client.test.tswaitForFileContenthelper now backswaitForPidandwaitForFileText(fix(env): retry a private env read that races an atomic replace instead of refusing it as a symlink #480 added the latter for thefinishedfile).waitForFileis removed. Each of its three call sites now waits for complete content: the cancellation case's pid before it aborts, the abort-mode keeper pid, and the late-writerfinishedtext.native-runtime-metadata,-not-running-cli,-preflight): none waits for a file.private-env-file.test.tsandnative-compose-storage-witness-xattr-owner.test.tssetDefaultTimeout:ENV_STORE_CASE_BUDGET_MS(60 s) and fix(env): retry a private env read that races an atomic replace instead of refusing it as a symlink #480'sSTORE_CASE_BUDGET_MS. Every case and hook gets a named budget.STORE_CASE_BUDGET_MSis raised from 180 s to 600 s. In one wave at a sustained load average of 110-190, every run failed the same four store cases (warm no-op up/restart, warm enrollment, adjacent verification) at 180 s while their assertions held. The slowest case had taken 63 s at a load of about 90. Those measurements predate perf(native-compose): cut redundant store rechecks per proof step #494, which cut the store's call counts. The budget is a hang guard, so it is sized with room to spare.30_000literals in the xattr file are removed so the named default applies.FAKE_CARRIER_LIFETIME_MS. A slow case can then only time out cleanly; it cannot fail because the carrier deadline expired. The one case that mocksDate.nowkeeps its own clock deadline.setDefaultTimeoutapplies to this file's cases,test.eachrows and hooks, and does not leak into a later file in the same run.Tests and mutation proofs
exec sleep 30, budget 200 ms.spawnSynctimeout, the guard returns[]after 30 s (no refusal): the test fails.timedOutbranch, the message is "git status failed": the test fails.killed 1 dangling process+checkout write guard: git status failed.waitForPid(pidFile)with an existence wait fails 3 of 3 runs withExpected: <pid> Received: 0, the reported failure. The fixed version passes 3 of 3.Evidence (local, macOS, 18 CPUs; ambient load from other agents, plus a
yesload generator stopped in a trap)Each run is a full
bun test <file>with a private HOME, HACK_HOME and DOCKER_CONFIG, and the checkout write guard preloaded. The load average is the 1-minute value, sampled as each run ended.At the exact PR head 384bf68:
yesloadprivate-env-file.test.tscheckout-write-guard.test.tsnative-runtime-client.test.tsnative-compose-storage-witness-xattr-owner.test.tsuptimeread 110-290 while the fifth wave ranEarlier rounds of the same change:
Gates:
bun run check(includes privacy:check) passes.bun run typecheckandbun run --cwd packages/cli typecheckreport 0 errors.ultracite checkon the five changed files is clean. gitleaks 8.30.1 overorigin/next..HEADfinds no leaks. The CLI surface and agent guidance did not change, so no generated docs needed regenerating.Full suite: hosted CI run 40143709633 passed at the exact PR head 384bf68, including
testshards 1-4. All 20 checks are green. The earlier head 7dc0359 was also fully green.Gaps
🤖 Generated with Claude Code