[roodboi]

Summary

Two follow-ups from the #399 and #467 verifiers (HACK-1427).

1. run_isolated no longer hides a noisy child's failure behind the hang guard

packages/runtime-core/src/provider/test_executable.rs: run_isolated (from #387, extended by #399) used to pipe the child's stdout and read it only after the child exited. A failing child that wrote more than a 64 KiB pipe buffer blocked on its write until the 120 s ISOLATED_HANG_GUARD. The guard then reported a watchdog panic instead of the real failure.

• stdout and stderr are both piped and drained on reader threads while the parent waits. Each reader keeps a bounded tail (16 KiB) for the failure message and drains the rest, so the child never blocks on a full pipe.
• The bound covers the child's exit and the readers reaching end of stream. A descendant that keeps a copy of a pipe open is reported, not waited on forever.
• A failure message now includes both streams' tails. A passing child must still print at most 4096 bytes on stdout and must show the exact test <name> ... ok / 1 passed lines, as before.
• The second helper, provider/graph/startup/runtime/rebind/tests.rs:219, only delegates to this shared helper, so this change fixes it too. No other run_isolated exists.

New test isolated_child_failure_with_large_output_is_reported launches the ignored child noisy_failure_child. That child writes 160 KiB to each stream and then panics. The test requires the parent to fail well within the guard with the child's own failure, not "watchdog". The message must hold the panic text and the stderr end marker, and must be bounded.

2. The busy refusal over an interrupted rollback names the command that works there

src/commands/config-adopt.ts: once a switch's rollback is journaled (publication.phase: rolling-back), plain hack config adopt --recover refuses whatever the override holds, because completePublication accepts only switching. The generic E_LEGACY_ADOPTION_BUSY step still named it first whenever the override was not literal. The busy step for rolling-back is now PENDING_ROLLBACK_BUSY_NEXT_STEP, which names only hack config adopt --recover --rollback. The switching wording (HACK-1414) is unchanged. The text is fixed and value-free. docs/reference/native-compose-adoption.md is updated.

src/lib/native-compose-adoption-marker.ts: the classify() comment now matches pendingSelectionMatches: a pending stop is also admitted over a rolled-back publication. LegacyComposeAdoptionSelectionDetail gains switchPhase so the CLI can tell the two pending-switch phases apart.

Validation

All at head 2adbd9dc:

• Rust (cargo 1.97.1): bun run check:local passes (fmt and clippy -D warnings). The focused provider::test_executable test passes.
• bun run test:local was run twice on a heavily loaded local host (load average 50 to 130). Each run had 1051 passing tests and one different failure: dependency_slots::concurrent_claim_writers_serialize_without_overlapping_slots (its 3 s lock deadline) in the first run, and relay_owner::publication::dead_publication_recovery_requires_unchanged_receipt_and_socket in the second. Neither test touches run_isolated or any changed file, and both modules pass when rerun alone (dependency_slots 20/20, relay_owner::publication 11/11 three times). Hosted Candidate core on macOS and ubuntu is green.
• Adoption: both HACK-1427 tests pass, run focused and serially with --timeout 60000 (2 pass, 146 s), because these tests are load-sensitive (HACK-1367).
• gitleaks 8.30.1 over origin/next..HEAD: no leaks. bun run check (includes privacy:check): exit 0. bun run --cwd packages/cli typecheck: 0 errors.
• Hosted CI at 2adbd9dc: all checks green, including Candidate core (macos-latest, ubuntu-latest), test shards 1-4/4, docker-e2e shards 1-4/4, Config compiler (4 targets), Runtime state models, linux-process-lifetime, runtime-images and secret-scan.

Refs HACK-1427

Round 2 (rework after verifier: no PR existed)

The first implementation round committed ad916597 locally but never pushed it, so the verifier found no branch and no PR. This round reviews that commit, adds the report's exact repro, reruns every gate, pushes the branch and opens this PR.

Added test. CLI busy refusal over an interrupted rollback of a literal-bound switch does not name plain recovery once the override is moved aside (HACK-1427) sets up the exact reported state. It starts from the HACK-1318 literal fixture, journals the switch with a first --recover --stop, rewrites the receipt phase to rolling-back, and moves the bound override aside. --recover --stop --json, plain --json and --stop --json must each return the store's BUSY message plus PENDING_ROLLBACK_BUSY_NEXT_STEP, and must never name plain --recover. Plain --recover must refuse and leave the receipt unchanged.

The existing round-1 test, ...names only the recovery rollback that finishes it (HACK-1427), covers an active adoption with a reference-form override, rewritten to rolling-back. It shows that the named --recover --rollback actually completes the rollback and restores the originals. Plain --recover refuses there.

Mutation proof.

• TS: with the rolling-back branch of pendingSwitchBusyNextStep disabled, both HACK-1427 adoption tests fail (the busy message is the generic step). Restored, both pass.
• Rust: with the reader threads started only after the child exits (the old ordering), isolated_child_failure_with_large_output_is_reported fails after 120 s with exceeded its watchdog. Restored, it passes.

Known limit (reported separately). In the reported state, the bound override has been moved aside. --recover --rollback then refuses as E_LEGACY_ADOPTION_STATE, because the saved generated-input proof pins the file, including its ctime. Moving the file back does not satisfy the proof, because a rename changes ctime. The message now names the only command that can finish a rolling-back publication, and that command works whenever the bound inputs are intact. With the bound file moved away, though, no hack config adopt command finishes the rollback. That is outside this unit and is reported as a follow-up finding. (Resolved in round 3 below, HACK-1437.)

Round 3 (HACK-1437: a supported way out of an interrupted rollback with a moved override)

The round-2 verifier found that the known limit above was a real dead end. In the HACK-1318 literal fixture, journal the switch with --recover --stop, then move .hack/.internal/compose.env.override.yml aside. --recover --rollback journaled rolling-back and then refused with E_CONFIG_INVALID. Plain --recover refuses a rollback. The busy message named --recover --rollback again, and moving the file back did not help, because a rename changes ctime. This round fixes that path. It merges origin/next twice, first 1281184b and then 7dac4a03 after next moved again. Both merged cleanly and neither touches the files this round changes. The fix commit is 0e17a563 and the head is 80c9b1fe.

What the proof protects. A saved read binds each generated override by its bytes and file identity, ctime included. This protects adopted execution and switch completion, which must run exactly the inputs that preparation qualified and whose Compose config hash the retained containers carry. The binding also keeps hack doctor --fix and the env scrubs from removing a pinned file. Rollback restores the held legacy originals, archives the candidate and verifies the original resources. It never reads, runs, rewrites or removes the env override, which is derived from the managed env layers and rewritten by the next hack up. No data depends on it.

Change.

• src/lib/native-compose-adoption-projection.ts: readSavedLegacyComposeAdoptionProjection takes envOverride: "bound" | "released", with bound as the default. In released mode the env override is neither read nor compared. Every other generated proof stays exact: the .hack/.internal directory identity and the runtime override's bytes and identity. The saved config_files path list keeps the override's path, so the containers are still matched against the label Compose recorded. The new value-free savedLegacyComposeEnvOverrideChanged reports a moved or changed override when the other generated inputs still match.
• src/lib/native-compose-adoption-generation.ts: rollback reads in released mode in rollback() (from active) and in completeRollback (--recover --rollback from switching or rolling-back). Execution, leases, publication and completion still use bound. The new read-only store method pendingSwitchEnvOverrideChanged() backs the CLI hint.
• src/commands/config-adopt.ts: every pending-switch step names only a command that the state does not refuse for a known reason. The new PENDING_CHANGED_SWITCH_BUSY_NEXT_STEP covers a switching publication whose bound override was moved or changed, and names only --recover --rollback. A refused plain --recover now adds the step when its cause is known (rolling-back, or a moved or changed override). The text is fixed and value-free.
• docs/reference/native-compose-adoption.md: a new "Rollback releases the generated env override (HACK-1437)" section gives the written rationale. It covers what the binding protects, why rollback can release it, what rollback still checks, and the step for each pending state. The version 3 paragraph notes the exception.

Tests (tests/native-compose-adoption-generation.test.ts; long tests use the named budget ROLLBACK_RECOVERY_BUDGET_MS, HACK-1367):

• State 2, reachable without hand edits: CLI recovery rollback finishes a pending switch whose bound literal override was moved aside and back, without hand edits (HACK-1437). With the override moved aside, BUSY names only --recover --rollback (PENDING_CHANGED_SWITCH_BUSY_NEXT_STEP). Plain --recover refuses, journals nothing and names the same step. After the file is moved back (same inode, new ctime), BUSY names the literal step. The test then parses the message and runs the one command it names. That command must finish: the originals are restored, the selection is rolled-back, and the literal file is untouched and no longer pinned.
• State 1, rolling-back with the override moved aside: the round-2 test (renamed ..., and that rollback finishes (HACK-1427, HACK-1437)) now also checks that plain --recover names PENDING_ROLLBACK_BUSY_NEXT_STEP, then runs the named command to completion. The override stays absent and the moved file is untouched.
• CLI rollback of an active adoption releases a replaced generated env override while plain recovery stays refused (HACK-1437): --rollback finishes with a different file at the bound path and does not rewrite it.
• Store level: rollback releases a moved-aside generated env override that execution still refuses checks that loadActive still refuses while rollback succeeds. rollback still binds the generated runtime override when the env override is released rewrites the runtime override with identical bytes (a new ctime) and checks that rollback still refuses before anything moves.

Mutation proof (each reverted afterwards):

• M1, rollback reads with the full binding (released disabled): store-level release, state 1, state 2 and active-rollback tests fail at the rollback step. The strictness test still passes.
• M2, the moved/changed check removed from the busy step: state 2 fails at the BUSY message (it gets the generic step naming plain --recover).
• M3, the hint on a refused plain --recover disabled: state 1 and state 2 fail at the completion message.
• M4, the released path dropped from config_files: the store-level release, state 1 and active-rollback tests fail (rollback refuses on the binding).
• M5, the released generated-proof comparison skipped: the runtime-override strictness test fails with "unexpected adoption success".

Gates (fix commit 0e17a563; rerun at head 80c9b1fe after the second merge):

• At both commits: gitleaks 8.30.1 (gitleaks dir over a git log -p origin/next..HEAD export): no leaks. bun run check (includes privacy:check): exit 0, 34 existing complexity warnings. bun run --cwd packages/cli typecheck: 0 errors. CLI surface and agent guidance are unchanged, so no regeneration was needed.
• Focused (local host at load 20 to 170): all 5 HACK-1437 tests pass together at 0e17a563 (714 s) and again at 80c9b1fe (555 s). The HACK-1318/1414/1427/1273 group passes 8/10. The two failures, ...keeps it loadable until rollback frees it (HACK-1273) and v3 value-free env override binds the engine config hash... (HACK-1273), time out at their fixed 20 s budget inside store.prepare, before any changed path. They fail the same way on the unchanged PR head 2adbd9dc on this host, so they are load-sensitive (HACK-1367), not a regression. For local runs only, the named budget was raised so that the load could not end a test. The committed value is 240 s.
• Hosted CI at 0e17a563 (run 40124835555): all 20 checks green, including test shards 1-4/4, docker-e2e shards 1-4/4, Candidate core (macos-latest, ubuntu-latest), Config compiler (4 targets), Runtime state models, linux-process-lifetime, runtime-images and secret-scan.
• Hosted CI at 80c9b1fe: all 20 checks green (run 40128748032), with the same set as above. After that run, next moved three more commits to 246f1b4e (test(e2e): prove network probe budget and resolved-address fetch (HACK-1428) #484, test(runtime-core): prove claim-writer serialization without a per-writer 3 s budget #485 and fix(runtime-core): name a node job's macOS boot by its session UUID, not calendar boot time #486: runtime-core and e2e tests, none in this change's files). git merge-tree merges them cleanly. They are not merged here, so GitHub reports BEHIND again.

Remaining limits. Rollback still refuses when something it checks has changed: a running original, an edited held original or candidate, changed resources, changed managed env layers, the runtime override, or conflicting bytes at an original path. That refusal keeps the journal and evidence. Rollback keeps the managed-revision proof. Hack's own hack env set refuses while a switch is pending: a temporary probe, not committed, showed it fails with the adoption selection refusal. A managed env layer file edited outside Hack during a pending switch may still leave a similar dead end, because --recover --rollback journals rolling-back before it reads its inputs. That case is not reproduced here and is reported as a lower-severity follow-up finding.

Refs HACK-1437

🤖 Generated with Claude Code

[linear-code]

HACK-1427

HACK-1437