Skip to content

test(runtime-core): prove relay refusals without socket timeouts and reap failed relay starts - #508

Merged
roodboi merged 2 commits into
nextfrom
wf/g112-relay-harness-einval
Oct 25, 2026
Merged

roodboi merged 2 commits into
nextfrom
wf/g112-relay-harness-einval

Conversation

@roodboi

@roodboi roodboi commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fix unit G112 for HACK-1342, items 1, 2 and 4. Test-harness only: one file changed (packages/runtime-core/tests/stream_relay_contract.rs). The relay (guest/stream-relay.c) is not changed, because the evidence shows it behaves correctly.

Item 3 (the 2 s deadlines in provider/graph/native/runtime/tests.rs) is left to #419 (HACK-1367). This PR does not touch runtime/tests.rs, exec_tests.rs, run_tests.rs or test_budget.rs.

Item Root cause Change
2. Relay::connect EINVAL (CI run 40119166731, stream_relay_contract.rs:108:14) The relay refuses an excess connection by accepting it and calling close() (stream-relay.c:489). For an unavailable target it calls release() after connect_target fails. Both can happen before the client's next syscall. Once the peer has closed a stream socket, macOS (XNU sosetopt) rejects SO_RCVTIMEO/SO_SNDTIMEO with EINVAL, because both directions are shut. Relay::connect unwraps set_read_timeout (line 105) and set_write_timeout (line 108), so it panics when the refusal arrives first. The hypothesis is confirmed. The relay is correct. The two refusal tests now call Relay::connect_refused(). It sets no socket timeouts. It waits up to REFUSAL_DEADLINE (10 s, as before) with poll and then still requires EOF or a reset (assert_refused). A peer that stays open fails the check (timeout), and so does a peer that sends data. Relay::connect still unwraps the timeout calls for streams the relay must keep, so EINVAL there is still a real failure.
4. Orphaned stream-relay-host on a readiness panic start_options spawned the child and ran its readiness assertions before any Relay existed, so a panic dropped the Child without killing it. The readiness checks moved into await_bare_ready. Until readiness passes, the existing PublisherStartup guard owns the child. Its Drop kills and reaps the child, and removes a fresh scratch root (never a root the caller passed in).
1. provider/publisher.rs probe-port race Already fixed on next by #399 (674a0a4): the fixture now runs --publish 0 and reads ready <port>. No probe listener is left. No change. It was re-verified under load (below).

Evidence

Root cause (macOS 26.5.1, Darwin 25.5.0)

A standalone Rust repro (not committed):

  • socketpair, peer still open: set_read_timeout = Ok(())
  • socketpair, peer dropped: set_read_timeout and set_write_timeout = Os { code: 22, InvalidInput }; read = Ok(0)
  • Unix listener, server accepts and then closes, as the relay does for an excess connection: 200/200 set_write_timeout calls return EINVAL
  • peer shut down only its write side: set_read_timeout = Ok(()). The error needs both directions to be shut, which matches XNU's check.

Parallel load on this macOS host (cargo +1.97.1, Zig 0.15.2 via mise, 8 concurrent processes × --test-threads 16, host load average ~105–116 on 18 cores)

Run Before (origin/next) After
Full stream_relay_contract binary 104/104 pass 104/104 pass
Focused (connection_limit, occupied_socket) 1994/2000 pass: 6 failures, all connection_limit_refuses_excess... with Os { code: 22 } at the set_read_timeout unwrap (:105:14) 2000/2000 pass
Item 1: provider::publisher lib tests (--features native-stream-relay) n/a (already fixed on next) 104/104 pass

The before failures appeared at line 105 locally and at line 108 on CI. Both are the same race; it depends on which setsockopt the relay's close() lands before.

New tests, each proven by mutation

  • refusal_check_accepts_a_peer_that_closed_before_the_check: drops the peer before the check. Mutation: restore the old pattern (set timeouts, then read) in assert_refused_within. Result: it fails deterministically with EINVAL.
  • refusal_check_rejects_a_live_peer and refusal_check_rejects_a_peer_that_sends_data (should_panic). Mutation: make the check return early (accept anything). Result: both fail. The new check does not weaken the refusal proof.
  • failed_relay_readiness_kills_and_reaps_the_child: a live child prints the wrong readiness line. The test then requires waitpid(pid, WNOHANG) = ECHILD (already killed and reaped) and the fresh root removed. Mutation: take the child out of the guard before the checks (the old ownership). Result: it fails with E_RELAY_STARTUP_ORPHAN, and the test kills the child itself.

Gates

  • cargo +1.97.1 fmt --check; clippy --all-targets -D warnings with default features (check:local) and with --all-features: clean
  • test:local (cargo +1.97.1 test --jobs 2): all pass (lib 1065 passed, 101 ignored)
  • stream_relay_contract with native-stream-relay: 29/29
  • bun run check (includes privacy:check) and bun run --cwd packages/cli typecheck: 0 errors
  • gitleaks 8.30.1 over origin/next..HEAD (gitleaks dir on a git log -p export): no leaks
  • Hosted CI at head 8cfc2a0 (run 40145181695): all 20 checks green, including test shards 1-4 and Candidate core (macos-latest) (optional feature contracts). This is the full-suite evidence.

Gaps

Round 2 (5752a32): stale scratch roots

The verifier hit one unrelated failure on next under 32-process load: fs::create_dir at line 52 returned EEXIST. Roots are named /tmp/<prefix>-<pid>-<n>, and a directory left by an earlier run whose PID is now reused collides. The new fresh_root(prefix) skips an existing name and panics on any other error. Every root site uses it, including the fixed-name hkr-netns, hkr-reservation, hkp-ack and hkp-private roots.

  • New test fresh_root_skips_a_stale_directory_with_the_next_name. A mutation that stops skipping makes it fail.
  • stream_relay_contract: 30/30 pass. 32 full-binary runs (4 rounds of 8 processes, --test-threads 16, load 92-154) all pass. fmt and clippy --all-targets --all-features -D warnings are clean. gitleaks found no leaks.
  • Not changed: the occupied-socket test still takes its "unavailable" port by binding and dropping a listener, so a parallel process could in theory take that port. I tried holding a bound socket that never listens. On macOS 26 a connect to such a port times out (ETIMEDOUT after about 7.8 s, with net.inet.tcp.blackhole 0) and is not refused, so that would turn the refusal into a timeout. The theoretical race is tracked separately.

Refs HACK-1342

🤖 Generated with Claude Code

…reap failed relay starts

HACK-1342 items 2 and 4 (item 1 is already fixed on next by #399; item 3
is left to #419).

Item 2: the relay refuses an excess connection (and one whose target is
down) by accepting and closing it, which can happen before the client's
next syscall. Once the peer has closed, macOS rejects SO_RCVTIMEO and
SO_SNDTIMEO with EINVAL, so `Relay::connect` panicked at the
`set_read_timeout`/`set_write_timeout` unwraps. The relay is correct.
The two refusal tests now use `Relay::connect_refused`, which sets no
socket timeouts: it waits for readability with a bounded poll and then
still requires EOF or a reset. A live peer or a peer that sends data
still fails the check.

Item 4: `Relay::start_options` now hands the spawned child to the
existing `PublisherStartup` guard until readiness is proven, so a failed
or panicking readiness check kills and reaps `stream-relay-host` (and
removes a fresh root) instead of orphaning it.

Refs HACK-1342

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 25, 2026

Copy link
Copy Markdown

HACK-1342

…ID (HACK-1342)

Relay test roots were named /tmp/<prefix>-<pid>-<n> and created with
create_dir().unwrap(), so a directory left by an earlier run whose PID is
now reused failed a fresh test with EEXIST. fresh_root() skips an existing
name, and every root site uses it, including the fixed-name ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@roodboi
roodboi marked this pull request as ready for review October 25, 2026 02:40
@roodboi
roodboi merged commit e5a6410 into next Oct 25, 2026
21 checks passed
@roodboi
roodboi deleted the wf/g112-relay-harness-einval branch October 25, 2026 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant