[rush-client-core] Bound the daemon startup reservation and surface startup errors

[Sean Larkin (TheLarkInn)]

Summary

A <lockfile>.starting reservation that was never released wedged a workspace permanently. After that, every rush-client command waited about 16 s and then fell back to in-process Rush, daemon start refused to run, and the real startup error was never shown. This PR puts a bound on the reservation and makes its owner checkable. It keeps the existing safety goal: one workspace never gets two daemons. It also stops an invalid request-scoped RUSH_* value from replacing a healthy daemon.

Root cause

• runDaemonStartupAsync kept the reservation on purpose whenever the launcher missed the deadline or exited before readiness. Nothing ever reclaimed it: the record held only a token, with no owner PID and no age limit.
• tryConnectAsync rejected any endpoint while a reservation existed, including a daemon that was healthy and ready. The starter loop then waited out the full 15 s deadline.
• A single request with an invalid env value (for example RUSH_ALLOW_WARNINGS_IN_SUCCESSFUL_BUILD=yes) caused a planned env restart. RushDaemonHost closed the healthy daemon, and the successor threw in EnvironmentConfiguration.validate, exited with code 1, and left the reservation behind.
• launchClient never printed result.errorMessage. A restart failure also bypassed the in-process fallback.

Fix

rush-client-core (new DaemonStartupReservation.ts)

• The reservation is now a JSON record: token, createdAt, timeout, the owner PID and start time, and the launcher PID once it has been spawned. The owner is the starting client, and becomes the helper once the helper is spawned; the helper is recorded before the IPC handoff.
• The helper releases the reservation when the launcher it started exits without publishing an endpoint.
• A ready endpoint is accepted even while a reservation exists, if it answers hello/ping and its published lockfile owner matches the pong PID. Any client removes the reservation if its owner is dead; a live helper still releases its own.
• A client that holds the start mutex reclaims a stale reservation right away and starts the daemon, instead of waiting out the deadline. A reservation is stale if both its owner and launcher are dead, or if it is older than its own timeout plus a 60 s grace. Unrecognized records, such as token-only ones from older clients, have no owner to check, so only their file age counts.
• Startup failures now include the last error lines of <lockfile>.log from the current attempt. A live reservation's error reports the owner/launcher state. A launcher or helper killed by a signal is reported as was terminated (SIGKILL).
• README: the reservation rules are rewritten. If a suspended launcher resumes after its reservation was reclaimed, the transport's bind-time ownership check still rejects whichever daemon binds second.

rush-daemon

• New validateRequestRushEnvironment() applies the EnvironmentConfiguration.validate rules (booleans, unknown RUSH_* names, mutually exclusive cache overrides) to the request env without touching global state. WorkspaceRequestLifecycle runs it before planning a restart and returns a pre-execution failure result with the native message. The current daemon keeps running.

rush-cli-client

• Failed results now print ERROR: <errorMessage>.
• If executeWithDaemonRestartAsync fails with startupFailed, the client falls back to in-process Rush. This only happens before execution, so no work is replayed.

Tests

• rush-client-core:
  • A launcher that exits non-zero before readiness releases the reservation. The error fails fast and includes the log tail, and the next start succeeds.
  • A stale reservation with a dead owner and launcher is reclaimed, and startup begins at once.
  • An unrecognized reservation is reclaimed only once it passes the age bound.
  • A reservation whose launcher is still alive keeps blocking.
  • A ready daemon is accepted despite a reservation: the reservation is cleaned up if its owner is dead and left alone if the owner is alive.
  • The existing tests that start the daemon with a pre-bind hold still pass. The handoff test now waits for the live helper to release its reservation.
• rush-daemon:
  • RushEnvironmentValidation.test.ts.
  • WorkspaceReloadTierStatus.test.ts: an invalid RUSH_ALLOW_WARNINGS_IN_SUCCESSFUL_BUILD returns exitCode: 1 with the native message and no retryAfterRestart. The successor launcher is never called, and the next build runs on the same daemon PID.
• On Linux, rush test passes for rush-client-core, rush-daemon and rush-cli-client, and rush build for them reports no lint warnings.

Linux validation (WSL Ubuntu 24.04, synthetic 8-project workspace)

Before (unfixed rush-client):

warm2              exit=0   1.72s
badenv             exit=1  17.37s  rush-client: Daemon startup has an unresolved startup handoff at …starting; refusing another launch…
good1              exit=0  19.46s  (same message) Using in-process Rush.
good2              exit=0  19.14s  (same)
stop               exit=1   0.86s  Could not connect to daemon
start              exit=1  15.88s  unresolved startup handoff
-- SIGSTOP a starting daemon for 17 s --
after1             exit=0  18.89s  unresolved startup handoff … Using in-process Rush.
after2             exit=0  19.11s  (same)
start2             exit=1  15.72s  unresolved startup handoff

After (fixed client, same workspace):

=== invalid env ===
warm1              exit=0   6.82s
warm2              exit=0   1.85s
badenv             exit=1   1.13s  ERROR: Invalid value "yes" for the environment variable RUSH_ALLOW_WARNINGS_IN_SUCCESSFUL_BUILD. Valid choices are 0 or 1.
good1              exit=0   1.67s  (same daemon pid before/after)
good2              exit=0   1.79s
stop               exit=0   1.02s
start              exit=0   2.92s
(no .starting file left)
=== SIGSTOP a starting daemon for 17 s ===
cold build         exit=0          rush-client: Daemon startup timed out awaiting hello/ping readiness. Inspect ….log and retry, or use --no-daemon. Using in-process Rush.
after1             exit=0  10.23s  (launcher resumes, becomes ready, and is accepted despite its reservation)
after2             exit=0   1.99s
stop2              exit=0   1.36s
start2             exit=0   3.71s
(no .starting file left)

A separate analysis agent also checked all four wedge triggers on Linux, and every one recovered.

Known unrelated CI failures

Three rush-daemon phased-batching tests currently fail on main (since #6081 / #6068) with The summary cannot be prepared until after close() is called.. They fail the same way on this branch. #6095 fixes them, and I'll merge main again once it lands.

Fixes #6050

---

This PR came out of the automated rushd Linux analysis (board bug reports #16, #23 and #33).

[Copilot]

Copilot review overview

🟡 Changes recommended

Daemon-specific environment values remain unvalidated, and reservation updates have a check-then-replace race.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Prevent stale helper from overwriting reclaimed reservation · New
• Validate daemon configuration before restarting healthy daemon · New

What changed in this PR

Bounds daemon startup reservations, improves startup diagnostics, and prevents invalid request environments from unnecessarily replacing healthy daemons.

Changes:

• Adds verifiable, expiring startup reservations with launcher log diagnostics.
• Pre-validates request-scoped Rush environments.
• Prints daemon errors and safely falls back to in-process Rush.

File	Description
libraries/​rush-daemon/​src/​WorkspaceRequestLifecycle.ts	Validates environments before restart.
libraries/​rush-daemon/​src/​RushEnvironmentValidation.ts	Implements request environment validation.
libraries/​rush-daemon/​src/​test/​RushEnvironmentValidation.test.ts	Tests validation behavior.
libraries/​rush-daemon/​src/​test/​WorkspaceReloadTierStatus.test.ts	Tests preservation of the healthy daemon.
libraries/​rush-client-core/​src/​DaemonStartupReservation.ts	Adds structured, bounded reservations.
libraries/​rush-client-core/​src/​DaemonStartup.ts	Tracks launcher ownership and cleanup.
libraries/​rush-client-core/​src/​DaemonLogFile.ts	Formats startup log diagnostics.
libraries/​rush-client-core/​src/​connectOrStartDaemon.ts	Reclaims stale reservations and accepts ready daemons.
libraries/​rush-client-core/​src/​test/​connectOrStartDaemon.test.ts	Tests reservation recovery scenarios.
libraries/​rush-client-core/​README.md	Documents reservation lifecycle rules.
apps/​rush-cli-client/​src/​launchClient.ts	Prints failures and adds safe fallback.
common/​changes/​@rushstack/​rush-daemon/​startup-wedge_2026-09-24-01-30.json	Records daemon patch.
common/​changes/​@rushstack/​rush-client-core/​startup-wedge_2026-09-24-01-30.json	Records core patch.
common/​changes/​@rushstack/​rush-cli-client/​startup-wedge_2026-09-24-01-30.json	Records CLI patch.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.