Skip to content

[rush-lib] Skip build cache writes when an operation's inputs changed during execution - #6086

Merged
Sean Larkin (TheLarkInn) merged 2 commits into
mainfrom
thelarkinn-fix-build-cache-poisoning-race
Sep 24, 2026
Merged

Sean Larkin (TheLarkInn) merged 2 commits into
mainfrom
thelarkinn-fix-build-cache-poisoning-race

Conversation

@TheLarkInn

@TheLarkInn Sean Larkin (TheLarkInn) commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes a build cache poisoning race. If a source file is edited or created while a build is running, Rush could store the post-edit outputs under the cache key computed from the pre-edit inputs. Any later build with the original inputs (local, or through a shared/remote cache) then restores wrong outputs. Native and daemon (rush-client/rushd) builds are both affected.

Root cause

An operation's cache key comes from the iteration's inputs snapshot, taken when the iteration is scheduled. CacheableOperationPlugin writes whatever is in the output folders after the operation finishes (OperationBuildCache.trySetCacheEntryAsync) and never re-verifies that the inputs still match the snapshot. Consumers' keys also embed the dependency's pre-edit state hash.

Fix

  • Capture at snapshot time. beforeExecuteIterationAsync runs right after the snapshot. For each enabled, cache-writable operation, it records:
    • a sha1 signature of each tracked input file's (size, mtimeNs, ino), from getTrackedFileHashesForOperation. Absolute dependsOnAdditionalFiles paths are resolved correctly.
    • the entry names of each in-repo folder that contains a tracked input.
  • Check before writing the cache entry. This applies to the normal and cobuild paths, for every operation that executed and was cache-writable. The inputs count as changed if:
    • the stat signature differs (a file was modified, deleted or replaced), or
    • new entries appeared in an input folder and git ls-files --others --exclude-standard finds untracked, non-ignored files among them. Only the new entries are checked, and the operation's output folders are excluded. If Git is unavailable or fails, the new entries are assumed to be inputs.
  • On change: Rush logs Input files changed while this operation was executing; not writing a build cache entry., skips the write, and sets isCacheWriteAllowed = false. The existing propagation then blocks consumers' writes too. Operation status is unchanged. The next snapshot sees the new content, so the next native, incremental or daemon iteration re-runs the operation.
  • The helpers are in the new InputFilesStatSignature.ts.

Overhead: at iteration start, one statSync per tracked input file plus one readdirSync per input folder, for each cache-writable operation. Operations that actually execute pay the same again before writing. Git is spawned only when new entries appear in an input folder, e.g. temp/ on a clean build, and only for those paths. Nothing is re-hashed.

Tests

  • InputFilesStatSignature.test.ts (10 tests):
    • unchanged inputs
    • modified file, same-size edit, deleted file
    • new file and new folder in an input folder
    • file created outside the input folders
    • absolute input paths
    • hasUntrackedGitFiles against a real temp Git repo: ignored files and excluded output folders return false, untracked files and folders return true
  • rush-lib targeted run InputFilesStatSignature|OperationGraph|OperationExecution|PhasedScriptAction|BuildCache: 149 passed / 0 failed.
  • rush build --to @microsoft/rush-lib, including lint, passes.

Linux validation (WSL Ubuntu-24.04)

The repro is a variant of cacherace.sh on a fresh synthetic workspace (8 projects, 500 ms per operation). It either edits p06/src/index.js or creates p06/src/zz-new.js after the snapshot but before p06 runs. It then reverts the source, deletes lib, and rebuilds natively from the shared cache.

client mode p06 on the reader build poisoned
native, before edit / new file restored / restored 1 / 1
native, after edit / new file re-executed / re-executed 0 / 0
daemon (RUSH_DAEMON=1 rush-client), before edit / new file restored / restored 1 / 1
daemon, after edit / new file re-executed / re-executed 0 / 0
  • Downstream (edit p06 mid-build, build --to p07): before, p07 was restored from a stale entry; after, p07 re-executes.
  • Normal build on a fresh workspace: 0 skip lines. After rm -rf lib, 8 of 8 operations were restored from the build cache.

Known limitations / optional follow-ups

  • Stat signatures can miss an edit that keeps the same size and inode and lands within the filesystem's mtime granularity. If that matters in practice, we could re-hash the inputs before writing.
  • Edits in the short window between computing the snapshot hashes and beforeExecuteIterationAsync aren't detected. Recording stat identity inside the snapshot itself would close that window.
  • A new file matching incrementalBuildIgnoredGlobs (but not .gitignore) conservatively skips the write.
  • New files matching a dependsOnAdditionalFiles glob in a folder that had no matches aren't detected.

This change is independent of #6064, which also touches CacheableOperationPlugin.

This came out of the automated rushd Linux performance/behavior analysis ("Rushd Hive").

Fixes #6073

… during execution

The cache key is derived from the iteration's inputs snapshot, but the outputs were written under that key without re-verifying the inputs. Record a stat signature of each cacheable operation's tracked input files right after the snapshot and refuse the cache write if it changed.

Fixes #6073

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

New files, absolute additional-input paths, and downstream operations can still produce poisoned cache entries.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 High severity

Open (2)
What changed in this PR

Adds a stat-based guard to prevent Rush from writing cache entries when operation inputs change during execution.

Changes:

  • Records tracked-input stat signatures before execution.
  • Rechecks signatures before normal and cobuild cache writes.
  • Adds focused signature tests and a patch change entry.
File Description
CacheableOperationPlugin.ts Integrates input-change detection into cache writes.
InputFilesStatSignature.ts Implements stat-signature generation and comparison.
InputFilesStatSignature.test.ts Tests signature behavior for file changes.
fix-build-cache-poisoning-race_2026-09-24.json Records the Rush patch change.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts Outdated
Comment thread libraries/rush-lib/src/logic/operations/CacheableOperationPlugin.ts
…, and resolve absolute input paths

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@TheLarkInn
Sean Larkin (TheLarkInn) merged commit 061f27d into main Sep 24, 2026
10 checks passed
@TheLarkInn
Sean Larkin (TheLarkInn) deleted the thelarkinn-fix-build-cache-poisoning-race branch September 24, 2026 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Closed

3 participants