Skip to content

Verify cache seeds without gating deploy smoke jobs - #435

Open
prk-Jr wants to merge 3 commits into
mainfrom
fix/verify-cache-seeds
Open

prk-Jr wants to merge 3 commits into
mainfrom
fix/verify-cache-seeds

Conversation

@prk-Jr

@prk-Jr prk-Jr commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Detect unpublished compiler, source and target cache seeds at the producer, retry transient cache-service failures within bounded limits, and stop immediately on compilation errors.
  • Require exact source/target-cache lookup hits even when the cache action returns early without applying fail-on-cache-miss.
  • Keep deployment release/smoke jobs independent of cache-probe availability. This extracts the unrelated hardening from Add pre-dispatch hooks and explicit request ingress fidelity #403 without its deploy-job coupling.

Changes

Crate / File Change
setup-rust-build-cache/tests/seed-compiler-cache.sh Verify native/WASM compiler-cache publication, accept confirmed existing hits, and retry cache write failures. Canonicalize the target root, validate the host path component, and reject symlinked target namespaces/debug directories before cleanup.
setup-rust-build-cache/tests/seed.sh Cover transient/permanent native and WASM failures, independent retry limits, existing hits, empty/malformed/error stats, compile failures, and symlink/traversal cleanup boundaries with outside sentinels.
setup-rust-build-cache/tests/{fixture,publication,run}.sh Verify lookup output wiring, all 32 hit-output combinations, action paths/keys and retry conditions; reject 38 workflow mutations; keep deployment jobs independent.
.github/workflows/deploy-action.yml Explicitly save/lookup-verify source and target seeds and retry each once. Retain fail-on-cache-miss diagnostics and enforce an exact lookup-hit assertion. Exhaustion fails only the cache-probe chain.

Closes

Closes #433

Test plan

Verified locally with Rust 1.98.1 after the independent review corrections:

  • cargo test --workspace --all-targets
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo fmt --all -- --check
  • cargo check --workspace --all-targets --features "fastly cloudflare spin"
  • cargo check -p edgezero-adapter-spin --target wasm32-wasip2 --features spin
  • bash .github/actions/setup-rust-build-cache/tests/run.sh — action, revision, review, fixture, workflow mutation and seed regressions pass.
  • ShellCheck all cache scripts/tests; actionlint; repository action-pin gate; git diff --check.
  • Independent compiler-cache and workflow investigations, separate fix-strategy proposals, and independent post-fix verification.
  • All 28 hosted checks pass on 3373552e, including cache publication/reuse and deployment smoke jobs, runtime tests and CodeQL.

Compiler seed retry backoff can add up to 180 seconds per target; source and target publication can each add 60 seconds. These waits affect only the cache-probe chain.

Review decisions

The review reproduced three publication/cleanup gaps and fixed all three:

  • actions/cache v5 restore can return success with cache-hit=false before parsing fail-on-cache-miss when feature availability fails. The final shell gate therefore requires true from one of the actual lookup outputs. Tests check both the expressions feeding that gate and its behavior; the assertion is not redundant.
  • Source archives previously saved only in a warning-only post-job hook. On a cold/evicted key, failed publication could leave the seed green while the source-restore consumer failed. The smoke producer now explicitly saves and verifies source archives with bounded retries before consumers start. The public action is unchanged; its later duplicate immutable save on an initial miss is benign.
  • Validating only the cleanup root allowed an intermediate target symlink to redirect rm -rf outside it. Cleanup now rejects namespace/debug symlinks and unsafe host components; tests preserve sentinels outside the confined directory.

The existing compiler retry strategy remains: pinned sccache v0.16.0 waits for cache writes before reporting compilation completion. Immediate stats inspection is valid, and an existing hit proves publication. Polling/restarting the server is unnecessary. Moving each attempt to another target directory would complicate compiler-cache identity and final archive ownership; guarded cleanup preserves the existing layout.

Local mutation tests simulate publication failures without relying on live outages. Hosted happy-path checks do not prove recovery during an actual cache-service outage.

Checklist

  • Follows CLAUDE.md; changes limited to cache CI/tooling.
  • Behavior and failure handling have committed regression coverage.
  • No runtime dependencies, routing, store wiring or application behavior changes.
  • Bash/YAML only; no Python added.
  • No credentials, secrets or bylines added.

@aram356 aram356 added this to the 202610 milestone Oct 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Verify cache seed publication without coupling deploy smoke jobs

2 participants