Skip to content

fix(e2e): serialize Kubernetes HA deployment tests - #3600

Closed
drew wants to merge 1 commit into
codex/fix-podman-e2e-workload-userfrom
codex/fix-kubernetes-ha-e2e-serialization
Closed

drew wants to merge 1 commit into
codex/fix-podman-e2e-workload-userfrom
codex/fix-kubernetes-ha-e2e-serialization

Conversation

@drew

@drew drew commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Serialize the Kubernetes HA E2E tests that mutate the same gateway deployment. Nextest launches each test in its own process, so the current in-process mutex cannot prevent one test from rolling the gateway while another is creating its sandbox.

Related Issue

Part of the #3583 stability campaign. The expanded Linux diagnostic run caught a gateway stream reset during the concurrent HA tests.

Changes

  • Restrict the HA test binary to one Nextest test thread in the Kubernetes E2E wrapper.
  • Document that deployment mutation tests require serial execution against a shared cluster.

Testing

  • bash -n e2e/rust/e2e-kubernetes.sh.
  • mise run pre-commit.
  • mise run ci.
  • Expanded GitHub Actions diagnostic for the combined stack tip.

Checklist

  • Conventional Commit with DCO sign-off.
  • Stacked with gh stack.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 23, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@drew
drew added this pull request to stack #3585 September 23, 2026 10:26
@drew drew added the automated-flake-find PRs produced by the automated E2E flake-finding campaign label Sep 23, 2026
@drew drew changed the title codex/fix kubernetes ha e2e serialization fix(e2e): serialize Kubernetes HA deployment tests Sep 23, 2026
@drew

drew commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator Author

Closing to narrow the automated flake investigation to the four selected fixes (#3583/#3588, #3598, #3604, #3608). The findings remain available here for later follow-up.

@drew drew closed this Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automated-flake-find PRs produced by the automated E2E flake-finding campaign

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant