feat(credentials): support database KEK rotation

[drew]

Summary

Improve key lifecycle management for the default encrypted database credential store.

Changes

• Support active and previous storage keys during staged rollouts.
• Add an opt-in, idempotent migration for stored credential wrapping metadata.
• Add deployment configuration, operational guidance, audit events, and regression coverage.

Testing

• mise run pre-commit
• mise run test
• mise run helm:test
• mise run ci (host-level Go checks are blocked by a pre-existing system gateway fixture and stale linter cache; changed-area checks pass)

Checklist

• Tests cover the new behavior and concurrent updates.
• User-facing and architecture documentation is updated.
• Generated Helm documentation is current.
• Commit is signed off for DCO compliance.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3557.docs.buildwithfern.com/openshell

[elezar]

The active/decrypt-only keyring model, envelope rewrap, CAS updates, backup guidance, and Helm Secret wiring are directionally sound. I found three issues that should be addressed before this is used to close FSR-DATA-KEK-ROTATION:

1. The rotation audit events do not reach structured OCSF JSONL output. emit_credential_rewrap_event builds a ConfigStateChange but calls tracing::info! directly (crates/openshell-server/src/credentials.rs:831-836). OcsfJsonlLayer only writes when an event has been placed in the OCSF bridge; the gateway path for that is emit_ocsf_event_routed. As written, operators may see shorthand logs while openshell-ocsf.log omits start/completion/failure and counts. Please use the structured routed emitter and add a JSONL capture regression test.

2. The operational sequence needs an explicit mixed-version prerequisite and rollback boundary. The previous gateway rejects the new keyring fields because its credential-storage config uses deny_unknown_fields. After the active key changes, new envelopes use it, and the new test correctly demonstrates that an old-key-only runtime cannot read migrated records (crates/openshell-driver-db-credstore/src/lib.rs:1620-1629). Please document that every replica must first run a keyring-aware gateway before adding these fields or switching the active key; no legacy replica may remain when the switch occurs; and binary downgrade is unsafe after the switch or partial rewrap until the old key has been made active and a reverse rewrap has completed and been verified.

3. The migration tests do not cover the paths needed to justify retiring the previous key. The production sweep pages at 100 records, but the new tests exercise one record, one injected conflict, and one startup migration. Please add coverage crossing a page boundary, interruption followed by idempotent resume, and concurrent sweepers or equivalent HA contention.

Local verification at db0f3e97: the credential-driver tests, mise run helm:test (159 tests), and mise run pre-commit passed. The focused server startup test could not link on my host because libz3 is unavailable. GitHub's required Branch Checks and Helm Lint contexts have not run and remain pending with “Waiting for /ok to test mirror.”

The smallest safe path is to fix and test structured OCSF emission, document the version/rollback constraints, add the migration boundary/failure tests, then rebase and run the required CI gates. The PR is currently still a draft; Seth's Red Hat feedback should also be incorporated before marking it ready.