Skip to content

Support in-place SSH key rotation for SSH fleets - #4458

Draft
peterschmidt85 wants to merge 3 commits into
masterfrom
issue_4450_ssh_key_rotation
Draft

peterschmidt85 wants to merge 3 commits into
masterfrom
issue_4450_ssh_key_rotation

Conversation

@peterschmidt85

@peterschmidt85 peterschmidt85 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Changing SSH-fleet keys currently requires re-creating the fleet, which cannot be done while it has active runs. Allow dstack apply to replace ssh_config.identity_file and ssh_config.proxy_jump.identity_file in place, including existing per-host overrides and changed key contents at the same path.

Authorize the replacement public key first. Apply updates the fleet configuration and existing instance keys in one database transaction. Public keys on hosts remain managed externally. Running tasks and services continue without host provisioning, container restarts, or run resubmission. Established SSH sessions and healthy service tunnels remain open. Revoking a public key prevents future authentication; it does not terminate an authenticated SSH session.

The apply plan identifies changed key fields:

$ dstack apply -f /e2e/fleet.dstack.yml -y
 Project        main             
 User           admin            
 Configuration  fleet.dstack.yml 
 Type           fleet            
 Fleet type     ssh              
 Nodes          1                
 Placement      any              

Found fleet rotation. Detected changes that can be updated in-place:
- Configuration properties:
  - ssh_config.identity_file
  - ssh_config.proxy_jump.identity_file

 NAME           NODES  GPU  SPOT  BACKEND       PRICE  STATUS  CREATED 
 rotation       1                 ssh                  active  12:44   
    instance=0         -          ssh (remote)         busy    12:44

Per-host changes show ssh_config.hosts[0].identity_file or ssh_config.hosts[0].proxy_jump.identity_file. ssh_config.hosts means the host list changed. Changing user, port, or the head-node address still requires fleet recreation, which is refused while the fleet is in use. Mixed key and connection-setting changes do not apply keys separately.

Head nodes below use proxy_jump; dstack gateways use type: gateway.

Scenario What happens
Fleet with no active runs Updates existing instance keys without provisioning the hosts again.
Fleet with no active runs and a running gateway Updates the fleet only. No service on the gateway uses this fleet; the gateway's own SSH key is independent.
Running task The task and server SSH connections continue. Provisioned hosts still use the project's SSH key; new server connections through the head node use the replacement proxy_jump key.
Running service with a gateway Background synchronization sends head-node keys in one batch per gateway replica. The gateway updates tunnel key files and saves its state once, preserving healthy tunnels, registrations, and Nginx configuration.
Running task with the SSH proxy The existing attach session continues. Fresh attaches use the updated head-node key after the existing 10-second cache expires.
Running service with the SSH proxy and a gateway Both the attach session and service tunnel continue. Gateway synchronization and SSH-proxy cache expiry happen independently.
  • dstack server: Use the fleet's existing lease fields to prevent fresh instance claims and let existing processing finish. Release the database connection between checks and wait up to 30 seconds before returning a retry error. Timeout, cancellation before saving, or a lost lease leaves stored keys unchanged. Concurrent fleet changes can also require a retry.
  • In-server proxy: Read the replacement head-node key on the next service request and update the tunnel key file. Reopen disconnected tunnels on demand.
  • dstack gateways: Apply returns before synchronization finishes. Backlog, an unavailable gateway, or failed updates can delay adoption; failed batch members retry during ordinary synchronization. One batch and one state-file save replace a request and full-state save per service replica. The gateway app must include this change; the normal server upgrade updates it unless that update is skipped or fails.
  • SSH proxy: Keep the existing 10-second credential cache. Revoking the old public key before cache expiry can prevent fresh attaches. Rapid retries can also trigger the head node's OpenSSH authentication penalties. Existing sessions continue.
  • CLI: Without the SSH proxy, users must also update the head-node key they supply through ~/.ssh/config or an SSH agent. For same-path replacement, a stale .pub file is rejected; it must match the replacement private key.

Keep the old public key authorized until gateways receive the replacement and SSH-proxy caches expire. Earlier revocation can prevent new connections or service-tunnel reconnects; dstack apply does not wait for either component.

Closes #4450

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Support in-place SSH key rotation for SSH fleets

1 participant