Running claims survive node loss: periodic snapshot replication behind a node lease

docs/deploy.md

@@ -143,14 +143,15 @@ gets the intersection of the two.
 | `capture_trim` (pool entry) | `false` | Trim the guest's copy-on-write disk before a promote or checkpoint of this pool's sandboxes (and of the clones of a template promoted from it), so the record carries only live data: blocks a build deleted — a `COPY` step's archive, a `RUN` step's caches — are no longer captured. sandboxd mounts the disk init resolved from `cocoon.cow` a second time at `/run/sandboxd-trim` through silkd as root, runs `fstrim` and unmounts, under the capture's transition lock (about 15 ms on a warm clone). A failed trim is logged and the capture proceeds, larger. A hibernated or archived sandbox is captured untrimmed. Config-owned like `egress`: `PUT /v1/pools` rejects it |
 | `egress_classes` | unset | named tenant egress layers: `[{"name": "desk", "egress": {…}, "egress_upstream_env": "DESK_UPSTREAM"}]`. A tenant names one with `egress_class` through the [tenant API](sandboxd-api.md#tenants-v1tenants) and its claims take that policy intersected with the pool's. `egress` is required; `intercept` stays a pool-rule flag. Reloadable: a class's policy reaches live claims on their next request. A class removed while tenants still name it leaves them with no egress and is logged at load and at reload |
 | `egress_upstream_env` (pool or egress class entry) | unset | names a node environment variable holding the default upstream proxy URL for the pool's or class's claims (class before pool, a claim's own entry before both); needs `egress_upstream`, and the URL must be on its `allow`. Read and checked at startup. Config-owned like `egress`: `PUT /v1/pools` rejects it |
+| `replicate_seconds` (pool entry) | 0 (off) | snapshot every running claim of this pool to the checkpoint store this often and point the claim's registry row at the snapshot, so another node restores the claim, memory and processes included, when this node is lost (see [running claims across node loss](cluster.md#running-claims-across-node-loss)). Each round pauses the guest for the length of a memory and disk capture. A restore resumes from the last committed replica and loses everything after it: up to the longer of the interval and one replication pass, plus the 5-second tick, plus one more pass, since a claim can run first in one pass and last in the next. A pass covers all the node's due claims of the replicating pools with bounded concurrency, so it grows with claim count and guest memory; the first round of a claim after a wake or adoption uploads the whole claim, so that pass is the long pass. The claim's row carries no retention deadline while it runs. Requires `archive_failover` and a `none`-lane pool, both checked at config load; a claim with volumes is skipped. Config-owned like `egress`: `PUT /v1/pools` rejects it. Restart only: a claim is replicated by its pool's setting from the moment it is claimed, so a reload that changes it is refused (a pool restarted with 0 drops its claims' rows and replicas on the first pass) |
 | `storage` (pool entry) | unset (cocoon's default, 10G) | size of the copy-on-write disk the pool's golden is cold-booted with, in cocoon's own `--storage` spelling (`"40G"`, `"40GiB"`, `"40Gi"`: Docker/Kubernetes units, all binary) and passed to it verbatim; every clone, fork, checkpoint branch and promoted template of the pool inherits it. At least cocoon's default of 10G. Equal sizes spelled differently build the same golden; a changed size rebuilds the golden on the next build, and claims already out keep their disk. Config-owned like `egress`: `PUT /v1/pools` rejects it |
 | `max_claims` | 0 (unlimited) | node-wide cap on live claims; claim/fork/branch requests beyond it answer 429 with the pool state unharmed (on a cluster a non-volume claim tries a warm-peer redirect first; a volume claim answers the 429 with no redirect) |
 | `audit_log` | false | append every relayed frame that opens an RPC — its op + addressing fields, never payloads — to `<data_dir>/audit.jsonl`; continuation `data`, `data_end`, `stdin`, and `stdin_close` frames pass through unrecorded. The file is size-rotated with one `.1` backup. Records are `{t, id, op}` plus whichever addressing fields the op carries (`argv`, `path`, `dest`, `from`, `to`, `url`, `session`, `port`), plus `method` (`GET`, `CONNECT`, `SOCKS5`, …), `decision` (`allow`, `deny`, `rate_limited`), `secret` (the ref name, never its value), `upstream` (the [upstream proxy](egress.md#upstream-proxies)'s `host:port`, when one carried it), `tenant`, and on an allowed connection `src_addr` and `dst_addr` (the node-side and destination `ip:port` of the upstream TCP connection, the tunnel target through an upstream proxy) on `egress` records, which a relayed frame cannot set; preview accesses record as op `preview`, one per request; a tenant change records as op `tenants` with the `added`, `changed` and `removed` names, never a token. An opening frame whose first line exceeds 4 KiB records as op `oversized` with no addressing fields |
 | `disk_reserve` | unset | `{"free": "100G", "dirs": ["/srv/sbx/run"]}`: while the filesystem of any listed directory (default `data_dir`) has less than `free` available, checked every 2 s, the node refuses new claims, forks, checkpoint claims, wakes of hibernated or archived sandboxes, hibernates, checkpoint captures and promotes with 429, and parks refill; the idle sweep and a lease-end archive leave their sandboxes running and retry on the next tick; `/v1/info` reports it as at capacity and the mesh sees no warm VMs on it. Running sandboxes keep running. On an XFS project quota the free space is the project's. List cocoon's `run_dir` when it sits on another filesystem, since the copy-on-write disks grow there. Size `free` to at least two sandboxes' worst-case growth (copy-on-write disk plus memory file each) and keep the refusal above your disk alert's warning level, so the warning comes first: on a 640G project with 40G disks and 8G memory files, `100G` refuses at 84% used |
 | `idle_hibernate_seconds` | 0 (off) | node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim is hibernated once it has had no open data-plane connection (relay, buffered exec, preview) and no egress request in flight for this long; the clock restarts when the last connection closes, so a long command is never cut short. The next call that reaches the guest wakes it transparently. Per-pool `idle_hibernate_seconds` does the same for that pool's claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice |
 | `archive_after_seconds` | 0 (off) | tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call that reaches the guest restores it transparently (a checkpoint restore's latency) with a fresh lease of the length the claim was granted (the server default of 5m when it asked for none); a lease moves only when a client calls [`/renew`](sandboxd-api.md#post-v1sandboxesidrenew). Keep `idle_hibernate_seconds` under the leases your claims use, or a woken sandbox that idles again is destroyed by the reap before it can hibernate, its archive already consumed by the wake; one still in use when its lease ends is destroyed the same way, unless it claimed `on_expire: archive`, which archives at the lease end whatever this setting. Requires `idle_hibernate_seconds > 0` and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool |
 | `archive_delete_after_seconds` | 0 (keep) | purge an archived claim's store checkpoint this long after it was archived; the claim is then gone for good. On S3, chunks and manifests remain until [offline reclamation](#reclaiming-s3-storage). On archive this retention window replaces the live claim deadline; 0 clears the deadline so the archive is kept forever. Same node-wide/per-pool split |
-| `archive_failover` | false | let another node adopt this node's archived claims when it leaves the mesh: each archive also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh`, all checked at config load. With the meta store down, archives, wakes and releases of archived claims fail and retry; running claims never touch it. Restart only |
+| `archive_failover` | false | let another node adopt this node's archived claims once its node lease expires, and the running claims of pools with `replicate_seconds`: each archive also writes a row (token SHA-256, owner, epoch, checkpoint id, retention deadline) to the shared PostgreSQL, and a `Lookup` that reaches any node takes the claim over there (see [archived claims across node loss](cluster.md#archived-claims-across-node-loss)). Requires `meta_store` with a `cell` (rows are scoped by it, so meshes sharing one database never adopt from each other), a `checkpoint_store` of kind `s3` and a `mesh`, all checked at config load. The node also renews a lease row every 5 s (see [node lease](cluster.md#node-lease)). With the meta store down, archives, wakes and releases of archived claims fail and retry, and replicated claims stop being served once the lease is overdue (see [running claims](cluster.md#running-claims-across-node-loss)); other running claims never touch it. Restart only |
 | `mesh` | unset | join a cluster ([Clusters](cluster.md)); unset = single node |
 | `pools[]` | — | warm pools, keyed by `(template, net, size)`. `warm` defaults to 4 only when both `warm` and `warm_max` are omitted; an explicit `warm: 0` is kept, so `warm: 0` under a `warm_max` starts the pool empty and grows it on demand, and `warm: 0` alone registers the key with no warm VMs (`PUT /v1/pools` never defaults `warm`); `net` is `none` or `egress`; `size` is a tier, below. Retune online without a restart via [`PUT /v1/pools`](sandboxd-api.md#put-v1pools) — omitted pools drain, and their goldens are deleted once they are idle (`warm: 0` keeps the golden). This is the **first-boot seed**: once a node takes a `PUT /v1/pools`, the applied set persists to `<data_dir>/pools.json` and overrides this section on every later boot (a startup log notes it); delete `pools.json` to return to config-owned pools. With a `meta_store` `cell`, the cell's set overrides it instead (see [shared pool set](#shared-pool-set)). Egress stays config-owned either way. A template re-pulled or re-imported under the same ref is picked up within a minute: the pool rebuilds its golden and replaces its warm VMs. See [state ownership](cluster.md#state-ownership) |
 

docs/sandboxd-api.md

@@ -578,7 +578,7 @@ is deleted; a pool kept with `warm: 0` keeps its golden.
 Answers the fresh `GET /v1/info` payload. 400 bad key, negative warm/idle,
 `warm_max` below `warm`, `idle_hibernate_seconds` on an egress pool, a
 negative archive duration or an `archive_after_seconds` not above the pool's
-`idle_hibernate_seconds`, duplicate pool, or a config-owned `egress`/`warmup`/`capture_trim`/`storage`/`egress_upstream_env`
+`idle_hibernate_seconds`, duplicate pool, or a config-owned `egress`/`warmup`/`capture_trim`/`storage`/`replicate_seconds`/`egress_upstream_env`
 field; 401 bad api
 token; 409 egress pool on a node without an egress attachment; 503 the
 `meta_store` cannot take a cell's write.
@@ -695,7 +695,8 @@ URL; releasing the sandbox ends it early either way.
 has no `preview_listen`. The signed token embeds the sandbox id, port, and
 owner `advertise_addr`, so any node's preview listener can serve it (forwarding
 to the owner's main listener) and a released sandbox's URL simply stops
-resolving — no revocation list. See [deploy](deploy.md#preview-urls).
+resolving — no revocation list. A request for a replicated claim answers 503
+`lease lost; retry` while its node's lease is overdue. See [deploy](deploy.md#preview-urls).
 
 ## POST /v1/sandboxes/{id}/checkpoint
 
@@ -957,7 +958,11 @@ before the 101, reach silkd in order; the Go and Python SDKs send their first
 frames that way over plain HTTP, so a proxy in front of a plain-HTTP endpoint
 must forward them too (Go's `httputil.ReverseProxy` drops them; Caddy forwards
 them). 426 without the upgrade header, 404 unknown sandbox, 409 a failed
-sandbox, 502 guest unreachable.
+sandbox, 502 guest unreachable, 503 `lease lost; retry` for a replicated claim
+while its node's lease is overdue (see
+[running claims](cluster.md#running-claims-across-node-loss)). A relay of a
+claim of a replicating pool that is already open when the lease turns overdue
+is closed, whether or not it has a replica yet.
 
 ## GET /v1/sandboxes/{id}/ports/{port}
 
@@ -987,7 +992,8 @@ fork in flight, answers `409` at once; an open passive relay still holds the san
 400 a port outside 1-65535; 401 missing bearer token; 404 unknown sandbox or
 wrong token; 409 a failed sandbox or a passive relay to a paused one; 426 without the upgrade
 header; 502 when the guest cannot be reached on that port — no listener, or a
-hibernated sandbox whose restore failed. The node log carries which.
+hibernated sandbox whose restore failed. The node log carries which; 503
+`lease lost; retry` for a replicated claim while its node's lease is overdue.
 
 ## POST /v1/sandboxes/{id}/exec
 
@@ -1012,16 +1018,19 @@ does not exist are the caller's, not 502s; 401 missing bearer token; 404 unknown
 413 when stdout+stderr exceed 8 MiB; 503 with `Retry-After: 1` when the
 node's buffered-exec memory is exhausted (64 MiB of output across every
 buffered exec in flight; the command is killed); 502 guest unreachable or any other
-silkd error; 409 a failed sandbox; 504 when the command outlives `timeout_seconds`. A client that hangs up first gets no reply and the command is killed. A hibernated sandbox wakes transparently like on the relay.
+silkd error; 409 a failed sandbox; 503 `lease lost; retry` for a replicated claim
+while its node's lease is overdue; 504 when the command outlives `timeout_seconds`. A client that hangs up first gets no reply and the command is killed. A hibernated sandbox wakes transparently like on the relay.
 
 ## GET /v1/sandboxes/{id}/owner
 
 Auth: the sandbox's own token. Answers `{"owner_addr": "host:port"}` when
 this node owns the sandbox, 404 otherwise. With
 [`archive_failover`](cluster.md#archived-claims-across-node-loss), a node that
-does not hold the sandbox adopts it when its archive row names an owner that
-left the mesh, and answers 200; 503 with `Retry-After` when the meta store is
-down. With `client_advertise` configured,
+does not hold the sandbox adopts it when its row names an owner whose node
+lease expired, and answers 200; 503 with `Retry-After` when the meta store is
+down. A row whose owner's lease is live is never adopted: 404. A node whose
+lease is overdue answers 404 for the replicated claims it holds, so the
+`Lookup` scatter moves on. With `client_advertise` configured,
 `owner_addr` is that node's full HTTP(S) origin instead; the same contract applies
 to claim and fork responses. Used by the SDK's `Lookup` scatter.
 

e2e/wire_parity_test.go

@@ -41,7 +41,7 @@ var wirePairs = []struct {
 	{"checkpointResponse", false, server.CheckpointResponse{}, nil},
 	{"checkpointClaimRequest", true, server.CheckpointClaimRequest{}, nil},
 	{"checkpointListResponse", false, server.CheckpointListResponse{}, nil},
-	{"checkpointRecord", false, types.Checkpoint{}, []string{"key", "tenant", "policy_source", "no_egress", "guest_env", "archive"}},
+	{"checkpointRecord", false, types.Checkpoint{}, []string{"node", "key", "tenant", "policy_source", "no_egress", "guest_env", "archive"}},
 	{"volumeListResponse", false, server.VolumeListResponse{}, nil},
 	{"VolumeInfo", false, types.VolumeInfo{}, nil},
 	{"VolumeState", false, types.VolumeState{}, nil},
@@ -59,7 +59,7 @@ var wirePairs = []struct {
 	{"SandboxSummary", false, pool.SandboxSummary{}, []string{"token"}},
 	{"sandboxListResponse", false, server.SandboxListResponse{}, nil},
 	{"poolUpdate", true, server.PoolUpdateRequest{}, nil},
-	{"PoolSpec", true, config.PoolSpec{}, []string{"egress", "warmup", "egress_upstream_env", "capture_trim", "storage", "net", "size"}},
+	{"PoolSpec", true, config.PoolSpec{}, []string{"egress", "warmup", "egress_upstream_env", "capture_trim", "storage", "replicate_seconds", "net", "size"}},
 	{"tenantsUpdate", true, server.TenantsRequest{}, nil},
 	{"tenantUpdate", true, server.TenantRequest{}, nil},
 	{"TenantSpec", true, reflect.Zero(reflect.TypeOf(server.TenantsRequest{}.Tenants).Elem()).Interface(), []string{"token"}},

sandboxd/archives/archives.go

@@ -1,4 +1,4 @@
-// Package archives keeps one cluster-wide row per archived claim, so another node can adopt the claim once its owner is gone.
+// Package archives keeps one cluster-wide row per archived or replicated claim, so another node can adopt the claim once its owner's lease expires.
 package archives
 
 import (
@@ -11,7 +11,7 @@ import (
 var (
 	// ErrUnavailable answers a call the registry cannot serve now; a retry may succeed.
 	ErrUnavailable = errors.New("archive registry down")
-	// ErrLost answers a compare-and-set whose row is gone, at another epoch, consumed, or held by another owner.
+	// ErrLost answers a compare-and-set whose row is gone, at another epoch, consumed, or held by an owner whose lease is live.
 	ErrLost = errors.New("archive row moved on")
 )
 
@@ -33,16 +33,20 @@ type Record struct {
 type Registry interface {
 	// Put writes rec for rec.Owner as a live row at a new epoch and returns it; ErrLost when another owner holds the row.
 	Put(ctx context.Context, rec Record) (int64, error)
-	// Take moves the live row at epoch expect to owner at a new epoch and returns it.
+	// Replace writes rec over the live row of rec.Owner at epoch expect, at a new epoch; with expect 0 it is Put.
+	Replace(ctx context.Context, rec Record, expect int64) (int64, error)
+	// Take moves the live row at epoch expect to owner at a new epoch and returns it; another node's row moves only once its owner's lease expired.
 	Take(ctx context.Context, id string, expect int64, owner string) (int64, error)
 	// Consume turns the live row at epoch expect into a tombstone at a new epoch and returns it.
 	Consume(ctx context.Context, id string, expect int64) (int64, error)
 	// Delete removes the row only while it is at epoch.
 	Delete(ctx context.Context, id string, epoch int64) error
 	// Get returns the row of id; false when there is none.
 	Get(ctx context.Context, id string) (Record, bool, error)
-	// Sweepable returns the rows past their deadline, consumed, or held by owner, without their claims.
+	// Sweepable returns the rows held by owner, consumed, or past their deadline with an owner whose lease expired, without their claims.
 	Sweepable(ctx context.Context, now time.Time, owner string) ([]Record, error)
 	// Rows returns the rows of ids by id, without their claims.
 	Rows(ctx context.Context, ids []string) (map[string]Record, error)
+	// Renew extends node's lease to ttl past the registry's clock.
+	Renew(ctx context.Context, node string, ttl time.Duration) error
 }