Repository navigation
Running claims survive node loss: periodic snapshot replication behind a node lease - #366
Merged
Merged
Conversation
CMGS
force-pushed
the
feat/363-snapshot-replication
branch
from
October 18, 2026 07:21
e4d69f3 to
3a60473
Compare
This was referenced Oct 18, 2026
Add sandboxd_nodes(cell, node, lease_until) and Registry.Renew, one upsert per node on the database clock. Take moves another node's row only once its owner holds no live lease, in the same statement as the epoch compare-and-set, and Sweepable lists a foreign expired row only for an owner without a lease. Open adds the nodes table to a database that already has the archives table.
Config load rejects it unless archive_failover is on and the pool is on the none lane. It is config-owned like storage: PUT /v1/pools refuses it, the pool seed hash ignores it, and a reload reports it as changed.
A pool with replicate_seconds snapshots each running claim through the existing capture path (pause, memory and disk, resume, publish) as a hidden replica record, puts the claim's registry row on it, then deletes the previous replica. A failed round logs a warning and retries next interval. Rounds run on the refill/maintenance budget and start none while the node is fenced. Adoption now rests on the node lease alone. Take refuses another node's row while its owner holds a live lease, in the same statement as the epoch compare-and-set, so the memberlist ownerAlive gate, WithArchiveOwners and mesh.Alive are gone. The adopter restores the claim from the replica through the archive wake, memory included. Each node renews its lease every 5 s for 15 s. A lease outlives a missed renewal, and the fence trips 3 s before it ends (12 s after the last renewal's start), so one lost renewal never fences while two do before any adopter may take the row. A fenced node answers 503 "lease lost; retry" on the relay, exec, ports and preview of its replicated claims and destroys nothing; a node starts fenced. The first successful renewal reads the rows of its replicated claims: rows still its own are served again, rows adopted elsewhere or consumed by a finished adopter drop the claim and remove its VM. Lifecycle: release consumes the row, then deletes replica and tombstone; hibernate leaves the row on the last replica; archive puts over the row and deletes the replica (its rollback now restores epoch 0, since the abandon deletes the row); a lease expiry drops row and replica before the reap destroys the claim; a restart keeps its own live row, takes the checkpoint a crashed round left on it, and drops a claim whose row another node holds or a finished adopter consumed; turning replicate_seconds off retires the rows. dropMoved is folded into dropAdopted, which now removes the local VM. Hot path: the claim path and data plane gain one atomic load. Per node, one lease upsert every 5 s and one in-memory scan of the claims every 5 s; per replicated claim per interval, one capture pause, upload, row read, row write and delete.
Extend the failover section of cluster.md with the node lease, the replication round, restore, lifecycle and self-fencing; add replicate_seconds and the lease to deploy.md; document the lease rule on /owner and the 503 lease-lost answer in sandboxd-api.md.
A parked keep-alive relay runs later RPCs back to back, and port and preview streams run for as long as the client keeps them, so none of them passes the claim-resolution gate again. WakeAgentSocket now returns a fence channel that the relay watches, DialPort and PreviewDial wrap their conns with it, and the channel closes when the fence trips; unfence installs a fresh one. An unreplicated claim gets none. Replicated state is an atomic on the sandbox, so resolution takes no mutex.
A round whose Put reply is lost left the claim with no record of the checkpoint the row now names, and deleted that checkpoint. If the partition went on, another node adopted a row whose checkpoint was gone and the returning owner then dropped its live VM: irreversible loss from one fault, and on the first round the claim was not even fenced. A round now persists ReplicaPending before the Put, deletes the checkpoint only on ErrLost, and leaves an unknown outcome to settle: the next round, release, archive, wake, the hourly sweep and a restart read the row. A row on the pending checkpoint promotes it; another owner's row drops the claim; an absent row means the adopter finished when the checkpoint is gone, and a never-committed Put when it is still there. A claim with a pending intent counts as replicated for the fence.
A replica row carried the retention deadline of its Put. A hibernated claim, which no round refreshes, saw that deadline pass while its owner was partitioned or down; another node's departed-owner sweep then purged the checkpoint and the row, and the returning owner dropped the claim. Expired rows were also refused for adoption. Replica rows now have no deadline. Retention starts where the claim leaves the node's care: the archive Put sets it as before, and adoption sets the adopter pool's window when the row has none. The rows of a node that never returns stay until someone adopts them.
…fence The renew loop ran unfence inline, so long Transition waits or VM removals there stopped renewals; the lease lapsed, an adopter took the row, and the unconditional Store(false) then let both nodes serve. Unfence now runs beside the renew loop, which keeps renewing. For each replicated claim it takes the row for this node in one compare-and-set, which the row lock serializes against adopters, retrying once on a stale epoch; a lost row forgets the claim at once and removes its VM in the background. The fence lifts only when all of that finished inside ttl-margin of the renewal that started it; otherwise the node stays fenced and the next renewal tries again.
AgentSocket had no fence check, so /owner kept answering 200 for a claim every other call on the node answered with 503, and the Lookup scatter kept binding to it. The node now answers 404 there and the scatter moves on; once the lease has expired another node adopts the claim.
The pool's replicate setting is read by the claim's policy key, as capture_trim is, so a claim cloned from a template promoted out of a replicating pool follows that pool. A replicated claim that gains volumes retires its replica instead of leaving an adoptable stale row. A retire that raced an archive finds no replica left and leaves the archive row alone (the guard is in dropReplica; this adds the tests).
…r answer State the fence guarantee with its precise conditions, the retention rule of replica rows, the unfence confirmation, and the /owner answer of a fenced node; align the lease, replicate_seconds and archive_failover comments with what the code does.
A round that outlived an adoption could Put after the adopter had woken the claim and deleted its row; the upsert inserted a fresh row owned by the old node, and its next renewal confirmed that row and lifted the fence while the adopter served. Registry.Replace is a compare-and-set on the row at the held epoch and owner (with epoch 0, the first replica, it is Put), and a replicated claim's archive writes through it too, since the same hole applied to a row the claim already had. A refused write reads the row: a row this node still owns only had a stale epoch and is refreshed; an absent or foreign row means the claim moved and its copy and VM are dropped.
…one atomic A stream resolved before a claim's first replica got no fence channel and survived the fence, because replicated was derived from ReplicaCk. A claim is now replicated when its policy pool replicates it (or already holds a replica from a journal), decided once at add time into a plain field, so an unreplicated claim reads nothing on the data plane and a replicated one reads a single atomic pointer: the fence state, which carries both the flag and the channel and is replaced on trip and on lift. replicate_seconds becomes restart-only, so the decision cannot go stale: a reload that changes it is refused with the pool named. A node that starts fenced lifts the fence only after its first confirmation, so the adopter's wake of an adopted claim of a replicating pool waits for that too. An unreplicated relay, port or preview connection no longer allocates a closure for the fence. CloseOnFence moves above the unexported functions.
A fence trip that fired between the window check and Store(false) found the node already fenced and did nothing, and the lift then ran past the lease with no timer left to trip it. The lift now replaces the fenced state this renewal saw, only if it is still current, and trips again when the window closed before it landed; the earlier window check goes, since the lift carries it.
…streams The fence state is initialised in every manager and a claim is replicated by pool policy, which needs archive_failover, so a journal that still holds replica_ck after failover was turned off no longer reaches a nil fence.
…g checkpoint An hourly sweep that reached reconcileOwnRow while a round's Put outcome was unsettled saw a row on a checkpoint the claim did not name and deleted both; the settle then found the row absent with a held epoch, took the claim as moved and destroyed a running claim no other node held. The pending checkpoint now counts as referenced by the sweep, is pinned, and is skipped by orphan reclaim at restart.
…enced After two refusals on a row whose owner was still this node, confirmOwnership fell through to false and unfence destroyed a claim it owns. It now returns an error, so the node stays fenced and the next renewal retries; only an absent row or another node's row drops the claim.
…d align the docs forgetAdopted logged under pool.dropAdopted and a var pair used a tuple declaration. The docs now state which registry writes are compare-and-sets and which stay owner-guarded upserts, the loss bound as the time since the last successful round, the 404 of /owner on a fenced node, the fence as decided by pool policy with its real hot-path cost, replicate_seconds as restart-only, and that an open relay of a replicating pool's claim closes whether or not it has a replica yet.
A client that cancels while its request wakes a sandbox made wakeGuest log the failed dial at error level although the wake completes. A dial that fails with context.Canceled while the request context is canceled is the client leaving and logs at debug; every other dial failure stays an error.
…a left at 0 costs A reload that changes replicate_seconds is refused, so the failover docs no longer say a reload turns it off. A claim that holds a replica when the node restarts with replication off stays replicated for the fence until the next restart: during a registry outage it answers 503, an availability cost only.
At a 30 s interval hardware lost 33.1 s: a round starts up to an interval plus the 5 s tick after the previous one and commits only when its capture, upload and registry write finish. State that bound instead of the interval alone.
wakeArchived consumes the row at the epoch it took, but commitWake left that epoch on the claim. The first replica round of the woken running claim then wrote through Replace at the consumed epoch, found no live row, and the refused-write settle destroyed the claim: every adopted-and-woken claim, and every archive-then-wake, in a replicating pool. commitWake now clears the epoch with the checkpoint, as consumeArchive does, and its rollback restores both. With no epoch held the round upserts, which also revives this node's own tombstone when its delete failed.
A claim is due when its last round started an interval ago, and a new pass cannot start while the previous one runs, so the gap between two round starts is the longer of the interval and one pass. State the bound with that term.
Reconcile runs before the lease loop, so a node adopting during it can move a claim's row and commit its own replica after the start-of-reconcile read. The reclaim now reads the candidates' rows right before deleting and skips one whose row names the checkpoint or belongs to another node; a failed read skips the reclaim until the next restart.
A round uploads only changed chunks and soft-deletes the previous replica, so its unique chunks live through the store grace plus a sweep. The first round after a wake or adoption is a full upload. A dead owner can also leave a committed pending or previous replica, whose live metas pin their chunks.
sandboxd no longer sweeps the S3 store, so a superseded replica's unique chunks and a dead owner's replica records stay until the operator reclaims them.
The checkpoint meta records the publishing node for archive and replica captures. The restart reclaim skips a checkpoint another node published, which closes the window where an adopter has published but not yet registered its checkpoint when the re-read finds no row. A meta without a node, published by an older build, is skipped too: crash residue from an older build leaks rather than risking a delete.
A second slow confirmation can start before the delay is removed and holds the fence for up to one lease TTL; the old wait of two renew periods made the test flaky (1 in 1800 runs under load, synctest deadlock on exit).
CMGS
force-pushed
the
feat/363-snapshot-replication
branch
from
October 18, 2026 07:52
3a60473 to
d882187
Compare
The teardowns were queued until every replicated claim was confirmed, so a registry error on a later claim returned early and dropped the queue; the forgotten claim's VM, already off the claim table and journal, then ran until the next restart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #363.
Builds on #364 (S3 chunked store, merged); #365 builds on this branch.
What
A pool entry gets
replicate_seconds(0 = off). Every interval each running claim of the pool is checkpointed through the existing capture path (pause, memory and disk, resume, publish) as a hidden replica record, the claim's registry row is moved onto it, and the previous replica is deleted. If the node is lost, another node adopts the row once the owner's node lease has expired and restores the claim from the replica through the archive wake, memory and processes included.archive_failoverand anone-lane pool (checked at load); config-owned likestorage(PUT /v1/poolsrejects it). Restart only: a claim is replicated by its pool's setting from the moment it is claimed, so a reload that changes it is refused.sandboxd_nodes(cell, node, lease_until), created at start when missing.Registry.Renewis one upsert per node on the database clock.Takemoves another node's row only when its owner holds no live lease, in the same statement as the epoch compare-and-set;Sweepablelists a foreign expired row only for an owner without a lease. Memberlist liveness is no longer consulted (OwnerAlive,WithArchiveOwners,ownerGoneandmesh.Aliveare removed); one rule covers archived rows and replica rows.Registry.Replace, a compare-and-set on the row's epoch and owner. A row another node took or deleted refuses the write, so a round that outlived an adoption cannot recreate the row; the refusal reads the row: one this node still owns only had a stale epoch (refreshed), an absent or foreign row drops the claim and its VM. Owner-guarded upserts remain only for the first write of a row (no epoch held: the first replica, the archive of a claim with no row) and the re-arm of the owner's own tombstone.ReplicaPendingbefore the registry write and delete the checkpoint only when the write is refused; an unknown outcome is settled by reading the row (next round, release, archive, wake, hourly sweep, restart). The sweep, the pin set and orphan reclaim know the pending checkpoint.nodein its meta), and a restarting node reclaims only orphans it published, after re-reading their rows. A node that restarts with a lapsed lease can be adopted from during its own reconcile, and it must never delete the adopter's fresh checkpoint.lease lost; retryon relay, exec, ports and preview of its replicated claims, answers 404 on/owner, closes every open stream of those claims, and destroys nothing. A claim is replicated when its policy pool replicates it, decided when the claim is added, so a stream opened before the first replica is covered. The fence is one atomic state (flag and channel). A node starts fenced; after a renewal it confirms each replicated claim with one compare-and-set Take beside the renew loop, forgets a claim another node holds (VM removed in the background), stays fenced if a claim of its own keeps refusing, and lifts the fence by compare-and-set on the state it saw, tripping again if more than 12 s passed since that renewal started.Why
Phase 1 (#353) covers archived claims. A running claim never touched the registry, so a partitioned owner that kept serving could race an adopter. The lease makes adoption and self-fencing agree on one clock.
Contract
Hot-path cost
Claim path: one map lookup per claim added on a failover node. Data plane: resolving a claim of a pool that does not replicate reads one plain field and no atomic; a replicated claim adds one atomic load of the fence state and its stream one watcher goroutine. Nothing per frame. Per node: one lease upsert every 5 s and one in-memory claim scan every 5 s. Per replicated claim per interval: one capture pause, an upload of the changed chunks plus a manifest and a meta (#364), one registry read, one write, and one soft delete of the previous replica. Rounds share the maintenance budget with refill. The first round after a wake or adoption uploads the whole claim again, and that is the long pass in the loss bound. The chunks only a superseded replica names stay in the bucket until the operator's reclaim runs (
docs/deploy.md→ Reclaiming S3 storage); with replication, that is the fastest-growing part of the store.Tests
Registry contract on PostgreSQL and the in-memory fake (take waits out a live lease, sweepable spares a live owner, renew extends, cell-scoped leases, upgrade adds the nodes table, replace is a compare-and-set); replication round (publish, move row, drop previous, failed capture, refused and lost put replies, a round that outlives an adoption, registry outage, due/skip rules, retire); lifecycle (release, hibernate, archive, expiry, restart variants, sweep, delete marker); self-fence (open port, passive, preview and relay streams of a replicated claim close when the fence trips, including one opened before the first replica; an unreplicated claim's stay open; new streams work after unfence), the lift race, a slow or contested confirmation, and recovery with two managers on one registry and store; config, reload (restart-only), setpools and server mappings. Each new test was mutation-checked.
Hardware acceptance
On the chunked store (stacked with #364 and #365)
Two bare-metal nodes (one cell), PostgreSQL registry, SeaweedFS 4.48, cocoon v0.7.3,
replicate_seconds30.GET /v1/checkpointsstayed at 0.15–0.20 slease lost; retry; B adopted at 41.384, first success 41.907 (a 522 ms wake): no overlap. The unreplicated claim on A answered 113 × 200 during the cut. After the heal, A unfenced and forgot the claimnode=b), A restarted with its old journal: it dropped both claims ("another node holds it"), reclaimed nothing, and the bucket's meta set and all 1,424 keys were unchanged. The archived claim then woke from B's checkpoint (599 ms) with its data intactOn the old store (before stacking)
Two bare-metal nodes (A, B), PostgreSQL registry, S3-compatible store,
replicate_seconds30, sandboxd at 0fe9d7d (e4d69f3 adds docs only).lease lost; retryat 06.169; B adopted at 09.239, first success 16.816: no overlap. After the heal A unfenced and forgot the claim; B kept and re-replicated it, 145 calls with 0 errorsA first run on fb4abb5 found that the adopter destroyed the woken claim on its next round (the wake left the consumed row's epoch on the claim); fixed in 05d49f1 with a regression test and rerun above.
Observation: a node killed mid-round leaves that round's partial upload in the store: chunks and possibly a manifest, with no
meta.json. The operator's reclaim rule removes them once they age past the grace.Review