Repository navigation
store/s3: sparse-aware, chunked, content-addressed checkpoint transfer - #364
Merged
Merged
Conversation
An export file is now a manifest of 16 MiB chunks keyed by SHA-256. Upload walks the data segments with SEEK_DATA/SEEK_HOLE, skips holes and all-zero chunks, and hashes each chunk once for both its key and the record digest. Fetch sizes the file and writes only the present chunks, so a mostly-hole cow.raw stays sparse on the fetching node and no longer travels at its logical size. The record digest is unchanged for the same content. Records written in the old layout are not readable.
Chunks live under chunks/<lineage>/<sha256>. A publish lists its lineage once and uploads only chunks the lineage lacks; the lineage defaults to the record id and a claim's captures pass the sandbox id so consecutive snapshots share their unchanged chunks. Delete removes a record's manifests and meta. SweepGenerations, a no-op until now, drops superseded manifests idle past the grace and then chunks that no manifest names and that are older than the grace, so an in-flight publish's fresh chunks survive. Publish gains a PublishOption and SweepGenerations a context; the dir backend ignores the option.
A record pulled for peer heal no longer streams holes. Each export file is a size entry followed by one tar entry per non-zero chunk carrying its index and SHA-256; the receiver sizes the file, writes the chunks at their offsets and fails the pull on a checksum mismatch. Endpoint and auth are unchanged.
Mutation checks showed the sparse tests passing without a trailing hole, a repeated extent in one chunk or a young manifest beside an old meta. Each now has a case that fails when its guard is removed.
A publish used to trust the lineage chunk listing, so it could skip uploading an unreferenced chunk that a concurrent sweep was about to delete. It now reads the committed manifests of the lineage and reuses only the chunks they name; any other chunk is uploaded again. Accepted residual: a sweeper that listed an old unreferenced chunk can still delete the fresh re-upload of the same key between its list and its delete, which needs a content repeat and a concurrent sweep.
A record deleted between a publish's reference read and its manifest upload used to lose the chunks the new record dedups. Delete now removes only meta.json and the labels and re-stamps the generation manifests; the sweep counts a manifest without a meta as a reference until it has been idle for the grace and frees its chunks only afterwards. A fetch that already selected the record also survives its deletion.
A re-publish between the sweep's listing and its meta GET left the listing's stale mtime in charge, so the sweep took the new generation for an old one and deleted chunks it dedups. The age now comes from the GET response that also names the current generation.
A publish into a shared lineage listed every record of the namespace and read two objects per record. It now writes a zero-byte marker lineages/<lineage>/<id> before the commit and reads only the records the markers name: one LIST plus two GETs per record of the lineage. The sweep removes a marker once its record has neither meta nor manifests and the marker is idle for the grace.
The checkpoint and template views of an s3 store share one prefix and the s3 sweep already reads both id namespaces, so sweeping the template view repeated the whole listing and widened every race window. The template view is skipped when the checkpoint store is shared.
Publish and PublishDigested take a lineage string instead of a functional-options set. s3.go places its type blocks directly under the Store methods and every utility after them, and no local shadows the path package. ScanFile reads with ReadAt, the zero-length manifest slice literal and a wrong comment tail are gone, and the WriteChunk cases carry an index field instead of branching on the case name.
When a record's meta disappeared between the sweep's listing and its meta GET, the sweep skipped the record's manifests, so a chunk another record of the lineage had just reused from them was deleted. The listed manifests now still count as references and are not removed, leaving reclamation to a later sweep.
Dedup safety relies on a publish finishing within store.GenerationGrace after the record it reused chunks from is soft-deleted. Without a bound, a slow publish could commit a manifest naming a chunk the sweep already deleted, and the hole would spread to every later capture through the manifests the dedup set is built from. Run publish under the budget fetch already uses, renamed transferBudget: a record that cannot be uploaded in that budget could not be fetched within it either, and the budget stays below the grace so the reuse window always covers a publish.
… Delete Delete removes only meta.json and the labels, so a soft-deleted record kept its <prefix><id>/ prefix (its manifests) for the grace plus up to one sweep period. The delimiter listing returned every such prefix and Metas paid a 404 meta GET for each; with replication publishing a checkpoint every interval and soft-deleting the previous one, GET /v1/checkpoints, the TTL sweep and restart reclaim scaled with (grace / interval) x replicated claims. Manifests move to <prefix>manifests/<id>/<gen>.json, beside chunks/ and lineages/, so the record prefix exists exactly while the record is live and the id pattern never matches the manifests/ prefix. Sweep semantics are unchanged: manifests of a live record are references, manifests of a vanished record stay references until idle past the grace and are then doomed, and chunks are freed only when no non-doomed manifest names them. Delete also removes the record's lineage markers at once, reading the lineage from the manifests it already refreshes. Markers only feed the dedup set, which resolves each marker through the record's meta, and the sweep's marker reclaim; chunk references come from manifests alone. The sweep's stale-marker pass stays for crash residue. The first capture after a wake uploads every chunk again because the wake deletes the record it restored from; deploy.md says so.
sandboxd no longer garbage-collects the S3 store: SweepGenerations is a no-op again and the sweep, its inventory and the stale-marker pass are gone. Delete keeps its soft semantics (drop meta, labels and the lineage markers, re-stamp the manifests so their LastModified is the delete time), which is what makes an external reclaim safe against in-flight publishes together with the 30 minute publish bound. deploy.md documents the key layout, what sandboxd never deletes, the safe reclaim rule, the invariant behind it and the cost of not running one.
This was referenced Oct 18, 2026
This was referenced Oct 18, 2026
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #362. Refs #183 (sandboxd no longer reclaims S3 at all; the reclaim contract is documented for the operator instead, see below).
Bottom of a stack: #366 builds on this branch, and #365 on #366. Merge in that order.
What
An S3 export file is now a manifest of fixed 16 MiB chunks (the existing
DigestChunkSize), each stored under its SHA-256.<prefix><id>/meta.json(andlabels.json) is the live record.meta.jsonis uploaded last as the commit marker, and the record prefix exists only while the record is live.<prefix>manifests/<id>/<generation>.jsonlists each file's path, size and present chunks, plus the lineage.<prefix>chunks/<lineage>/<sha256>.<prefix>lineages/<lineage>/<id>.ScanFilewalks data segments withSEEK_DATA/SEEK_HOLE, so a chunk lying fully in a hole is never read, and a chunk read as all zero is skipped. The chunk SHA-256 is both the object key and the record-digest chunk entry, so there is no second hash pass; the record digest (PublishDigested) is unchanged for the same content. A publish runs under a 30-minute budget, the same as a fetch.ErrNotFound.Publish/PublishDigestedtake a lineage (""means the record id). The pool passes the sandbox id, so every checkpoint, archive and replica of one claim shares chunks. A publish reads the current manifests of the records its lineage markers list and uploads only the chunks they lack.meta.json, the labels and the lineage markers, and re-stamps the record's manifests. It never deletes a chunk or a manifest, so a fetch that already selected the record still completes.SweepGenerationsstays a no-op, as on main. Reclaiming chunks and manifests is an offline operator job:docs/deploy.md→ Reclaiming S3 storage documents the key layout and the procedure. Stop every node on the prefix, build one inventory, keep the manifests live metas select, delete the rest and unreferenced chunks. Online reclamation is not supported. The dir backend keeps its own local generation sweep.transfermanageris gone: chunks are at most one part (go.mod−1).Hot-path cost
None on the claim path or the data plane.
Metas(checkpoint listing): one GET per live record. A deleted record leaves no prefix to list.Owner decision: no compatibility, upgrade step
Records in the old S3 layout are not readable after this change, and there is no fallback reader. Before starting the new binary on an S3
checkpoint_store, do one of these:prefixat an empty location.docs/deploy.mddescribes this and warns against an expiry lifecycle rule on the prefix, which would delete chunks a live record still names.Tests
Metassends no meta GET for it, and its chunks survive.SweepGenerationsleaves the bucket untouched.Hardware acceptance
Two bare-metal nodes (one cell, PostgreSQL registry, S3 via SeaweedFS 4.48, cocoon v0.7.3,
rt:24.04, 10 GiB disk). The stack head contains this branch's final code; later commits on this branch are docs only.meta.json, alabels.jsonor a lineage marker, with 0 underchunks/ormanifests/. All 1,739 chunk and manifest keys ever seen were still present at the end.GET /v1/checkpointsstayed at 0.15–0.20 s.Review