Repository navigation
Per-owner persistent volumes managed at runtime: attach to warm claims, single writer, survive hibernate and image upgrades #300
Description
A control plane that keeps each user's data (home directory, workspace, app databases) on a persistent disk, and replaces only the system disk on an image upgrade, needs volumes whose lifetime is the user's box, not one claim. Today's dataset volumes (catalog entries from the config, ro/rw, attached at claim; see docs/sandboxd-api.md and #75) do not fit: they are config-owned and shared by name, a claim with a volume cannot hibernate (409), and nothing ties a writable volume to one owner or keeps two claims from writing it at once.
Use case: a desktop box per user. An upgrade today saves the user's files to object storage and restores them into a new machine (about 9 GB for a real user, ~90 s of restore). With a per-box data volume, an upgrade is: stop the old claim, claim the new template, attach the same volume.
Request: per-owner persistent volumes, managed at runtime.
- API:
PUT /v1/volumes/{name}{size_gib, tenant}creates one (a raw sparse file on the node's volume dir, on a reflink-capable filesystem where possible),GET,DELETE, and an online grow. Tenant-scoped: a tenant token sees and attaches only its own volumes; root sees all. - Attach at claim (
"volumes": [{"name": "...", "mode": "rw", "attach_only": true}]or a new field), including to a warm VM: hot-attach through cocoon'svm disk attachafter the warm claim, so a box with a volume still gets the warm-pool latency. - Single writer: an
rwvolume is attached to at most one live claim; a secondrwattach answers 409 naming the holder. Releasing (or the claim's VM being destroyed) frees it; the lock survives a sandboxd restart (persisted with the claim). - Hibernate and wake with the volume attached: depends on Hibernate and snapshot a VM with external raw disks attached, re-attaching them on restore cocoon#280 (snapshot/hibernate with external raw disks); until then sandboxd may do the detach / capture / re-attach itself, with the guest told to umount and re-mount (silkd hook), but the preferred path is silkd: set TCP_NODELAY on guest egress connections #280.
- Placement: a volume lives on one node; a claim naming it is placed on that node (redirect to the owner node in a mesh), and fails clearly if that node cannot take it.
- Release and archive: releasing a claim never deletes its volume;
archiveexports the volume with the claim's state (or refuses with a clear error until supported).
Later (not in this issue): the volume's backing becomes an S3-backed block device with a node-local cache, so volumes stop being node-bound; the API above should not need to change for that.
Acceptance: create a 20 GiB volume for tenant t1; claim a warm VM with it rw, write a file in the guest; a second claim naming it rw gets 409; release, claim a different template with the same volume, the file is there; hibernate and wake keep it attached (with cocoon#280); t2's token cannot see or attach it.
Activity
CMGS commentedon Oct 2, 2026
Update: cocoonstack/cocoon#280 is closed — cocoon stays loosely coupled, and handling attached volumes around a capture is control-plane logic. So for this issue, sandboxd owns the sequence:
cocoon vm disk detach, then capture as today.cocoon vm disk attachthe same backing file, then the guest mounts it (silkd hook) before the claim is reported ready.rwvolume).minatoaquaMK2 commentedon Oct 9, 2026
Hi, we'd like to discuss a related use case here and align our contribution with #300.
Our use case is a long-running agent sandbox that only discovers which datasets it needs after startup. We'd like to create the sandbox once, through Kubernetes or an SDK, then attach and detach data volumes as the task changes while keeping the same VM, processes and session state.
The direction we'd like to contribute in separate steps is:
For the current draft, unfinished operations are recorded in the existing local
claims.json; it adds no database dependency. Changes commit per volume, so a batch may partially complete, with the pending operation retained for retry or cleanup.We see this as complementary to the per-owner writable volumes in #300. We read the follow-up about sandboxd owning the quiesce/unmount/detach/capture and restore/reattach sequence. The draft does not implement that sequence, volume creation/growth, or ownership across hibernation; it keeps the existing restriction on capture while volumes are attached. We'd like the live attachment API and recovery behavior to fit that future lifecycle.
Would you be open to landing the narrower read-only capability separately, or would you prefer agreeing on the broader volume API/identity model first? We'd also appreciate your preferred direction for the Kubernetes entry point and feedback on the per-volume commit/retry contract.
This is our first contribution here. The draft has local race tests and Linux/Darwin lint passing; real-VM hotplug/unplug and restart-recovery validation are still pending. We're keeping it in draft to discuss the direction and complete that validation.
CMGS commentedon Oct 17, 2026
Status: #339 landed live attach and detach of read-only catalog volumes on a running claim (
GET/PUT/PATCH /v1/sandboxes/{id}/volumes). Still open: tenant-scoped volume create, list, delete and grow at runtime; rw attach to a warm claim with single-writer 409; hibernate and wake with a volume attached, which is still refused withErrVolumeCapture; placement by volume; archive.