Read-only file access to hibernated or archived sandboxes without waking them

[CMGS]

Read-only file access to hibernated or archived sandboxes without waking them

Closes #351

Needs cocoon v0.7.3 or newer (vm disk attach --from-snapshot / --from-dir, cocoonstack/cocoon#285). Builds on #364 and #366 (both merged).

What changes

A per-pool flag opts in:

{"pools": [{"template": "rt:24.04", "warm": 4, "reader": true, "idle_hibernate_seconds": 600}]}

Absent, nothing changes. reader needs warm >= 1, is config-owned (PUT /v1/pools rejects it), and is reloadable; turning it off leaves open reader connections to finish and the next connection wakes.

With it on, a /v1/sandboxes/{id}/agent connection to a hibernated or archived claim of that pool is answered with the 101 first, and the opening frame is read before any wake is chosen:

• fs_stat, fs_list, fs_read, fs_pull go to the claim's reader VM. The claim stays hibernated or archived, no wake event is recorded, no idle clock is stamped.
• An info frame pipelined ahead of a read (what the SDKs send on a first dial) rides along. A lone info waits 100 ms and then wakes.
• Any other call wakes the claim and hands the same connection to its silkd with the unread bytes replayed. A claim that woke meanwhile is handed over the same way; one that slept again gets a fresh reader, so a reader never answers for a stale sleep.
• No reader available (no warm VM in the pool, low disk, a failed attach, an archive without disk_attach_files, an image without silkd at /usr/local/bin/silkd, a sleep that is not reader-safe): wakes as today, with a warning naming the claim and the reason.
• Running claims and the claim path take no new branch: WakeAgentSocketOrRead differs from WakeAgentSocket only on the non-running path.
• Node lease fence (Running claims survive node loss: periodic snapshot replication behind a node lease #366): a reader dial goes through the same fence as a wake. A fenced node answers a replicated sleeping claim's read with an error frame (lease lost; retry), and a fence trip closes an open reader connection and a handed-over relay, as it closes any relay.

Why hand over instead of closing

Closing a parked connection when a write frame arrives would fail that call: the SDK takes a parked connection after its liveness peek, sends the frame, and a close after that surfaces as an error on the in-flight RPC. Replaying the frame into the woken guest on the same connection costs nothing and the client never sees a close.

Reader

sandboxd talks to cocoon only through its CLI contract: it never reads cocoon file names, the sidecar, or blob paths. One reader per sleeping claim, never shared across claims or tenants:

1. VM. A warm VM popped from the claim's own pool, so it boots the claim's image with the claim's layers (cocoon checks the image and layer order on attach). An empty pool never cold-boots one: the call wakes. The refill replaces the taken VM.
2. Disk. Hibernated: cocoon vm disk attach READER --from-snapshot SNAP, under the claim's Transition lock so a racing wake waits for cocoon's copy (a reflink: 58 ms on hardware); cocoon also holds the snapshot's lease. Archived: only snapshot.json and the files its disk_attach_files lists are fetched into <data_dir>/readers/<id>-<rand> (store.FetchFile, never the shared fetch cache; on S3 only the chunks of those files are downloaded, holes stay holes; names with a separator or .. are refused), then --from-dir, and the fetched files are deleted once cocoon has its copy. An archive exported by an older cocoon (no disk_attach_files) wakes before any VM is taken. cocoon attaches the copy read-write and reports its guest serial, so the guest replays the ext4 journal onto the copy, never the original.
3. View. Through silkd as root: mount the copy by its serial, mount read-only the layers the reader booted with (its own /proc/cmdline cocoon.layers=, the boot/init contract), mount a read-only overlay (ro,metacopy=on,redirect_dir=follow,lowerdir=<upper>:<layers>) at /view, and start the claim's own silkd as env SILKD_PORT=2051 chroot /view /usr/local/bin/silkd. The chroot confines paths and symlinks to the view without a protocol change; the read-only overlay makes writes fail at the filesystem.
4. Lifetime. Kept 30 s after its last connection (the idle timer is armed at registration, so a reader no connection acquires is reaped too; a newer setup retires the one it replaces), then the VM is destroyed and cocoon removes the copy with it. Concurrent first reads share one setup (singleflight). Boot reconcile empties the readers dir.

A reader has its pool's lane: no NIC on the none lane, the claim's bridge NIC on the bridge lane (egress-lane claims never hibernate). It is the pool's size, so a medium pool's reader costs a medium VM. Leave reader off on a Firecracker node or with an older cocoon: sandboxd does not know the hypervisor, so there each read would spend a warm VM on a failing attach before waking.

Freshness

With the flag on, a guest sync runs right before the hibernate pause of that pool's claims. A reader shows the disk as of that sync: a write in the instant between the sync and the pause is visible only after a wake. A hibernate whose sync fails or times out is marked not reader-safe (reader_safe, persisted, cleared on wake), as is any sleep from before the pool had reader, and its reads wake the claim. A replica row (#366) never carries reader_safe: a replica is a capture of a running claim, so after an adoption a read of it wakes the claim instead of serving a disk that misses dirty guest pages.

Also in this PR

• wire.RequestOp: op tag of a request line.
• store.Store.FetchFile (dir and s3, today's layouts) and utils.CloneFile (FICLONE with a hole-skipping copy fallback, used by the dir backend).
• engine.DiskAttachSnapshot / DiskAttachDir (parse the attach JSON name), StartReader / DialReader; dialVsock/probe take a port.
• Docs: deploy.md (pool flag and a section on the reader), sandboxd-api.md (/agent behavior), sdk.md, silkd.md, README.

Size of this layer (against #366): prod Go +1111/−102, tests +2044/−14.

Verification

• Codex rounds 1–6 on one thread, CLOSED at the head. Two rounds after the rebase found that a replica row could carry reader_safe into an adoption (fixed in putRow) and that the reader path skipped the lease fence (fixed). A later review found that a connection moved to a fresh reader never closed its client when that reader ended, because the superseded flag stayed set; fixed with a regression test.

• make go-lint: 10 × 0 issues. (5 modules × linux/darwin), fmt clean; go test for config, engine, pool, server, store green; asl clean.

• git diff origin/main has no reference to cocoon's file names, sidecar schema or blob paths.

• Unit tests cover interception, reader lifecycle (attach then mount order, singleflight, idle release, stale sleep, setup failure cleanup, the Transition lock held only around the copy), the archived path (only snapshot.json plus listed files fetched, no VM taken when the list is absent, unsafe names refused), FetchFile for dir and s3, and an in-process e2e through the real relay and SDK.

Hardware acceptance

On the full stack (with #364 and #366)

Two bare-metal nodes (one cell), PostgreSQL registry, SeaweedFS, cocoon v0.7.3, pools with reader: true (one of them also replicating).

• Hibernated read: the reader was ready in 139 ms. stat, list, a 20 MiB read and a pull all matched, and a symlink to /etc/shadow returned the claim's own file. Usage events: claim and hibernate only; the claim stayed hibernated.
• Archived read from S3: ready in 708 ms with no unarchive. During the reads the store saw 3 meta, 3 manifest and 20 chunk GETs: cow.raw 18/18, cocoon.json 1/1, snapshot.json 1/1, memory image 0/30. The reader dir was emptied right after the attach.
• Fence: a replicated, hibernated claim was served by a reader before the cut (ready in 165 ms). During the cut, reads sent straight to the fenced node got lease lost; retry (http 503), and no reader was opened.
• Adopted replica: a replicated, reader-safe claim was hibernated on A (a reader served it in 129 ms), then A was killed. The registry row's claim carried reader_safe false both before and after the adoption. A read on B woke the claim from A's replica (unarchive, 599 ms, data intact), and B opened no reader.
• e2e: PASS on both nodes (23.0 s / 21.5 s), and on the final head 22.2 s.

On main with cocoon master 2375369 (before stacking)

Bare-metal node, cocoon master 2375369 (v0.7.2-2-g2375369), rt:24.04, pools small and medium with warm: 2, reader: true, head 310761d (later commits are docs only).

• No wake (medium): files including a 50 MiB file, a hostname change, a new user, symlinks and an unsynced last write were stat/list/read/pull'ed through the SDK; every checksum matched, the claim stayed hibernated with only claim and hibernate usage events. The reader showed one attached snap-* disk; 30.3 s after the last close the reader VM and its run dir were gone, no copy left, pool back to 2/2.
• Confinement and hand-over: a symlink to /etc/shadow returned the claim's shadow (after useradd), not the image's. A write on the same handle after a reader-served stat woke the claim and succeeded (141 ms); the SDK saw no error.
• Archived (archived by the new cocoon): envelope disk_attach_files: ["cocoon.json","cow.raw"]; read served in 264 ms with no unarchive. strace of sandboxd shows store opens limited to the record meta, snapshot.json and the two listed files; the memory image was never opened, and the reader dir was emptied 10 ms after the attach.
• Old archive (archived by cocoon v0.7.1): the read woke the claim with the right content; warning archive envelope lists no disk_attach_files; no warm VM was taken.
• Empty pool: warm: 0 → the read woke the claim; no cold boot (tier="cold" 0), VM set unchanged.
• Race: 5 × hibernate then concurrent read and write on one claim: every call succeeded, data intact, the claim ends running; no orphan VM, reader dir or copy after 40 s.

	reader	plain wake of the same medium claim
first call	138 ms (setup 125–137 ms)	127 ms
later stat/list/read	0.2–0.6 ms	—
memory after reading 50 MiB (RSS / PSS)	305 / 232 MB (medium reader VM)	181 / 175 MB

Setup breakdown (by hand on a medium VM): attach --from-snapshot 58 ms, ext4 mount with journal replay 42 ms, erofs 3.6 ms, overlay 1.7 ms, the rest cmdline read, silkd spawn and probe.

As before, the cost against a plain wake is about even: a wake restores memory lazily. What the feature provides is reading a sleeping claim without running it: its processes do not resume, so no CPU time, no egress side effects, no idle-clock reset, and the claim stays asleep.