feat: bound what one cloudsync_payload_chunks() call prepares

[andinux]

Why

#75 removed the quadratic term from the chunked download path, so a drain is now linear in the stream. Linear is still unbounded: cost stays proportional to total stream bytes, so a large enough tenant still cannot finish inside a caller's time budget — it just takes a bigger tenant to get there.

And because nothing is durable until the stream reports is_final, an attempt that runs out of time keeps no progress: the next one restarts from chunk 0. That is the shape of the production incident this line of work came from, and #75 did not address it.

Nothing anywhere bounds preparation today. Everything that looks like a limit bounds something else — chunks per HTTP response, one response's size, an artifact list query, one chunk, chunks applied per client drain.

What

max_window_bytes ends the scan once that many payload bytes have been emitted, at the next complete db_version, and reports an ordinary complete stream over a smaller window: is_final, with watermark_db_version lowered to that point, plus a new window_capped output saying it stopped on the budget rather than because the window was drained.

A second pair carries the budget across calls: the window_bytes output reports what the window has spent so far, and a ninth argument resume_window_bytes seeds it. This is required, not optional — a stateless caller fetches one chunk per call and resumes through resume_*, so every call starts a fresh scan. Without it the count restarts each time and a budget larger than one chunk is never reached, which is how the /check job pages. Caught in review; there is now a regression test in both suites that pages one chunk per call under a multi-chunk budget.

The framing is what makes this cheap: cap the window, not the stream. A capped window is not a truncated stream needing new state — it is a complete stream over a smaller window, so every existing invariant holds. The caller checkpoints at the reported watermark and asks again. No protocol change, no resumable state, no server change. Unset, behaviour is byte-identical to today.

Two conditions are load-bearing

A window may never end mid-value or mid-db_version. cloudsync.h:132-136 already states the invariant: the receive cursor must land on a complete db_version, because the next request resumes with db_version > since and no seq. Ending a window mid-version silently skips the unapplied remainder — data loss, not a slow path.

Because that boundary test is what stops the scan, a db_version larger than the whole budget is still emitted in full. Otherwise a window could come out empty and the drain would never advance at all — a livelock strictly worse than a timeout, because it never even fails.

A consequence worth knowing: the budget is an approximate floor, not a ceiling. A tenant whose transactions are consistently larger than a chunk gets weak capping.

Verification

Both backends, by tiling: repeated capped calls must reproduce the uncapped stream exactly.

• SQLite (test_payload_window_cap): windows abut (first == since + 1), each advances, and summed chunks and bytes equal the uncapped totals — at a 200 KB budget and at a 1-byte budget.
• PostgreSQL (65_payload_window_cap.sql): the same four checks. Suite goes 557 → 561 pass, 0 fail.

Mutation-verified. Removing the db_version boundary guard fails the PG 1-byte check with 5 chunks / 844,416 bytes instead of 6 / 884,987 — the silent data loss, caught. Notably the 200 KB budget still passed under that mutation (at that size the cap lands on boundaries by luck), which is why both budgets are in the test. On SQLite, dropping the guard and dropping the end-of-scan check each fail too.

An earlier version of the SQLite test was blind to its own subject: with one row per transaction every chunk boundary was already a version boundary, so the guard never mattered. The data now mixes 40 single-row transactions with two 15-row ones (~300 KB, larger than a chunk) so boundaries fall inside a version.

SQLite suites: unit (156), review_regressions, network_unit, integration_bootstrap — all exit 0.

Compatibility

window_capped and window_bytes are output columns, so they do not move the hidden columns a table-valued call binds positionally. max_window_bytes and resume_window_bytes are declared last, taking arguments 8 and 9 and leaving 1..7 as they were — inserting either earlier would silently rebind every existing argument. An 8-argument caller still works: resume_window_bytes defaults to 0.

On PostgreSQL both new arguments are read behind PG_NARGS(). Between installing the 1.2 binary and running ALTER EXTENSION cloudsync UPDATE, the 1.1 SQL definition still points at this function and calls it with seven arguments, and reading the eighth and ninth slots would run off the end of fcinfo->args.

Rollout has one ordering constraint: an old caller against a new extension is unaffected (cap unset → identical to today), but a new caller against an old extension is a hard SQL error. Tenants must be upgraded before anything passes the new argument.

Version: the SQL surface changes, so scripts/check-postgres-migration.sh requires EXTVERSION 1.1 → 1.2 and the binary moves to 1.2.0 — new optional functionality, backward compatible, a MINOR bump. cloudsync--1.1--1.2.sql drops the old function before creating the new one: CREATE OR REPLACE cannot change a return type, and because the new parameter has a default, keeping both would leave every existing 7-argument call matching two candidates.

Also in here

An explicit NULL for resume_db_version now means "not given" on SQLite, as it always has for filter_site_id and on PostgreSQL (cloudsync_postgresql.c:1353). It was read as a resume point of 0, which silently ignored since_db_version and restarted the scan at the start of the window. Latent before, but unavoidable now that reaching the budget arguments means passing NULLs for 5..7 — it produced a real livelock while testing this change, twenty windows returning the same chunk. No in-repo caller passes a NULL resume, and the two server call sites (4-arg and 7-arg) are unaffected.

resume_db_version / resume_seq / resume_frag_offset are now documented. They have been callable since the chunked path shipped in #50, but never appeared in API.md, which left a stateless caller no documented way to page a stream.

🤖 Generated with Claude Code

[andinux]

Pushed dd5a95e fixing a real defect found in review: the cap could fail to fire at all.

The check only looked at whether a finished chunk happened to end on a db_version boundary. When a transaction's row count and a chunk's capacity stay out of step, none ever does — every chunk ends a row or two inside a version, the check postpones, and the cap never fires. Reproduced with 256 KB chunks, 20 KB rows and transactions of 12 then 13 rows: a 200 KB budget emitted the whole 13,011,454-byte history with window_capped = 0, byte-identical to uncapped. Every chunk showed next_db_version == db_version_max.

That is not the bounded overshoot the PR described — it is no bound at all, so a large history can still blow a preparation deadline.

The chunk builder now stops adding rows at the first db_version boundary once the budget is spent, so the boundary the cap needs is produced rather than waited for. Same change in both backends.

Two consequences worth flagging for review:

• Ending a chunk early means a capped drain uses slightly more chunks than an uncapped one, each with its own header. So the invariant is conservation of payload rows, not bytes or chunk count — the tests were asserting byte equality and had to change. API.md and the changelog now say this.
• Only the final chunk's watermark_db_version is the one to store. Earlier chunks of a capped window still report the window's provisional upper bound, because the cap is not known until the stream ends. My own test driver aggregated with max() and silently skipped data, so this is a real footgun; API.md now says it explicitly.

Both suites gained the out-of-step transaction shape — without it the bug reproduces in neither — plus a per-window size bound that a never-firing cap cannot satisfy. Mutation-verified: removing the new boundary stop fails both suites.

SQLite: unit (156), review_regressions, network_unit, integration_bootstrap all exit 0. PostgreSQL: 561 pass, 0 fail.

[andinux]

Pushed 83a4e5f and 4ee6737.

83a4e5f fixes a regression I introduced in dd5a95e: moving the byte accounting into the ordinary chunk builder left out fragment emission, which returns before reaching it. A history of oversized values therefore spent nothing and the cap never fired — every budget emitted the whole 6,006,460-byte history with window_capped = 0. Both fragment emitters now add their encoded bytes. Capping still cannot happen mid-value, since the cap is only evaluated once a value's last fragment has been emitted.

Both suites gained a purely fragmented history. The existing cases could not have caught this — none of their values are large enough to fragment. Mutation-verified in both backends.

Worth knowing before enabling this against PostgreSQL tenants

While building the fragment test I hit issue #69 defect 2: a pinned xmin leaves the cached db_version unreloaded, so several separate transactions share one db_version. A session that has been draining chunks stops advancing db_version for subsequent single-statement transactions, which silently produces a history with one version and nothing for a window to end on.

That matters here because the cap rests on a db_version boundary meaning "everything at or below this is in the window". When versions collapse, a later transaction's changes land on a version already served and the next window's db_version > since skips them.

The cap does not cause this and does not make it worse in kind — the uncapped path and the positional resume rest on the same assumption, and #69 already calls it "the damaging one". But the cap multiplies the number of checkpoint boundaries in a drain, so it multiplies the chances of a collapsed version straddling one. I'd settle #69 before turning capping on for PostgreSQL tenants; the SQLite side is unaffected.

4ee6737 records that in the test, which keeps its oversized values in the already-synced table to stay clear of it.

SQLite: all four suites exit 0. PostgreSQL: 562 pass, 0 fail.