From 935d75a507d195dac05fd8996f151c59723d5342 Mon Sep 17 00:00:00 2001 From: Mick van Dijke <12992260+mickvandijke@users.noreply.github.com> Date: Mon, 12 Oct 2026 15:46:54 +0200 Subject: [PATCH 1/6] docs(adr): ADR-0023 spread hot-record reads and refuse full browser sessions Records why every client read a popular record from one holder, why a full node looked unreachable for 30 minutes, and the decision to spread first reads across the close group and answer capacity refusals with a STUN error. Amends ADR-0015's capacity-rejection rule and includes Chromium/WebKit measurements. Co-Authored-By: Claude Opus 5.5 --- ...-reads-and-refuse-full-browser-sessions.md | 317 ++++++++++++++++++ docs/adr/README.md | 1 + 2 files changed, 318 insertions(+) create mode 100644 docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md diff --git a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md new file mode 100644 index 00000000..26a6b833 --- /dev/null +++ b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md @@ -0,0 +1,317 @@ +# ADR-0023: Spread hot-record reads across the close group, and refuse full browser sessions explicitly + +- **Status:** Proposed +- **Date:** 2026-10-12 +- **Decision owners:** +- **Reviewers:** +- **Supersedes:** none +- **Superseded by:** none +- **Related:** [ADR-0015](./ADR-0015-direct-browser-clients-over-webrtc-direct.md) + (amends its capacity-rejection rule), + ADR-0020 (get-or-closer chunk reads, open in #249), + ant-client ADR-0004 (direct browser read client), + [RFC 8489 §6.3.4](https://www.rfc-editor.org/rfc/rfc8489#section-6.3.4) (STUN error responses), + [RFC 8445 §7.2.5.2.4](https://www.rfc-editor.org/rfc/rfc8445#section-7.2.5.2.4) (connectivity-check failure) + +## Context + +The question that prompted this ADR: can the network serve thousands of +browser clients who all want the same chunks at once, for example because they +are watching the same film? + +A node admits a bounded number of browser sessions (ADR-0015, "Public-listener +resource isolation"): + +| Setting | Default | +|---|---:| +| `max_connections` | 32 | +| `max_connections_per_ip` | 4 | +| `max_concurrent_requests` | 16 | +| `max_requests_per_second` | 256 | + +saorsa-transport refuses to bind a listener with `max_connections` above 256 +(`MAX_PENDING_ASSOCIATIONS`). Configuration can raise the limits below that, +but only through a `--config` file. No CLI flag sets them. + +Three findings matter more than the limits themselves. + +**1. Every client reads a record from the same holder.** Each record is stored +on its close group of `CLOSE_GROUP_SIZE` = 7 nodes. Both read paths in the +shared client engine (`ant-core/src/client_engine/read.rs`, used by native and +browser clients) order candidates by XOR distance. Early reads during +discovery always take the nearest unattempted hint. The final ordinary path +sorts the close group nearest-first. A second holder is asked only after the +1-second hedge delay or a failure. So every client sends its first GET for a +record to the same node, and a popular record is served by one node while +the other six idle. + +**2. Upload bandwidth runs out before session slots do.** Take a 5 Mbit/s +stream and 4 MiB chunks, so each chunk is about 7 seconds of playback. If +1,000 viewers are at the same point, one chunk window asks for 4 GiB in about +7 seconds, roughly 5 Gbit/s. One node cannot supply that. Spread evenly over +all seven holders it is about 700 Mbit/s each. Viewers who started at +different times spread across the file's chunks, but every viewer reads the +data map and the first chunks, so those are hot whether or not viewers are in +sync. Raising session limits alone admits more viewers to the same uplink. + +**3. A full node looks unreachable.** +- When a proven association finds no free slot, the transport ignores the + browser's ICE check without replying (`DirectUdpMux::run`). +- Application-level capacity and rate rejections close the channel or + association, also without a reply. ADR-0015 chose this to avoid amplifying + attacker traffic. +- The browser therefore waits out its 10-second channel-open timeout. It then + records a dial failure, and `ENDPOINT_FAILURE_COOLDOWN` keeps that node out + of every lookup and read for **30 minutes**. + +Under load, busy nodes vanish from each client's view one by one, so the +network looks smaller exactly when demand peaks. ant-client's lookup notes say +a third to a half of the WebRTC Direct endpoints discovered on mainnet time +out when dialled. Some of those may be full rather than down. That is +unverified, but each node's `rejected_connections` diagnostic can show it. + +## Decision Drivers + +- Use the capacity of the whole close group before anything costlier. +- A busy node should cost a client about one round trip, not ten seconds, + and should come back into rotation in seconds, not half an hour. +- No new wire message, no protocol version, and no flag day. Every mix of old + and new nodes and clients must behave at least as well as today. +- Keep ADR-0015's resource bounds and its refusal to amplify traffic. +- Leave the larger fixes for hot content (more holders, delegated lookups) + open for their own decisions. + +## Considered Options + +1. **Raise the per-node limits only.** Rejected as the sole measure. It is + capped at 256 by the transport, and it fixes neither single-holder + concentration nor full nodes looking dead. Raising the limits after a load + test remains worthwhile and is left to a separate decision. +2. **Spread reads across the close group.** A client-only change to read order. +3. **An explicit "busy" signal.** Three ways to send one: + 1. *A new browser-protocol status.* This needs a wire change across + saorsa-transport, ant-node and ant-client. Worse, sending it needs a + full ICE, DTLS and SCTP association for every refused attempt, which is + exactly the cost the session limit exists to bound. + 2. *A STUN error response at transport admission.* A browser fails a + candidate pair as soon as its connectivity check gets an error response + (RFC 8445 §7.2.5.2.4). A refusal costs the node one STUN parse, one HMAC + and one send, after the reachability-cookie round trip it already + requires. No wire change. + 3. *Client heuristics alone,* such as a shorter cooldown for every dial + failure. Rejected: clients would redial genuinely unreachable endpoints + every few seconds, at ten seconds per attempt. +4. **More holders for hot records** (caching along lookup paths), browser + peer-to-peer sharing, or a CDN-backed HTTP gateway. These are the real + answer to finding 2, but each needs its own ADR. Deferred. + +## Decision + +We adopt options 2 and 3.2, and the client classification that makes 3.2 +useful. + +### 1. Spread first reads across the close group (ant-client) + +For every record, the read order is: +1. the close group, meaning the `CLOSE_GROUP_SIZE` nearest candidates known + at that moment, in uniformly random order; +2. then the remaining candidates, nearest first. + +How each read path applies this: +- **Early reads during discovery** choose uniformly among the unattempted + members of the currently known close group. When all of them have been + tried, the nearest remaining hint is next. +- **The ordinary path** shuffles the close group at the front of its final + candidate list. + +The hedge delay, the early-read allowance, the at-most-two-GETs rule and the +absence rule are unchanged. The engine takes the choice as a parameter. +Production passes a uniform random choice, and tests pass nearest-first to pin +order. Native clients use the same engine, so they spread too. + +### 2. Refuse proven associations explicitly when full (saorsa-transport) + +**When it applies.** An association becomes proven when it has returned the +source-bound reachability cookie (ADR-0015). If a proven association then +finds no slot, because the listener is at `max_connections`, the pending queue +is full or the accept queue is full, or because its IPv4 address or IPv6 /64 +holds its full share, the mux no longer ignores the request that would have +admitted it. + +**What the mux sends.** It answers with a STUN Binding error response that: +- carries the same transaction ID as the request; +- has `ERROR-CODE` 486 (*Allocation Quota Reached*), with the reason phrase + "Insufficient Capacity" or "Source Quota Reached". A 4xx code ends the STUN + transaction (RFC 8489 §6.3.4), so any compliant ICE agent fails the pair at + once. A 5xx code such as 508 *Insufficient Capacity* may be retried, so it + is not used. Browsers do not expose STUN errors to applications; the phrase + only makes packet captures readable. +- carries `MESSAGE-INTEGRITY` keyed by the listener credential that + authenticated the request, so the browser accepts it as the answer to its + own check; +- carries `FINGERPRINT`; +- is never larger than the request it answers. If it would be, it is not + sent. + +**What stays silent.** Duplicates of an association already being built. +Answering one would fail that browser's live candidate pair. + +**Counters.** Refusals are counted in `rejected_connections`, as silent +rejections were. + +**Application-level rejections.** In ant-node, admission, channel, request, +rate and byte rejections after ICE and DTLS keep their current behaviour: the +node closes the association or channel. The client reads that close as the +refusal (section 3). + +This amends ADR-0015's rule that "capacity or rate rejection closes the +channel or association without sending an error response". At the transport, +capacity rejection of a proven association now sends one, under the existing +amplification bound. + +### 3. Tell a declining node from an unreachable one (ant-client, browser) + +**What counts as `Refused`.** A failed fresh dial is `Refused` when the node +answered it and turned it away: +- **Transport refusal:** the association never connected, and the browser + declared it failed within `REFUSAL_DETECTION_WINDOW` (3 s) of its ICE checks + starting; or +- **Application refusal:** ICE and DTLS completed, and the node then closed + the channel or association before the post-quantum session was established. + +**Cooldowns.** A refused endpoint is suppressed for `ENDPOINT_REFUSAL_COOLDOWN` +(15 s). Every other dial failure keeps the 30-minute cooldown. That includes +an association that connected and then failed authentication with its channel +still open. + +**Waiting less.** The channel-open wait now ends as soon as the browser +declares the association dead, instead of running out its 10-second timeout. + +**Why the window separates the two cases.** A refusal arrives within a few +round trips, one cookie round trip plus one retransmitted check: 66 ms in +Chromium and 100 ms in WebKit on loopback. Against a node that never answered +(the baseline below), both browsers were still checking when the client's +10-second channel-open timeout fired. Playwright Firefox reported its failed +dials after 5.0 s, which is also outside the window. + +**Why 15 s.** It is long enough not to redial a full node for every chunk of +a stream, since a 4 MiB chunk plays for several seconds at video bit rates. It +is short enough that the node rejoins lookups and reads soon after viewers +leave. + +## Consequences + +### Positive + +- **Up to seven times the serving capacity for a hot record,** with no + protocol change. First reads land evenly on its holders instead of on one. +- **A full node costs a browser a round trip, not ten seconds.** Local + Chromium and WebKit measurements (Validation) show 66–100 ms instead of + 10.0 s. +- **A full node rejoins rotation after 15 s instead of 30 minutes,** so + overload no longer shrinks the network a client can see. +- **Refusing costs the node almost nothing.** A STUN refusal allocates no RTC + state, and it is sent only to a source that has proved reachability, at most + once per returned cookie. +- **Mixed deployments behave at least as well as today:** + - New client, old node: the old node stays silent, and the client times out + and applies the 30-minute cooldown exactly as before. + - Old client, new node: the browser fails the association at once, but the + old client still waits for its channel-open timeout and records a 30-minute + failure, as before. + +### Negative / Trade-offs + +- **A random holder may not have the record yet,** for example a node that + just joined the close group and has not finished replicating. That costs one + `NotFound` round trip, and possibly a dial, before the next holder. The + network A/B in Validation measures it. +- **Spreading changes which node a client reads each record from.** It does + not change how many it reads from: one per record, plus the hedge. + Connection reuse may change slightly, because the nearest holder is no + longer favoured. +- **Transport refusals are recognised by timing.** Browser JavaScript cannot + read the STUN error code. A fast ICE failure for another reason, such as no + usable local candidate for the endpoint's address family or a client that + just went offline, is also classed `Refused` and gets 15 s instead of 30 + minutes. Those dials fail at once, so the extra retries are cheap. +- **A node or network that drops mid-handshake** after DTLS is also classed + `Refused`. That is acceptable: the node was demonstrably up seconds earlier. +- **Application refusals still cost the node a DTLS handshake** per attempt, + as today. Only transport refusals are cheaper. +- **Neither change adds capacity for finding 2.** Synchronised demand for one + chunk still outruns seven uplinks. Spreading and fast refusal make the best + use of the holders that exist. More holders need the deferred options. + +### Neutral / Operational + +- No limit changes here. Raising `max_connections`, + `max_concurrent_requests` and the transport's 256 ceiling is a separate + decision, based on a load test of CPU and memory per session. +- `rejected_connections` now counts answered refusals. An operator who sees + it rise knows browsers are being turned away, and no longer silently dropped. +- Once ADR-0020 (get-or-closer) lands, its holder choice should follow the + same rule: spread across the known close group, then nearest first. + +## Validation + +**Unit and regression tests.** +- saorsa-transport, `webrtc::direct`: + - A refusal echoes the request's transaction, passes the listener + credential's integrity check and fingerprint, and is no larger than the + request. + - An unparseable request is never answered. + - A listener at capacity and one at its source quota both answer a proven + check with 486 and their own reason phrase. In both cases + `rejected_connections` increments and no association is admitted. +- ant-client, `client_engine::read`: + - Over 7,000 trials, each close-group holder serves about 1/7 of first reads + (within ±300), and peers beyond the close group keep nearest-first order. + - Early reads choose within the known close group, then fall back to + nearest-first. + - An out-of-range choice stays inside the close group. +- ant-client, `EndpointFailureCache`: a refusal expires after its shorter + cooldown while a failure does not. +- ant-client, `wasm-tests/capacity-refusal.test.mjs`: + - A mock node that refuses at its transport, and one that closes during the + handshake, both fail the dial in under 2 s. + - Both are suppressed inside the cooldown, and both are dialled again and + connect after it. + - With refusal detection disabled, this test fails: both nodes stay + suppressed. + +**Real browsers (2026-10-12).** +- **Setup:** a local 7-node devnet with `max_connections` = 2 and + `max_connections_per_ip` = 1. One `BrowserNetworkClient` holds node 0's + per-source slot, a second client connects to the same seed, and the first + client leaves after the second is refused. This exercises the source-quota + refusal. A capacity refusal sends the same error code, so it is handled the + same way. + + | Browser | Refused after (baseline → this ADR) | Error shown | After 16 s, holder gone | + |---|---|---|---| + | Chromium | 10,011 ms → **66 ms** | "DataChannel opening timed out" → "peer connection failed" | still suppressed → **connects in 140 ms** | + | WebKit | 10,005 ms → **100 ms** | same | still suppressed → **connects in 122 ms** | + +- **Firefox:** Playwright Firefox does not establish WebRTC Direct sessions + with local nodes at all, even with loopback ICE enabled. This is independent + of this ADR, so the refusal path is unverified there. Its failed dials were + reported after 5.0 s, outside the 3 s window, and were correctly kept at the + 30-minute cooldown. + +**On a network, before Accepted.** +- Run an A/B with the WASM benchmark harness: one hot file read by N clients. + Compare the per-holder GET distribution, chunk read latency p50/p95, GETs per + chunk (the `NotFound` cost), and nodes' `rejected_connections`. +- Check mainnet nodes' `rejected_connections` to learn how many of the + endpoints that "time out when dialled" were full. + +**Review triggers.** +- A browser changes how it handles STUN error responses to connectivity + checks. +- ADR-0020 changes the read path. +- A hot-record caching ADR lands. + +## Notes for AI-assisted work + +AI tools may help draft this ADR, but **must not mark it Accepted without human review**. Accepted ADRs are immutable: create a new superseding ADR rather than editing an Accepted ADR. diff --git a/docs/adr/README.md b/docs/adr/README.md index a26b321f..dde9a529 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -38,3 +38,4 @@ See [`TOOLING.md`](./TOOLING.md) for `adrs`, `adr-kit`, and AI harness setup. - [ADR-0016: Pointers — paid mutable references with an immutable owner](./ADR-0016-pointers-immutable-owner.md) - [ADR-0017: Bounded fresh-replication offers and copy-free message sends](./ADR-0017-bounded-fresh-offers-and-copy-free-sends.md) - [ADR-0019: Pointer audits serve the record round 1 bound](./ADR-0019-pointer-audits-serve-the-record-round-one-bound.md) +- [ADR-0023: Spread hot-record reads across the close group, and refuse full browser sessions explicitly](./ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md) From 6de9aad012356bf5ff702bedaf219f62db5e9572 Mon Sep 17 00:00:00 2001 From: Mick van Dijke <12992260+mickvandijke@users.noreply.github.com> Date: Mon, 12 Oct 2026 16:10:06 +0200 Subject: [PATCH 2/6] docs(adr): align ADR-0023 with the review fixes - A transport refusal requires that ICE never connected; DTLS failures, authentication failures and client-initiated closes keep the 30-minute cooldown, and a full failure cache evicts the soonest-expiring record. - Record two trade-offs: a browser replacing its own association can hit its source quota, and only the admitting request is refused. - Update the real-browser measurements to the final client build. Co-Authored-By: Claude Opus 5.5 --- ...-reads-and-refuse-full-browser-sessions.md | 47 +++++++++++++------ 1 file changed, 33 insertions(+), 14 deletions(-) diff --git a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md index 26a6b833..0d208faa 100644 --- a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md +++ b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md @@ -173,23 +173,31 @@ amplification bound. **What counts as `Refused`.** A failed fresh dial is `Refused` when the node answered it and turned it away: -- **Transport refusal:** the association never connected, and the browser - declared it failed within `REFUSAL_DETECTION_WINDOW` (3 s) of its ICE checks - starting; or +- **Transport refusal:** ICE never connected, and the browser declared the + association failed within `REFUSAL_DETECTION_WINDOW` (3 s) of its ICE checks + starting. The test is "ICE never connected" rather than "ICE reads failed", + because Chromium declares the association failed while its legacy + `iceConnectionState` still reads "checking". Or: - **Application refusal:** ICE and DTLS completed, and the node then closed - the channel or association before the post-quantum session was established. + the channel, or the association died, before the post-quantum session was + established. **Cooldowns.** A refused endpoint is suppressed for `ENDPOINT_REFUSAL_COOLDOWN` -(15 s). Every other dial failure keeps the 30-minute cooldown. That includes -an association that connected and then failed authentication with its channel -still open. +(15 s). Every other dial failure keeps the 30-minute cooldown. That includes: +- DTLS failing after ICE connected, which means a different node answered at + the endpoint; +- an authentication failure with the channel still open; +- a lane the client closed itself because the node sent malformed data. + +A full failure cache evicts whichever record expires first, so a refusal +never pushes out a longer failure. **Waiting less.** The channel-open wait now ends as soon as the browser declares the association dead, instead of running out its 10-second timeout. **Why the window separates the two cases.** A refusal arrives within a few -round trips, one cookie round trip plus one retransmitted check: 66 ms in -Chromium and 100 ms in WebKit on loopback. Against a node that never answered +round trips, one cookie round trip plus one retransmitted check: 67 ms in +Chromium and 54 ms in WebKit on loopback. Against a node that never answered (the baseline below), both browsers were still checking when the client's 10-second channel-open timeout fired. Playwright Firefox reported its failed dials after 5.0 s, which is also outside the window. @@ -206,7 +214,7 @@ leave. - **Up to seven times the serving capacity for a hot record,** with no protocol change. First reads land evenly on its holders instead of on one. - **A full node costs a browser a round trip, not ten seconds.** Local - Chromium and WebKit measurements (Validation) show 66–100 ms instead of + Chromium and WebKit measurements (Validation) show 54–67 ms instead of 10.0 s. - **A full node rejoins rotation after 15 s instead of 30 minutes,** so overload no longer shrinks the network a client can see. @@ -235,6 +243,16 @@ leave. usable local candidate for the endpoint's address family or a client that just went offline, is also classed `Refused` and gets 15 s instead of 30 minutes. Those dials fail at once, so the extra retries are cheap. +- **A browser replacing its own association can hit its source quota.** Its + stale association still holds a slot until ICE closes it or the 10 s pending + expiry runs. Before, a silent listener could admit the retry once that slot + freed, if that happened before the client's 10 s timeout. Now the retry is + refused at once and the client tries again after 15 s. +- **Only the request that completes admission is refused.** A retransmission + of it that arrives later meets a fresh reachability challenge rather than + the same error. Chromium and WebKit failed the dial on the first refusal, + and the client closes the peer connection as soon as it is reported dead, + so later challenges reach a closed connection. - **A node or network that drops mid-handshake** after DTLS is also classed `Refused`. That is acceptable: the node was demonstrably up seconds earlier. - **Application refusals still cost the node a DTLS handshake** per attempt, @@ -277,8 +295,9 @@ leave. handshake, both fail the dial in under 2 s. - Both are suppressed inside the cooldown, and both are dialled again and connect after it. - - With refusal detection disabled, this test fails: both nodes stay - suppressed. + - A third mock node that fails DTLS after ICE connected is not dialled again. + - With refusal detection disabled, or with the earlier "dead and fast" test + that ignored ICE state, this test fails. **Real browsers (2026-10-12).** - **Setup:** a local 7-node devnet with `max_connections` = 2 and @@ -290,8 +309,8 @@ leave. | Browser | Refused after (baseline → this ADR) | Error shown | After 16 s, holder gone | |---|---|---|---| - | Chromium | 10,011 ms → **66 ms** | "DataChannel opening timed out" → "peer connection failed" | still suppressed → **connects in 140 ms** | - | WebKit | 10,005 ms → **100 ms** | same | still suppressed → **connects in 122 ms** | + | Chromium | 10,011 ms → **67 ms** | "DataChannel opening timed out" → "peer connection failed" | still suppressed → **connects in 132 ms** | + | WebKit | 10,005 ms → **54 ms** | same | still suppressed → **connects in 115 ms** | - **Firefox:** Playwright Firefox does not establish WebRTC Direct sessions with local nodes at all, even with loopback ICE enabled. This is independent From b42e790a5f3ee77f78c0534e35cbee4fe732ccc1 Mon Sep 17 00:00:00 2001 From: Mick van Dijke <12992260+mickvandijke@users.noreply.github.com> Date: Mon, 12 Oct 2026 16:16:01 +0200 Subject: [PATCH 3/6] docs(adr): correct ADR-0023 details found in a fact-check Spreading applies to chunk reads (pointer reads already ask the whole close group); known-peer fallbacks keep nearest-first; shutdown attempts stay silent; precise eviction and ICE-state wording; application-level closes from old nodes also get 15 s; admission tests listed under their module. Co-Authored-By: Claude Opus 5.5 --- ...-reads-and-refuse-full-browser-sessions.md | 45 ++++++++++++------- 1 file changed, 28 insertions(+), 17 deletions(-) diff --git a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md index 0d208faa..f789d0d8 100644 --- a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md +++ b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md @@ -112,11 +112,15 @@ useful. ### 1. Spread first reads across the close group (ant-client) -For every record, the read order is: -1. the close group, meaning the `CLOSE_GROUP_SIZE` nearest candidates known - at that moment, in uniformly random order; +For every chunk read, the read order is: +1. the close group, meaning the `CLOSE_GROUP_SIZE` nearest candidates that + discovery has found so far, in uniformly random order; 2. then the remaining candidates, nearest first. +Known-peer fallbacks, used when discovery comes up short, keep nearest-first +order. Pointer reads are unaffected: they already ask the whole close group +at once. + How each read path applies this: - **Early reads during discovery** choose uniformly among the unattempted members of the currently known close group. When all of them have been @@ -153,8 +157,11 @@ admitted it. - is never larger than the request it answers. If it would be, it is not sent. -**What stays silent.** Duplicates of an association already being built. -Answering one would fail that browser's live candidate pair. +**What stays silent.** +- Duplicates of an association already being built. Answering one would fail + that browser's live candidate pair. +- Attempts that arrive while the listener is shutting down (its accept queue + is closed). That is not capacity. **Counters.** Refusals are counted in `rejected_connections`, as silent rejections were. @@ -173,11 +180,12 @@ amplification bound. **What counts as `Refused`.** A failed fresh dial is `Refused` when the node answered it and turned it away: -- **Transport refusal:** ICE never connected, and the browser declared the - association failed within `REFUSAL_DETECTION_WINDOW` (3 s) of its ICE checks - starting. The test is "ICE never connected" rather than "ICE reads failed", - because Chromium declares the association failed while its legacy - `iceConnectionState` still reads "checking". Or: +- **Transport refusal:** the browser declared the association failed within + `REFUSAL_DETECTION_WINDOW` (3 s) of its ICE checks starting, without it ever + reaching connected, and with ICE not connected at that moment. The test is + "ICE not connected" rather than "ICE reads failed", because Chromium declares + the association failed while its legacy `iceConnectionState` still reads + "checking". Or: - **Application refusal:** ICE and DTLS completed, and the node then closed the channel, or the association died, before the post-quantum session was established. @@ -189,8 +197,8 @@ answered it and turned it away: - an authentication failure with the channel still open; - a lane the client closed itself because the node sent malformed data. -A full failure cache evicts whichever record expires first, so a refusal -never pushes out a longer failure. +A full failure cache evicts the stored record that expires soonest, so a +stored refusal is dropped before any longer-lived failure. **Waiting less.** The channel-open wait now ends as soon as the browser declares the association dead, instead of running out its 10-second timeout. @@ -222,8 +230,10 @@ leave. state, and it is sent only to a source that has proved reachability, at most once per returned cookie. - **Mixed deployments behave at least as well as today:** - - New client, old node: the old node stays silent, and the client times out - and applies the 30-minute cooldown exactly as before. + - New client, old node: at transport capacity the old node stays silent, and + the client times out and applies the 30-minute cooldown exactly as before. + Application-level closes behave the same on old and new nodes, so they get + the 15 s cooldown either way. - Old client, new node: the browser fails the association at once, but the old client still waits for its channel-open timeout and records a 30-minute failure, as before. @@ -274,11 +284,12 @@ leave. ## Validation **Unit and regression tests.** -- saorsa-transport, `webrtc::direct`: +- saorsa-transport, `webrtc::direct::admission`: - A refusal echoes the request's transaction, passes the listener credential's integrity check and fingerprint, and is no larger than the request. - An unparseable request is never answered. +- saorsa-transport, `webrtc::direct`: - A listener at capacity and one at its source quota both answer a proven check with 486 and their own reason phrase. In both cases `rejected_connections` increments and no association is admitted. @@ -293,8 +304,8 @@ leave. - ant-client, `wasm-tests/capacity-refusal.test.mjs`: - A mock node that refuses at its transport, and one that closes during the handshake, both fail the dial in under 2 s. - - Both are suppressed inside the cooldown, and both are dialled again and - connect after it. + - Both are suppressed inside the cooldown, both are dialled again after it, + and the client connects. - A third mock node that fails DTLS after ICE connected is not dialled again. - With refusal detection disabled, or with the earlier "dead and fast" test that ignored ICE state, this test fails. From b796535db5f25bf1704d6b20d19f3a144bc70d60 Mon Sep 17 00:00:00 2001 From: Mick van Dijke <12992260+mickvandijke@users.noreply.github.com> Date: Tue, 13 Oct 2026 11:40:42 +0200 Subject: [PATCH 4/6] feat(webrtc): raise the browser listener's session-shaped defaults Browser sessions go from 32 to 256, the transport's ceiling. Channels follow sessions (32 to 256), and request slots keep their one-per-two-sessions ratio (16 to 128): the PQ handshake and every request take a request slot, so more sessions without more slots would close their channels instead. The per-IP quota, rate buckets and in-flight byte budgets are unchanged, so frame and response memory stays bounded as before. ADR-0023 records the change and asks the network A/B to measure the per-session cost. Co-Authored-By: Claude Opus 5.5 --- docs/WEBRTC_DIRECT_TESTNET.md | 6 +- ...-reads-and-refuse-full-browser-sessions.md | 65 +++++++++++++++---- src/config.rs | 6 +- 3 files changed, 59 insertions(+), 18 deletions(-) diff --git a/docs/WEBRTC_DIRECT_TESTNET.md b/docs/WEBRTC_DIRECT_TESTNET.md index cae9d65b..da23a4ec 100644 --- a/docs/WEBRTC_DIRECT_TESTNET.md +++ b/docs/WEBRTC_DIRECT_TESTNET.md @@ -188,11 +188,11 @@ limits are not shared with browser traffic. The defaults are: | Setting | Default | Scope | |---|---:|---| -| `max_connections` | 32 | listener | +| `max_connections` | 256 | listener | | `max_connections_per_ip` | 4 | IPv4 address or IPv6 /64 | | `max_channels_per_connection` | 2 | association | -| `max_channels` | 32 | listener and channel-handler tasks | -| `max_concurrent_requests` | 16 | listener work slots | +| `max_channels` | 256 | listener and channel-handler tasks | +| `max_concurrent_requests` | 128 | listener work slots | | `max_requests_per_second` | 256 | listener work token bucket | | `max_requests_per_second_per_ip` | 32 | source-prefix work token bucket | | `max_requests_per_second_per_connection` | 16 | association work token bucket | diff --git a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md index f789d0d8..b320f46c 100644 --- a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md +++ b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md @@ -7,7 +7,7 @@ - **Supersedes:** none - **Superseded by:** none - **Related:** [ADR-0015](./ADR-0015-direct-browser-clients-over-webrtc-direct.md) - (amends its capacity-rejection rule), + (amends its capacity-rejection rule and three listener defaults), ADR-0020 (get-or-closer chunk reads, open in #249), ant-client ADR-0004 (direct browser read client), [RFC 8489 §6.3.4](https://www.rfc-editor.org/rfc/rfc8489#section-6.3.4) (STUN error responses), @@ -20,12 +20,13 @@ browser clients who all want the same chunks at once, for example because they are watching the same film? A node admits a bounded number of browser sessions (ADR-0015, "Public-listener -resource isolation"): +resource isolation"). Before this ADR the defaults were: | Setting | Default | |---|---:| | `max_connections` | 32 | | `max_connections_per_ip` | 4 | +| `max_channels` | 32 | | `max_concurrent_requests` | 16 | | `max_requests_per_second` | 256 | @@ -85,8 +86,9 @@ unverified, but each node's `rejected_connections` diagnostic can show it. 1. **Raise the per-node limits only.** Rejected as the sole measure. It is capped at 256 by the transport, and it fixes neither single-holder - concentration nor full nodes looking dead. Raising the limits after a load - test remains worthwhile and is left to a separate decision. + concentration nor full nodes looking dead. Raising the session-shaped + limits alongside the other changes is cheap, so we do that too (Decision, + section 4). 2. **Spread reads across the close group.** A client-only change to read order. 3. **An explicit "busy" signal.** Three ways to send one: 1. *A new browser-protocol status.* This needs a wire change across @@ -108,7 +110,8 @@ unverified, but each node's `rejected_connections` diagnostic can show it. ## Decision We adopt options 2 and 3.2, and the client classification that makes 3.2 -useful. +useful. We also raise the listener's session-shaped defaults (option 1, not +as the sole measure). ### 1. Spread first reads across the close group (ant-client) @@ -215,6 +218,34 @@ a stream, since a 4 MiB chunk plays for several seconds at video bit rates. It is short enough that the node rejoins lookups and reads soon after viewers leave. +### 4. Raise the session-shaped listener defaults (ant-node) + +| Setting | Before | Now | +|---|---:|---:| +| `max_connections` | 32 | 256 | +| `max_channels` | 32 | 256 | +| `max_concurrent_requests` | 16 | 128 | + +- **256 sessions is the transport's ceiling** (`MAX_PENDING_ASSOCIATIONS`). + Going higher needs a saorsa-transport change and the load test below. +- **Sessions alone would not help.** A session that opens a channel needs a + channel slot. The post-quantum handshake and every request take a request + slot, and when none is free the node closes the channel. With 16 request + slots, a node already turns away new channels once 16 GETs are queued. The + channel and request limits therefore keep their ratio to sessions: one + channel and half a request slot per session. +- **What does not change:** the per-IP session quota (4), channels per session + (2), every rate bucket, and both in-flight byte budgets. + - Frame and response memory is bounded by the byte budgets, not by slot + counts. A GET builds its response only while it holds one of the four + bulk-response slots that the 64 MiB budget allows. While it waits for one + it holds only its small request frame. + - The rate buckets bound handshake and request CPU. +- **What grows** is per-association state outside those budgets: ICE, DTLS and + SCTP state and tasks for each session, and one handler task per channel. + The network A/B measures node memory and CPU at the new limits. +- A node whose `--config` file sets these limits keeps its values. + ## Consequences ### Positive @@ -226,6 +257,8 @@ leave. 10.0 s. - **A full node rejoins rotation after 15 s instead of 30 minutes,** so overload no longer shrinks the network a client can see. +- **Eight times the browser sessions per node by default,** with the request + slots to serve them, under the same memory and rate budgets. - **Refusing costs the node almost nothing.** A STUN refusal allocates no RTC state, and it is sent only to a source that has proved reachability, at most once per returned cookie. @@ -267,15 +300,19 @@ leave. `Refused`. That is acceptable: the node was demonstrably up seconds earlier. - **Application refusals still cost the node a DTLS handshake** per attempt, as today. Only transport refusals are cheaper. -- **Neither change adds capacity for finding 2.** Synchronised demand for one - chunk still outruns seven uplinks. Spreading and fast refusal make the best - use of the holders that exist. More holders need the deferred options. +- **A saturated node holds more per-session state.** At 256 sessions each + association's ICE, DTLS and SCTP state and tasks sit outside the byte + budgets. Until the network A/B measures it, that cost is an estimate. +- **None of these changes adds capacity for finding 2.** Synchronised demand + for one chunk still outruns seven uplinks; more sessions only queue more + viewers on the same uplink. Spreading and fast refusal make the best use of + the holders that exist. More holders need the deferred options. ### Neutral / Operational -- No limit changes here. Raising `max_connections`, - `max_concurrent_requests` and the transport's 256 ceiling is a separate - decision, based on a load test of CPU and memory per session. +- The new defaults reach nodes on upgrade. Raising the transport's 256 + ceiling is a separate decision, based on the per-session cost the network + A/B measures. - `rejected_connections` now counts answered refusals. An operator who sees it rise knows browsers are being turned away, and no longer silently dropped. - Once ADR-0020 (get-or-closer) lands, its holder choice should follow the @@ -333,8 +370,12 @@ leave. - Run an A/B with the WASM benchmark harness: one hot file read by N clients. Compare the per-holder GET distribution, chunk read latency p50/p95, GETs per chunk (the `NotFound` cost), and nodes' `rejected_connections`. +- In the same run, measure node memory and CPU with sessions near the new + limits. - Check mainnet nodes' `rejected_connections` to learn how many of the - endpoints that "time out when dialled" were full. + endpoints that "time out when dialled" were full. The node binary does not + log or serve its listener diagnostics today, so this needs them exported + first. **Review triggers.** - A browser changes how it handles STUN error responses to connectivity diff --git a/src/config.rs b/src/config.rs index 725995a2..539fbaf5 100644 --- a/src/config.rs +++ b/src/config.rs @@ -244,7 +244,7 @@ fn default_webrtc_direct_bind() -> SocketAddr { } const fn default_webrtc_direct_max_connections() -> usize { - 32 + 256 } const fn default_webrtc_direct_max_connections_per_ip() -> usize { @@ -256,11 +256,11 @@ const fn default_webrtc_direct_max_channels_per_connection() -> usize { } const fn default_webrtc_direct_max_channels() -> usize { - 32 + 256 } const fn default_webrtc_direct_max_concurrent_requests() -> usize { - 16 + 128 } const fn default_webrtc_direct_max_requests_per_second() -> usize { From 21e44639cd13067cd0fcd598bd1fc03a660cd03c Mon Sep 17 00:00:00 2001 From: Mick van Dijke <12992260+mickvandijke@users.noreply.github.com> Date: Tue, 13 Oct 2026 15:57:08 +0200 Subject: [PATCH 5/6] feat(webrtc): size browser channels and request slots at two per session The testnet A/B showed browsers hold 1.4 channels per session at the median and 1.9 at p90, so channel slots ran out before session slots: the old 32 channels filled at 18-23 sessions, and with 256 sessions, 256 channels and 128 request slots both filled at 194 sessions under 192 browsers, closing about 2,300 channels a round. At 512 channels and 512 request slots no channel or request was rejected and all 768 downloads completed, at the same node memory (268 vs 271 MiB peak). ADR-0023 records the network A/B: at 64 synchronised browsers, today's limits completed 61-66% of downloads, all failures being false "record not found", and the new limits completed all of them. Co-Authored-By: Claude Opus 5.5 --- docs/WEBRTC_DIRECT_TESTNET.md | 4 +- ...-reads-and-refuse-full-browser-sessions.md | 130 ++++++++++++++---- src/config.rs | 4 +- 3 files changed, 111 insertions(+), 27 deletions(-) diff --git a/docs/WEBRTC_DIRECT_TESTNET.md b/docs/WEBRTC_DIRECT_TESTNET.md index da23a4ec..10a331ec 100644 --- a/docs/WEBRTC_DIRECT_TESTNET.md +++ b/docs/WEBRTC_DIRECT_TESTNET.md @@ -191,8 +191,8 @@ limits are not shared with browser traffic. The defaults are: | `max_connections` | 256 | listener | | `max_connections_per_ip` | 4 | IPv4 address or IPv6 /64 | | `max_channels_per_connection` | 2 | association | -| `max_channels` | 256 | listener and channel-handler tasks | -| `max_concurrent_requests` | 128 | listener work slots | +| `max_channels` | 512 | listener and channel-handler tasks | +| `max_concurrent_requests` | 512 | listener work slots | | `max_requests_per_second` | 256 | listener work token bucket | | `max_requests_per_second_per_ip` | 32 | source-prefix work token bucket | | `max_requests_per_second_per_connection` | 16 | association work token bucket | diff --git a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md index b320f46c..7cc5e786 100644 --- a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md +++ b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md @@ -223,17 +223,23 @@ leave. | Setting | Before | Now | |---|---:|---:| | `max_connections` | 32 | 256 | -| `max_channels` | 32 | 256 | -| `max_concurrent_requests` | 16 | 128 | +| `max_channels` | 32 | 512 | +| `max_concurrent_requests` | 16 | 512 | - **256 sessions is the transport's ceiling** (`MAX_PENDING_ASSOCIATIONS`). - Going higher needs a saorsa-transport change and the load test below. + Going higher needs a saorsa-transport change and a load test beyond the + one below. - **Sessions alone would not help.** A session that opens a channel needs a channel slot. The post-quantum handshake and every request take a request - slot, and when none is free the node closes the channel. With 16 request - slots, a node already turns away new channels once 16 GETs are queued. The - channel and request limits therefore keep their ratio to sessions: one - channel and half a request slot per session. + slot. When either kind of slot runs out, the node closes the channel. +- **Two channels and two request slots per session.** A browser opens up to + `max_channels_per_connection` (2) channels per session. On the testnet, + sessions held 1.4 channels at the median and 1.9 at p90. + - The old 32 channels ran out at 18–23 sessions, before the 32-session + limit, so channel slots, not sessions, turned browsers away. + - With 192 browsers, the busiest node in each round used about 360 + channels and 280 request slots. Limits of 256 channels and 128 request + slots both filled while sessions peaked at 194 of 256 (Validation). - **What does not change:** the per-IP session quota (4), channels per session (2), every rate bucket, and both in-flight byte budgets. - Frame and response memory is bounded by the byte budgets, not by slot @@ -243,7 +249,8 @@ leave. - The rate buckets bound handshake and request CPU. - **What grows** is per-association state outside those budgets: ICE, DTLS and SCTP state and tasks for each session, and one handler task per channel. - The network A/B measures node memory and CPU at the new limits. + On the testnet this cost about 0.5 MiB per session. A node peaked at + 184 MiB with 32 sessions and 268–271 MiB with 192. - A node whose `--config` file sets these limits keeps its values. ## Consequences @@ -252,13 +259,17 @@ leave. - **Up to seven times the serving capacity for a hot record,** with no protocol change. First reads land evenly on its holders instead of on one. + On the testnet, the share of a chunk's first reads that went to its most + popular holder fell from 48% to 18–21%. - **A full node costs a browser a round trip, not ten seconds.** Local Chromium and WebKit measurements (Validation) show 54–67 ms instead of 10.0 s. - **A full node rejoins rotation after 15 s instead of 30 minutes,** so overload no longer shrinks the network a client can see. -- **Eight times the browser sessions per node by default,** with the request - slots to serve them, under the same memory and rate budgets. +- **Eight times the browser sessions per node by default,** with the channel + and request slots to use them, under the same memory and rate budgets. On + the testnet this turned 64 synchronised downloads from 61–66% complete into + 100% complete. - **Refusing costs the node almost nothing.** A STUN refusal allocates no RTC state, and it is sent only to a source that has proved reachability, at most once per returned cookie. @@ -275,8 +286,9 @@ leave. - **A random holder may not have the record yet,** for example a node that just joined the close group and has not finished replicating. That costs one - `NotFound` round trip, and possibly a dial, before the next holder. The - network A/B in Validation measures it. + `NotFound` round trip, and possibly a dial, before the next holder. On the + testnet, `NotFound` answers per chunk rose from 1.05 to 1.39 with spreading + and the new limits, and GETs per chunk from 3.37 to 3.45. - **Spreading changes which node a client reads each record from.** It does not change how many it reads from: one per record, plus the hedge. Connection reuse may change slightly, because the nearest holder is no @@ -300,9 +312,25 @@ leave. `Refused`. That is acceptable: the node was demonstrably up seconds earlier. - **Application refusals still cost the node a DTLS handshake** per attempt, as today. Only transport refusals are cheaper. -- **A saturated node holds more per-session state.** At 256 sessions each - association's ICE, DTLS and SCTP state and tasks sit outside the byte - budgets. Until the network A/B measures it, that cost is an estimate. +- **A saturated node holds more per-session state.** Each association's ICE, + DTLS and SCTP state and tasks sit outside the byte budgets. The testnet + measured about 0.5 MiB per session, so a full listener costs about 130 MiB + more than an idle one. That matters to operators who run many nodes on one + host. +- **A read still gives up before a refusal's cooldown ends.** If no holder + answers, the read engine retries the close group once, after 1 s, and the + browser then reports the record as not found. A holder that refused is + suppressed for 15 s, so the faster cooldown does not help a read already in + flight. Wherever nodes fill, a busy holder still surfaces as a false "record + not found": + - on the testnet at the old limits, 34–39% of synchronised downloads failed + this way, with today's client and with this one alike; + - at the adopted limits none of 1,152 downloads failed. With 256 channels + and 128 request slots, which filled before sessions did at 192 browsers, + 6 of 768 failed this way. + + Distinguishing "no holder answered" from "absent", and waiting out the + shortest refusal cooldown before giving up, is follow-up client work. - **None of these changes adds capacity for finding 2.** Synchronised demand for one chunk still outruns seven uplinks; more sessions only queue more viewers on the same uplink. Spreading and fast refusal make the best use of @@ -311,8 +339,8 @@ leave. ### Neutral / Operational - The new defaults reach nodes on upgrade. Raising the transport's 256 - ceiling is a separate decision, based on the per-session cost the network - A/B measures. + ceiling is a separate decision, based on the per-session cost measured in + Validation. - `rejected_connections` now counts answered refusals. An operator who sees it rise knows browsers are being turned away, and no longer silently dropped. - Once ADR-0020 (get-or-closer) lands, its holder choice should follow the @@ -366,16 +394,72 @@ leave. reported after 5.0 s, outside the 3 s window, and were correctly kept at the 30-minute cooldown. -**On a network, before Accepted.** -- Run an A/B with the WASM benchmark harness: one hot file read by N clients. - Compare the per-holder GET distribution, chunk read latency p50/p95, GETs per - chunk (the `NotFound` cost), and nodes' `rejected_connections`. -- In the same run, measure node memory and CPU with sessions near the new - limits. +**On a network (2026-10-13).** +- **Setup:** + - A 60-node testnet across 9 regions. 18 nodes sit behind symmetric NAT, + where browsers cannot reach them. + - 64 browser droplets in lon1, nyc1, sfo3 and sgp1, each running one + headless Chromium on its own IP. + - In each round every browser downloads the same 50 MiB private file + (16 chunks) with a fresh client, all starting within the same 10 s. + Three files rotate between rounds. Half the browsers run a traced build. + - Arms ran in blocks of three rounds, in the order A B C C B A. Each block + follows a restart of every node. + - Nodes logged their listener counters every 5 s. This logging was in the + test build only. + - Scripts and raw data: `ant-testnet/state/comparisons/v2-1508-hot-reads-ab-1013`. + + | 64 browsers | A: today | C: new code, old limits | B: new code, 256 sessions | + |---|--:|--:|--:| + | Downloads completed | 252/384 (66%) | 236/384 (61%) | 384/384 | + | Most popular holder's share of a chunk's first reads | 48% | 18% | 21% | + | Channels a full node closed (traced browsers) | 2,477 | 1,953 | 0 | + | Silent transport drops / refusals per round | 1,413 | 238 | 0 | + | Channel / request rejections per round | 842 / 108 | 714 / 64 | 0 / 10 | + | Node peak memory / CPU | 184 MiB / 1.25 cores | 177 MiB / 1.04 | 203 MiB / 1.19 | + + B ran with 256 channels and 128 request slots, below the adopted 512. +- **Scale:** 192 browsers (three per droplet), with arms in the order B D D B + and two rounds per block: + + | 192 browsers, 256 sessions | 256 channels / 128 requests | 512 / 512 (adopted) | + |---|--:|--:| + | Downloads completed | 762/768 | 768/768 | + | Peak sessions / channels / requests in use | 194 / 256 / 128 | 192 / 362 / 284 | + | Channel / request rejections per round | 1,583 / 707 | 0 / 0 | + | Node peak memory / CPU | 271 MiB / 1.22 cores | 268 MiB / 1.28 cores | + +- **What the run showed:** + - Every failed download ended in a false "record not found" (Negative / + Trade-offs). Two browsers reading the same files on their own finished in + 19 s and 54 s. + - At the old limits, channel slots ran out before sessions. Today's client + and the new one failed at the same rate. + - Spreading cut the most popular holder's share of a chunk's first reads + from 48% to 18–21%. Per-node bytes barely changed: the most-loaded node + sent 5–7% of a round's bytes in every arm, because a 16-chunk file + already spans many close groups. + - Fast transport refusal did little at this scale. Full nodes mostly turned + browsers away at the channel limit, after DTLS, where the client already + saw a fast close. + - Dials that timed out after 10 s went to nodes behind NAT: 1,951 of 1,952 + in A and all 3,302 in B. Those nodes advertise WebRTC endpoints that + browsers cannot reach, the testnet counterpart of the mainnet endpoints + that "time out when dialled". + - A restart of every node penalises the next round. Arm A's first round, + 90 s after a restart, completed 23 of 64. Every later block waited 240 s. + Without that round, A completed 229 of 320 (72%). + - Unrelated to this ADR, in every arm: under load, a seed node occasionally + returned an ML-DSA-65 handshake signature that the browser rejected + (102 of 148,755 traced connects). + +**Before Accepted.** - Check mainnet nodes' `rejected_connections` to learn how many of the endpoints that "time out when dialled" were full. The node binary does not log or serve its listener diagnostics today, so this needs them exported first. +- Decide the client follow-up for reads that end while their holders are + refusing (Negative / Trade-offs). **Review triggers.** - A browser changes how it handles STUN error responses to connectivity diff --git a/src/config.rs b/src/config.rs index 539fbaf5..d7828a43 100644 --- a/src/config.rs +++ b/src/config.rs @@ -256,11 +256,11 @@ const fn default_webrtc_direct_max_channels_per_connection() -> usize { } const fn default_webrtc_direct_max_channels() -> usize { - 256 + 512 } const fn default_webrtc_direct_max_concurrent_requests() -> usize { - 128 + 512 } const fn default_webrtc_direct_max_requests_per_second() -> usize { From 93bcf01b8790d7311e67316273243d8f31e17bac Mon Sep 17 00:00:00 2001 From: Mick van Dijke <12992260+mickvandijke@users.noreply.github.com> Date: Wed, 14 Oct 2026 15:43:36 +0200 Subject: [PATCH 6/6] docs(adr): ADR-0023 adopts waiting out a holder's refusal before reporting a record absent The client follow-up the ADR left open is now in ant-client #225. Section 5 describes it, the Negative bullet that called it future work now states its cost, Validation lists its tests, and "Before Accepted" asks for a network run of it instead of a decision. Related links the three implementing PRs. Co-Authored-By: Claude Opus 5.5 --- ...-reads-and-refuse-full-browser-sessions.md | 96 +++++++++++++++---- 1 file changed, 77 insertions(+), 19 deletions(-) diff --git a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md index 7cc5e786..656ccff0 100644 --- a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md +++ b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md @@ -12,6 +12,8 @@ ant-client ADR-0004 (direct browser read client), [RFC 8489 §6.3.4](https://www.rfc-editor.org/rfc/rfc8489#section-6.3.4) (STUN error responses), [RFC 8445 §7.2.5.2.4](https://www.rfc-editor.org/rfc/rfc8445#section-7.2.5.2.4) (connectivity-check failure) +- **Implementation:** ant-client #225 (sections 1, 3 and 5), + saorsa-transport #183 (section 2), ant-node #250 (section 4) ## Context @@ -111,7 +113,9 @@ unverified, but each node's `rejected_connections` diagnostic can show it. We adopt options 2 and 3.2, and the client classification that makes 3.2 useful. We also raise the listener's session-shaped defaults (option 1, not -as the sole measure). +as the sole measure). Finally, a read waits out a holder's refusal before it +reports the record absent, so a busy holder is not mistaken for a missing +record. ### 1. Spread first reads across the close group (ant-client) @@ -253,6 +257,39 @@ leave. 184 MiB with 32 sessions and 268–271 MiB with 192. - A node whose `--config` file sets these limits keeps its values. +### 5. Wait out a holder's refusal before reporting a record absent (ant-client) + +**The problem.** A refused holder is suppressed for 15 s (section 3), and a +suppressed peer is left out of every lookup and read. A read that finds no +holder retries the close group once, after 1 s, and then reports the record +not found. So a read that starts while the holders it can find are refusing +reports a false "record not found", and it can even meet the absence rule, +because only the peers left in view are counted. On the testnet at the old +limits, every failed download ended this way (Validation). + +**What changes.** When a round ends without the record, the read engine asks +the client which peers it is suppressing for a refusal, and how long each has +left. A refused peer may hold the record unless it answered `NotFound` itself, +or `CLOSE_GROUP_SIZE` peers nearer the record did. The test counts peers +refused when the round started as well as at its end. A refusal that ended +during the round counts as ready now. + +While any refused peer may hold the record: +- the read does not report the record absent, even if the absence rule holds; +- it waits until the first such refusal ends, and at least the usual 1 s, + then runs another round; +- after `MAX_REFUSAL_WAITS` (3) waits it fails with an error saying the nodes + that may hold the record are busy, rather than that the record was not found. + +Otherwise the read keeps today's rule: one retry after 1 s, then absence. + +**Cost.** At most 45 s and three lookups more, and only for a read that would +otherwise have failed. + +**Scope.** Only the browser client classifies refusals, so native reads are +unchanged. The browser client's network adapter gains a `refused_peers()` +hook, which defaults to none. + ## Consequences ### Positive @@ -266,6 +303,10 @@ leave. 10.0 s. - **A full node rejoins rotation after 15 s instead of 30 minutes,** so overload no longer shrinks the network a client can see. +- **A busy holder no longer looks like a missing record.** A read waits for + refused holders to return, and if they keep refusing it reports them busy. + In a mock network whose only holder refuses once, the download completes + after 15 s instead of failing with "record not found" (Validation). - **Eight times the browser sessions per node by default,** with the channel and request slots to use them, under the same memory and rate budgets. On the testnet this turned 64 synchronised downloads from 61–66% complete into @@ -281,6 +322,8 @@ leave. - Old client, new node: the browser fails the association at once, but the old client still waits for its channel-open timeout and records a 30-minute failure, as before. + - Section 5 needs only the client. It applies to every refusal the client + classifies, including application-level closes from old nodes. ### Negative / Trade-offs @@ -317,20 +360,18 @@ leave. measured about 0.5 MiB per session, so a full listener costs about 130 MiB more than an idle one. That matters to operators who run many nodes on one host. -- **A read still gives up before a refusal's cooldown ends.** If no holder - answers, the read engine retries the close group once, after 1 s, and the - browser then reports the record as not found. A holder that refused is - suppressed for 15 s, so the faster cooldown does not help a read already in - flight. Wherever nodes fill, a busy holder still surfaces as a false "record - not found": - - on the testnet at the old limits, 34–39% of synchronised downloads failed - this way, with today's client and with this one alike; - - at the adopted limits none of 1,152 downloads failed. With 256 channels +- **A read whose holders keep refusing takes up to 45 s longer to fail** + (section 5), and the application must still retry later or show the "busy" + error. The wait cannot tell a full holder from one that has lost the + record, so a read may wait for a refused peer that turns out not to have it. +- **Wherever holders stay full, reads still fail.** Section 5 changes how they + fail, not whether. On the testnet, before section 5, every failed download + was a busy holder reported as a false "record not found": + - at the old limits, 34–39% of synchronised downloads failed this way, with + today's client and with sections 1 and 3 alike; + - at the adopted limits, none of 1,152 downloads failed. With 256 channels and 128 request slots, which filled before sessions did at 192 browsers, 6 of 768 failed this way. - - Distinguishing "no holder answered" from "absent", and waiting out the - shortest refusal cooldown before giving up, is follow-up client work. - **None of these changes adds capacity for finding 2.** Synchronised demand for one chunk still outruns seven uplinks; more sessions only queue more viewers on the same uplink. Spreading and fast refusal make the best use of @@ -365,7 +406,16 @@ leave. nearest-first. - An out-of-range choice stays inside the close group. - ant-client, `EndpointFailureCache`: a refusal expires after its shorter - cooldown while a failure does not. + cooldown while a failure does not, and only unexpired refusals are listed + with their remaining time. +- ant-client, `client_engine::read` (section 5), on a simulated clock: + - A read whose holder is refused and hidden from discovery waits out the + 15 s refusal and reads the record. + - A refusal that ends during a round is retried after the usual 1 s. + - A holder that keeps refusing makes the read fail as busy after three + waits of 15 s, not as absent. + - A refused peer with a close group of nearer `NotFound` answers does not + delay absence. - ant-client, `wasm-tests/capacity-refusal.test.mjs`: - A mock node that refuses at its transport, and one that closes during the handshake, both fail the dial in under 2 s. @@ -374,6 +424,13 @@ leave. - A third mock node that fails DTLS after ICE connected is not dialled again. - With refusal detection disabled, or with the earlier "dead and fast" test that ignored ICE state, this test fails. +- ant-client, `wasm-tests/refused-holder.test.mjs` (section 5): a client + whose seed points at a single holder that declines its first dial: + - downloads the file after about 15 s; + - when the holder declines every dial, fails after about 45 s with an error + that says "busy" and not "not found"; + - without section 5, fails with "record … not found", the error the + testnet's failed downloads ended with. **Real browsers (2026-10-12).** - **Setup:** a local 7-node devnet with `max_connections` = 2 and @@ -430,9 +487,9 @@ leave. | Node peak memory / CPU | 271 MiB / 1.22 cores | 268 MiB / 1.28 cores | - **What the run showed:** - - Every failed download ended in a false "record not found" (Negative / - Trade-offs). Two browsers reading the same files on their own finished in - 19 s and 54 s. + - Every failed download ended in a false "record not found". Two browsers + reading the same files on their own finished in 19 s and 54 s. Section 5 + addresses this. It came after this run and has not run on a network. - At the old limits, channel slots ran out before sessions. Today's client and the new one failed at the same rate. - Spreading cut the most popular holder's share of a chunk's first reads @@ -458,8 +515,9 @@ leave. endpoints that "time out when dialled" were full. The node binary does not log or serve its listener diagnostics today, so this needs them exported first. -- Decide the client follow-up for reads that end while their holders are - refusing (Negative / Trade-offs). +- Run section 5 on a network. Re-run the old-limits arm (C) with it, and + check that downloads which failed with "record not found" now complete or + report busy. **Review triggers.** - A browser changes how it handles STUN error responses to connectivity