diff --git a/docs/WEBRTC_DIRECT_TESTNET.md b/docs/WEBRTC_DIRECT_TESTNET.md index cae9d65b..10a331ec 100644 --- a/docs/WEBRTC_DIRECT_TESTNET.md +++ b/docs/WEBRTC_DIRECT_TESTNET.md @@ -188,11 +188,11 @@ limits are not shared with browser traffic. The defaults are: | Setting | Default | Scope | |---|---:|---| -| `max_connections` | 32 | listener | +| `max_connections` | 256 | listener | | `max_connections_per_ip` | 4 | IPv4 address or IPv6 /64 | | `max_channels_per_connection` | 2 | association | -| `max_channels` | 32 | listener and channel-handler tasks | -| `max_concurrent_requests` | 16 | listener work slots | +| `max_channels` | 512 | listener and channel-handler tasks | +| `max_concurrent_requests` | 512 | listener work slots | | `max_requests_per_second` | 256 | listener work token bucket | | `max_requests_per_second_per_ip` | 32 | source-prefix work token bucket | | `max_requests_per_second_per_connection` | 16 | association work token bucket | diff --git a/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md new file mode 100644 index 00000000..656ccff0 --- /dev/null +++ b/docs/adr/ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md @@ -0,0 +1,530 @@ +# ADR-0023: Spread hot-record reads across the close group, and refuse full browser sessions explicitly + +- **Status:** Proposed +- **Date:** 2026-10-12 +- **Decision owners:** +- **Reviewers:** +- **Supersedes:** none +- **Superseded by:** none +- **Related:** [ADR-0015](./ADR-0015-direct-browser-clients-over-webrtc-direct.md) + (amends its capacity-rejection rule and three listener defaults), + ADR-0020 (get-or-closer chunk reads, open in #249), + ant-client ADR-0004 (direct browser read client), + [RFC 8489 §6.3.4](https://www.rfc-editor.org/rfc/rfc8489#section-6.3.4) (STUN error responses), + [RFC 8445 §7.2.5.2.4](https://www.rfc-editor.org/rfc/rfc8445#section-7.2.5.2.4) (connectivity-check failure) +- **Implementation:** ant-client #225 (sections 1, 3 and 5), + saorsa-transport #183 (section 2), ant-node #250 (section 4) + +## Context + +The question that prompted this ADR: can the network serve thousands of +browser clients who all want the same chunks at once, for example because they +are watching the same film? + +A node admits a bounded number of browser sessions (ADR-0015, "Public-listener +resource isolation"). Before this ADR the defaults were: + +| Setting | Default | +|---|---:| +| `max_connections` | 32 | +| `max_connections_per_ip` | 4 | +| `max_channels` | 32 | +| `max_concurrent_requests` | 16 | +| `max_requests_per_second` | 256 | + +saorsa-transport refuses to bind a listener with `max_connections` above 256 +(`MAX_PENDING_ASSOCIATIONS`). Configuration can raise the limits below that, +but only through a `--config` file. No CLI flag sets them. + +Three findings matter more than the limits themselves. + +**1. Every client reads a record from the same holder.** Each record is stored +on its close group of `CLOSE_GROUP_SIZE` = 7 nodes. Both read paths in the +shared client engine (`ant-core/src/client_engine/read.rs`, used by native and +browser clients) order candidates by XOR distance. Early reads during +discovery always take the nearest unattempted hint. The final ordinary path +sorts the close group nearest-first. A second holder is asked only after the +1-second hedge delay or a failure. So every client sends its first GET for a +record to the same node, and a popular record is served by one node while +the other six idle. + +**2. Upload bandwidth runs out before session slots do.** Take a 5 Mbit/s +stream and 4 MiB chunks, so each chunk is about 7 seconds of playback. If +1,000 viewers are at the same point, one chunk window asks for 4 GiB in about +7 seconds, roughly 5 Gbit/s. One node cannot supply that. Spread evenly over +all seven holders it is about 700 Mbit/s each. Viewers who started at +different times spread across the file's chunks, but every viewer reads the +data map and the first chunks, so those are hot whether or not viewers are in +sync. Raising session limits alone admits more viewers to the same uplink. + +**3. A full node looks unreachable.** +- When a proven association finds no free slot, the transport ignores the + browser's ICE check without replying (`DirectUdpMux::run`). +- Application-level capacity and rate rejections close the channel or + association, also without a reply. ADR-0015 chose this to avoid amplifying + attacker traffic. +- The browser therefore waits out its 10-second channel-open timeout. It then + records a dial failure, and `ENDPOINT_FAILURE_COOLDOWN` keeps that node out + of every lookup and read for **30 minutes**. + +Under load, busy nodes vanish from each client's view one by one, so the +network looks smaller exactly when demand peaks. ant-client's lookup notes say +a third to a half of the WebRTC Direct endpoints discovered on mainnet time +out when dialled. Some of those may be full rather than down. That is +unverified, but each node's `rejected_connections` diagnostic can show it. + +## Decision Drivers + +- Use the capacity of the whole close group before anything costlier. +- A busy node should cost a client about one round trip, not ten seconds, + and should come back into rotation in seconds, not half an hour. +- No new wire message, no protocol version, and no flag day. Every mix of old + and new nodes and clients must behave at least as well as today. +- Keep ADR-0015's resource bounds and its refusal to amplify traffic. +- Leave the larger fixes for hot content (more holders, delegated lookups) + open for their own decisions. + +## Considered Options + +1. **Raise the per-node limits only.** Rejected as the sole measure. It is + capped at 256 by the transport, and it fixes neither single-holder + concentration nor full nodes looking dead. Raising the session-shaped + limits alongside the other changes is cheap, so we do that too (Decision, + section 4). +2. **Spread reads across the close group.** A client-only change to read order. +3. **An explicit "busy" signal.** Three ways to send one: + 1. *A new browser-protocol status.* This needs a wire change across + saorsa-transport, ant-node and ant-client. Worse, sending it needs a + full ICE, DTLS and SCTP association for every refused attempt, which is + exactly the cost the session limit exists to bound. + 2. *A STUN error response at transport admission.* A browser fails a + candidate pair as soon as its connectivity check gets an error response + (RFC 8445 §7.2.5.2.4). A refusal costs the node one STUN parse, one HMAC + and one send, after the reachability-cookie round trip it already + requires. No wire change. + 3. *Client heuristics alone,* such as a shorter cooldown for every dial + failure. Rejected: clients would redial genuinely unreachable endpoints + every few seconds, at ten seconds per attempt. +4. **More holders for hot records** (caching along lookup paths), browser + peer-to-peer sharing, or a CDN-backed HTTP gateway. These are the real + answer to finding 2, but each needs its own ADR. Deferred. + +## Decision + +We adopt options 2 and 3.2, and the client classification that makes 3.2 +useful. We also raise the listener's session-shaped defaults (option 1, not +as the sole measure). Finally, a read waits out a holder's refusal before it +reports the record absent, so a busy holder is not mistaken for a missing +record. + +### 1. Spread first reads across the close group (ant-client) + +For every chunk read, the read order is: +1. the close group, meaning the `CLOSE_GROUP_SIZE` nearest candidates that + discovery has found so far, in uniformly random order; +2. then the remaining candidates, nearest first. + +Known-peer fallbacks, used when discovery comes up short, keep nearest-first +order. Pointer reads are unaffected: they already ask the whole close group +at once. + +How each read path applies this: +- **Early reads during discovery** choose uniformly among the unattempted + members of the currently known close group. When all of them have been + tried, the nearest remaining hint is next. +- **The ordinary path** shuffles the close group at the front of its final + candidate list. + +The hedge delay, the early-read allowance, the at-most-two-GETs rule and the +absence rule are unchanged. The engine takes the choice as a parameter. +Production passes a uniform random choice, and tests pass nearest-first to pin +order. Native clients use the same engine, so they spread too. + +### 2. Refuse proven associations explicitly when full (saorsa-transport) + +**When it applies.** An association becomes proven when it has returned the +source-bound reachability cookie (ADR-0015). If a proven association then +finds no slot, because the listener is at `max_connections`, the pending queue +is full or the accept queue is full, or because its IPv4 address or IPv6 /64 +holds its full share, the mux no longer ignores the request that would have +admitted it. + +**What the mux sends.** It answers with a STUN Binding error response that: +- carries the same transaction ID as the request; +- has `ERROR-CODE` 486 (*Allocation Quota Reached*), with the reason phrase + "Insufficient Capacity" or "Source Quota Reached". A 4xx code ends the STUN + transaction (RFC 8489 §6.3.4), so any compliant ICE agent fails the pair at + once. A 5xx code such as 508 *Insufficient Capacity* may be retried, so it + is not used. Browsers do not expose STUN errors to applications; the phrase + only makes packet captures readable. +- carries `MESSAGE-INTEGRITY` keyed by the listener credential that + authenticated the request, so the browser accepts it as the answer to its + own check; +- carries `FINGERPRINT`; +- is never larger than the request it answers. If it would be, it is not + sent. + +**What stays silent.** +- Duplicates of an association already being built. Answering one would fail + that browser's live candidate pair. +- Attempts that arrive while the listener is shutting down (its accept queue + is closed). That is not capacity. + +**Counters.** Refusals are counted in `rejected_connections`, as silent +rejections were. + +**Application-level rejections.** In ant-node, admission, channel, request, +rate and byte rejections after ICE and DTLS keep their current behaviour: the +node closes the association or channel. The client reads that close as the +refusal (section 3). + +This amends ADR-0015's rule that "capacity or rate rejection closes the +channel or association without sending an error response". At the transport, +capacity rejection of a proven association now sends one, under the existing +amplification bound. + +### 3. Tell a declining node from an unreachable one (ant-client, browser) + +**What counts as `Refused`.** A failed fresh dial is `Refused` when the node +answered it and turned it away: +- **Transport refusal:** the browser declared the association failed within + `REFUSAL_DETECTION_WINDOW` (3 s) of its ICE checks starting, without it ever + reaching connected, and with ICE not connected at that moment. The test is + "ICE not connected" rather than "ICE reads failed", because Chromium declares + the association failed while its legacy `iceConnectionState` still reads + "checking". Or: +- **Application refusal:** ICE and DTLS completed, and the node then closed + the channel, or the association died, before the post-quantum session was + established. + +**Cooldowns.** A refused endpoint is suppressed for `ENDPOINT_REFUSAL_COOLDOWN` +(15 s). Every other dial failure keeps the 30-minute cooldown. That includes: +- DTLS failing after ICE connected, which means a different node answered at + the endpoint; +- an authentication failure with the channel still open; +- a lane the client closed itself because the node sent malformed data. + +A full failure cache evicts the stored record that expires soonest, so a +stored refusal is dropped before any longer-lived failure. + +**Waiting less.** The channel-open wait now ends as soon as the browser +declares the association dead, instead of running out its 10-second timeout. + +**Why the window separates the two cases.** A refusal arrives within a few +round trips, one cookie round trip plus one retransmitted check: 67 ms in +Chromium and 54 ms in WebKit on loopback. Against a node that never answered +(the baseline below), both browsers were still checking when the client's +10-second channel-open timeout fired. Playwright Firefox reported its failed +dials after 5.0 s, which is also outside the window. + +**Why 15 s.** It is long enough not to redial a full node for every chunk of +a stream, since a 4 MiB chunk plays for several seconds at video bit rates. It +is short enough that the node rejoins lookups and reads soon after viewers +leave. + +### 4. Raise the session-shaped listener defaults (ant-node) + +| Setting | Before | Now | +|---|---:|---:| +| `max_connections` | 32 | 256 | +| `max_channels` | 32 | 512 | +| `max_concurrent_requests` | 16 | 512 | + +- **256 sessions is the transport's ceiling** (`MAX_PENDING_ASSOCIATIONS`). + Going higher needs a saorsa-transport change and a load test beyond the + one below. +- **Sessions alone would not help.** A session that opens a channel needs a + channel slot. The post-quantum handshake and every request take a request + slot. When either kind of slot runs out, the node closes the channel. +- **Two channels and two request slots per session.** A browser opens up to + `max_channels_per_connection` (2) channels per session. On the testnet, + sessions held 1.4 channels at the median and 1.9 at p90. + - The old 32 channels ran out at 18–23 sessions, before the 32-session + limit, so channel slots, not sessions, turned browsers away. + - With 192 browsers, the busiest node in each round used about 360 + channels and 280 request slots. Limits of 256 channels and 128 request + slots both filled while sessions peaked at 194 of 256 (Validation). +- **What does not change:** the per-IP session quota (4), channels per session + (2), every rate bucket, and both in-flight byte budgets. + - Frame and response memory is bounded by the byte budgets, not by slot + counts. A GET builds its response only while it holds one of the four + bulk-response slots that the 64 MiB budget allows. While it waits for one + it holds only its small request frame. + - The rate buckets bound handshake and request CPU. +- **What grows** is per-association state outside those budgets: ICE, DTLS and + SCTP state and tasks for each session, and one handler task per channel. + On the testnet this cost about 0.5 MiB per session. A node peaked at + 184 MiB with 32 sessions and 268–271 MiB with 192. +- A node whose `--config` file sets these limits keeps its values. + +### 5. Wait out a holder's refusal before reporting a record absent (ant-client) + +**The problem.** A refused holder is suppressed for 15 s (section 3), and a +suppressed peer is left out of every lookup and read. A read that finds no +holder retries the close group once, after 1 s, and then reports the record +not found. So a read that starts while the holders it can find are refusing +reports a false "record not found", and it can even meet the absence rule, +because only the peers left in view are counted. On the testnet at the old +limits, every failed download ended this way (Validation). + +**What changes.** When a round ends without the record, the read engine asks +the client which peers it is suppressing for a refusal, and how long each has +left. A refused peer may hold the record unless it answered `NotFound` itself, +or `CLOSE_GROUP_SIZE` peers nearer the record did. The test counts peers +refused when the round started as well as at its end. A refusal that ended +during the round counts as ready now. + +While any refused peer may hold the record: +- the read does not report the record absent, even if the absence rule holds; +- it waits until the first such refusal ends, and at least the usual 1 s, + then runs another round; +- after `MAX_REFUSAL_WAITS` (3) waits it fails with an error saying the nodes + that may hold the record are busy, rather than that the record was not found. + +Otherwise the read keeps today's rule: one retry after 1 s, then absence. + +**Cost.** At most 45 s and three lookups more, and only for a read that would +otherwise have failed. + +**Scope.** Only the browser client classifies refusals, so native reads are +unchanged. The browser client's network adapter gains a `refused_peers()` +hook, which defaults to none. + +## Consequences + +### Positive + +- **Up to seven times the serving capacity for a hot record,** with no + protocol change. First reads land evenly on its holders instead of on one. + On the testnet, the share of a chunk's first reads that went to its most + popular holder fell from 48% to 18–21%. +- **A full node costs a browser a round trip, not ten seconds.** Local + Chromium and WebKit measurements (Validation) show 54–67 ms instead of + 10.0 s. +- **A full node rejoins rotation after 15 s instead of 30 minutes,** so + overload no longer shrinks the network a client can see. +- **A busy holder no longer looks like a missing record.** A read waits for + refused holders to return, and if they keep refusing it reports them busy. + In a mock network whose only holder refuses once, the download completes + after 15 s instead of failing with "record not found" (Validation). +- **Eight times the browser sessions per node by default,** with the channel + and request slots to use them, under the same memory and rate budgets. On + the testnet this turned 64 synchronised downloads from 61–66% complete into + 100% complete. +- **Refusing costs the node almost nothing.** A STUN refusal allocates no RTC + state, and it is sent only to a source that has proved reachability, at most + once per returned cookie. +- **Mixed deployments behave at least as well as today:** + - New client, old node: at transport capacity the old node stays silent, and + the client times out and applies the 30-minute cooldown exactly as before. + Application-level closes behave the same on old and new nodes, so they get + the 15 s cooldown either way. + - Old client, new node: the browser fails the association at once, but the + old client still waits for its channel-open timeout and records a 30-minute + failure, as before. + - Section 5 needs only the client. It applies to every refusal the client + classifies, including application-level closes from old nodes. + +### Negative / Trade-offs + +- **A random holder may not have the record yet,** for example a node that + just joined the close group and has not finished replicating. That costs one + `NotFound` round trip, and possibly a dial, before the next holder. On the + testnet, `NotFound` answers per chunk rose from 1.05 to 1.39 with spreading + and the new limits, and GETs per chunk from 3.37 to 3.45. +- **Spreading changes which node a client reads each record from.** It does + not change how many it reads from: one per record, plus the hedge. + Connection reuse may change slightly, because the nearest holder is no + longer favoured. +- **Transport refusals are recognised by timing.** Browser JavaScript cannot + read the STUN error code. A fast ICE failure for another reason, such as no + usable local candidate for the endpoint's address family or a client that + just went offline, is also classed `Refused` and gets 15 s instead of 30 + minutes. Those dials fail at once, so the extra retries are cheap. +- **A browser replacing its own association can hit its source quota.** Its + stale association still holds a slot until ICE closes it or the 10 s pending + expiry runs. Before, a silent listener could admit the retry once that slot + freed, if that happened before the client's 10 s timeout. Now the retry is + refused at once and the client tries again after 15 s. +- **Only the request that completes admission is refused.** A retransmission + of it that arrives later meets a fresh reachability challenge rather than + the same error. Chromium and WebKit failed the dial on the first refusal, + and the client closes the peer connection as soon as it is reported dead, + so later challenges reach a closed connection. +- **A node or network that drops mid-handshake** after DTLS is also classed + `Refused`. That is acceptable: the node was demonstrably up seconds earlier. +- **Application refusals still cost the node a DTLS handshake** per attempt, + as today. Only transport refusals are cheaper. +- **A saturated node holds more per-session state.** Each association's ICE, + DTLS and SCTP state and tasks sit outside the byte budgets. The testnet + measured about 0.5 MiB per session, so a full listener costs about 130 MiB + more than an idle one. That matters to operators who run many nodes on one + host. +- **A read whose holders keep refusing takes up to 45 s longer to fail** + (section 5), and the application must still retry later or show the "busy" + error. The wait cannot tell a full holder from one that has lost the + record, so a read may wait for a refused peer that turns out not to have it. +- **Wherever holders stay full, reads still fail.** Section 5 changes how they + fail, not whether. On the testnet, before section 5, every failed download + was a busy holder reported as a false "record not found": + - at the old limits, 34–39% of synchronised downloads failed this way, with + today's client and with sections 1 and 3 alike; + - at the adopted limits, none of 1,152 downloads failed. With 256 channels + and 128 request slots, which filled before sessions did at 192 browsers, + 6 of 768 failed this way. +- **None of these changes adds capacity for finding 2.** Synchronised demand + for one chunk still outruns seven uplinks; more sessions only queue more + viewers on the same uplink. Spreading and fast refusal make the best use of + the holders that exist. More holders need the deferred options. + +### Neutral / Operational + +- The new defaults reach nodes on upgrade. Raising the transport's 256 + ceiling is a separate decision, based on the per-session cost measured in + Validation. +- `rejected_connections` now counts answered refusals. An operator who sees + it rise knows browsers are being turned away, and no longer silently dropped. +- Once ADR-0020 (get-or-closer) lands, its holder choice should follow the + same rule: spread across the known close group, then nearest first. + +## Validation + +**Unit and regression tests.** +- saorsa-transport, `webrtc::direct::admission`: + - A refusal echoes the request's transaction, passes the listener + credential's integrity check and fingerprint, and is no larger than the + request. + - An unparseable request is never answered. +- saorsa-transport, `webrtc::direct`: + - A listener at capacity and one at its source quota both answer a proven + check with 486 and their own reason phrase. In both cases + `rejected_connections` increments and no association is admitted. +- ant-client, `client_engine::read`: + - Over 7,000 trials, each close-group holder serves about 1/7 of first reads + (within ±300), and peers beyond the close group keep nearest-first order. + - Early reads choose within the known close group, then fall back to + nearest-first. + - An out-of-range choice stays inside the close group. +- ant-client, `EndpointFailureCache`: a refusal expires after its shorter + cooldown while a failure does not, and only unexpired refusals are listed + with their remaining time. +- ant-client, `client_engine::read` (section 5), on a simulated clock: + - A read whose holder is refused and hidden from discovery waits out the + 15 s refusal and reads the record. + - A refusal that ends during a round is retried after the usual 1 s. + - A holder that keeps refusing makes the read fail as busy after three + waits of 15 s, not as absent. + - A refused peer with a close group of nearer `NotFound` answers does not + delay absence. +- ant-client, `wasm-tests/capacity-refusal.test.mjs`: + - A mock node that refuses at its transport, and one that closes during the + handshake, both fail the dial in under 2 s. + - Both are suppressed inside the cooldown, both are dialled again after it, + and the client connects. + - A third mock node that fails DTLS after ICE connected is not dialled again. + - With refusal detection disabled, or with the earlier "dead and fast" test + that ignored ICE state, this test fails. +- ant-client, `wasm-tests/refused-holder.test.mjs` (section 5): a client + whose seed points at a single holder that declines its first dial: + - downloads the file after about 15 s; + - when the holder declines every dial, fails after about 45 s with an error + that says "busy" and not "not found"; + - without section 5, fails with "record … not found", the error the + testnet's failed downloads ended with. + +**Real browsers (2026-10-12).** +- **Setup:** a local 7-node devnet with `max_connections` = 2 and + `max_connections_per_ip` = 1. One `BrowserNetworkClient` holds node 0's + per-source slot, a second client connects to the same seed, and the first + client leaves after the second is refused. This exercises the source-quota + refusal. A capacity refusal sends the same error code, so it is handled the + same way. + + | Browser | Refused after (baseline → this ADR) | Error shown | After 16 s, holder gone | + |---|---|---|---| + | Chromium | 10,011 ms → **67 ms** | "DataChannel opening timed out" → "peer connection failed" | still suppressed → **connects in 132 ms** | + | WebKit | 10,005 ms → **54 ms** | same | still suppressed → **connects in 115 ms** | + +- **Firefox:** Playwright Firefox does not establish WebRTC Direct sessions + with local nodes at all, even with loopback ICE enabled. This is independent + of this ADR, so the refusal path is unverified there. Its failed dials were + reported after 5.0 s, outside the 3 s window, and were correctly kept at the + 30-minute cooldown. + +**On a network (2026-10-13).** +- **Setup:** + - A 60-node testnet across 9 regions. 18 nodes sit behind symmetric NAT, + where browsers cannot reach them. + - 64 browser droplets in lon1, nyc1, sfo3 and sgp1, each running one + headless Chromium on its own IP. + - In each round every browser downloads the same 50 MiB private file + (16 chunks) with a fresh client, all starting within the same 10 s. + Three files rotate between rounds. Half the browsers run a traced build. + - Arms ran in blocks of three rounds, in the order A B C C B A. Each block + follows a restart of every node. + - Nodes logged their listener counters every 5 s. This logging was in the + test build only. + - Scripts and raw data: `ant-testnet/state/comparisons/v2-1508-hot-reads-ab-1013`. + + | 64 browsers | A: today | C: new code, old limits | B: new code, 256 sessions | + |---|--:|--:|--:| + | Downloads completed | 252/384 (66%) | 236/384 (61%) | 384/384 | + | Most popular holder's share of a chunk's first reads | 48% | 18% | 21% | + | Channels a full node closed (traced browsers) | 2,477 | 1,953 | 0 | + | Silent transport drops / refusals per round | 1,413 | 238 | 0 | + | Channel / request rejections per round | 842 / 108 | 714 / 64 | 0 / 10 | + | Node peak memory / CPU | 184 MiB / 1.25 cores | 177 MiB / 1.04 | 203 MiB / 1.19 | + + B ran with 256 channels and 128 request slots, below the adopted 512. +- **Scale:** 192 browsers (three per droplet), with arms in the order B D D B + and two rounds per block: + + | 192 browsers, 256 sessions | 256 channels / 128 requests | 512 / 512 (adopted) | + |---|--:|--:| + | Downloads completed | 762/768 | 768/768 | + | Peak sessions / channels / requests in use | 194 / 256 / 128 | 192 / 362 / 284 | + | Channel / request rejections per round | 1,583 / 707 | 0 / 0 | + | Node peak memory / CPU | 271 MiB / 1.22 cores | 268 MiB / 1.28 cores | + +- **What the run showed:** + - Every failed download ended in a false "record not found". Two browsers + reading the same files on their own finished in 19 s and 54 s. Section 5 + addresses this. It came after this run and has not run on a network. + - At the old limits, channel slots ran out before sessions. Today's client + and the new one failed at the same rate. + - Spreading cut the most popular holder's share of a chunk's first reads + from 48% to 18–21%. Per-node bytes barely changed: the most-loaded node + sent 5–7% of a round's bytes in every arm, because a 16-chunk file + already spans many close groups. + - Fast transport refusal did little at this scale. Full nodes mostly turned + browsers away at the channel limit, after DTLS, where the client already + saw a fast close. + - Dials that timed out after 10 s went to nodes behind NAT: 1,951 of 1,952 + in A and all 3,302 in B. Those nodes advertise WebRTC endpoints that + browsers cannot reach, the testnet counterpart of the mainnet endpoints + that "time out when dialled". + - A restart of every node penalises the next round. Arm A's first round, + 90 s after a restart, completed 23 of 64. Every later block waited 240 s. + Without that round, A completed 229 of 320 (72%). + - Unrelated to this ADR, in every arm: under load, a seed node occasionally + returned an ML-DSA-65 handshake signature that the browser rejected + (102 of 148,755 traced connects). + +**Before Accepted.** +- Check mainnet nodes' `rejected_connections` to learn how many of the + endpoints that "time out when dialled" were full. The node binary does not + log or serve its listener diagnostics today, so this needs them exported + first. +- Run section 5 on a network. Re-run the old-limits arm (C) with it, and + check that downloads which failed with "record not found" now complete or + report busy. + +**Review triggers.** +- A browser changes how it handles STUN error responses to connectivity + checks. +- ADR-0020 changes the read path. +- A hot-record caching ADR lands. + +## Notes for AI-assisted work + +AI tools may help draft this ADR, but **must not mark it Accepted without human review**. Accepted ADRs are immutable: create a new superseding ADR rather than editing an Accepted ADR. diff --git a/docs/adr/README.md b/docs/adr/README.md index 4d8dd0f9..0e493cab 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -39,3 +39,4 @@ See [`TOOLING.md`](./TOOLING.md) for `adrs`, `adr-kit`, and AI harness setup. - [ADR-0017: Bounded fresh-replication offers and copy-free message sends](./ADR-0017-bounded-fresh-offers-and-copy-free-sends.md) - [ADR-0019: Pointer audits serve the record round 1 bound](./ADR-0019-pointer-audits-serve-the-record-round-one-bound.md) - [ADR-0022: Remove the LMDB Chunk Store](./ADR-0022-remove-the-lmdb-chunk-store.md) +- [ADR-0023: Spread hot-record reads across the close group, and refuse full browser sessions explicitly](./ADR-0023-spread-hot-reads-and-refuse-full-browser-sessions.md) diff --git a/src/config.rs b/src/config.rs index 21058f52..51b421da 100644 --- a/src/config.rs +++ b/src/config.rs @@ -243,7 +243,7 @@ fn default_webrtc_direct_bind() -> SocketAddr { } const fn default_webrtc_direct_max_connections() -> usize { - 32 + 256 } const fn default_webrtc_direct_max_connections_per_ip() -> usize { @@ -255,11 +255,11 @@ const fn default_webrtc_direct_max_channels_per_connection() -> usize { } const fn default_webrtc_direct_max_channels() -> usize { - 32 + 512 } const fn default_webrtc_direct_max_concurrent_requests() -> usize { - 16 + 512 } const fn default_webrtc_direct_max_requests_per_second() -> usize {