[grumbach]

Requesters now keep their own in-flight fetches to each answering holder within that holder's admission share. This removes the ordinary over-share bursts that the holder dropped unanswered, each of which cost the requester a 30 s timeout and charged the holder a trust penalty it had not earned.

The problem. A fetch responder admits two outstanding requests per requesting peer and drops later arrivals without a reply while that share is full. The fetch worker did not respect that:

• it ran one fetch per hardware thread;
• it sent each one to the first verified holder of its key;
• the nearest-first queue often groups keys that share holders.

When more than two fetch slots were available and several queued keys shared a holder, the worker could send more than that holder's share. Each resulting drop:

• cost the requester the full 30 s fetch timeout;
• recorded a connection-timeout trust event against a healthy holder;
• led to a retry that again ignored how busy the next holder was.

On the production fleet, the responders' own cumulative counters showed:

period	inbound fetch requests dropped at admission
0.21.0	36.6%
0.22.0	44.3%
25-hour window containing a wave of 177 freshly installed nodes filling their ranges (2026-10-11 20:00 to 2026-10-12 21:00 UTC)	65.8% overall by cumulative counters (3.37M); WARN lines recorded 3.38M drops, 3.22M from the new nodes, about 2.8 per chunk attributed to them

In that window, 68.5% of drops happened with only 2 of the responder's 12 admission slots in use, both held by the refused peer.

The change (node only; no wire, storage or production public API change). The worker keeps at most two fetches in flight to any one answering source, the same share the responder grants.

• Each free slot goes to the nearest queued key that can start now, on the first answering source with room in its source-preference order.
• A key whose untried sources are all answering and full is parked, indexed under each of them, and starts again as soon as one frees a share. Blocked keys never hide a startable key behind them.
• A failed fetch whose remaining sources are all answering and full is parked rather than sent to a full holder. It keeps its tried sources and its verification retry reservation. With no untried source left, or a full fetch queue, it returns to verification as before.
• The sources a queued or in-flight key is associated with, tried ones included, are capped at close_group_size + paid_list_close_group_size + MAX_HINT_SOURCES_PER_KEY from the running configuration. This never truncates a verification round's own sources, and it bounds the new index. A re-advertisement of such a key merges into its verification retry entry under MAX_HINT_SOURCES_PER_KEY, as into a pending entry, instead of without a bound.

Holders that stop answering. A holder that answers nothing has no share to keep within. Holding it to two requests per timeout would drain the keys only it holds four times more slowly than before, which a local simulation confirmed (40.1 s against 10.2 s). So a source counts as silent while a request sent to it after its latest answer has gone without any response. Requests and answers are ordered by one process-wide sequence, so the verdict does not depend on the order outcomes arrive in. An answer reaches the scheduler as soon as it arrives, before its chunk is checked and stored, so a timeout it outdates cannot mark its source while that chunk is still being written. A silent source is fetched from past the limit, as before this change, until it answers again, or until two fetch timeouts pass with no newer unanswered request to it, after which it is probed under the limit. The cost is one fetch timeout to notice the silence.

Observability. A replication fetch dispatch summary line every 5 minutes and at shutdown reports the parked backlog and its oldest age, sources at the limit, attempts by outcome, stored bytes, attempt latency, lock wait and hold times, and the silent-source counts.

What it does not do:

• It cannot rule out every refusal: a request this node has already timed out on may still hold its share at the responder until a worker sheds it. That needs a "busy" reply, which is a wire change and is left for later.
• It does not reduce the bandwidth a joining node needs to fill its range. Fills can finish faster, so a filling node's instantaneous load can rise; each responder still serves at most its share per requester.
• Older requesters keep overshooting until they upgrade.
• There is no runtime switch. Undoing it means a release without it.

Linear issue

Closes V2-1514

Risk tier

• T0: docs / tooling / CI / pure UX-output. Repo CI only.
• T1: client-only, no network-facing behavior change. CI + prod compat smoke.
• T2: node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3: protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

• Wire: none. Requests are sent less eagerly to answering holders, and every message is unchanged.
• Storage: none.
• API: none for production builds (the new scheduling items are pub(crate)); public ReplicationQueues fetch methods keep their signatures and now apply the per-source scheduling and source limit. Additive and test-only, compiled under cfg(test) or the test-utils feature: ReplicationEngine::{set_fetch_concurrency_for_test, close_fetch_serve_gate_for_test, open_fetch_serve_gate_for_test, fetch_pipeline_snapshot_for_test}, replication::FetchPipelineSnapshot and ReplicationQueues::fetch_pipeline_snapshot.

Semver impact

• breaking
• feature
• fix

Test evidence

Rebased onto main 2598bcf4 (after #250, ADR-0023), with the change itself unchanged. The only conflict was in the ADR index, where ADR-0023 and this change's ADR-0024 each add a line after ADR-0022; both lines are kept. The rebase before that, onto b0e29c32 (after #256, which moves zip to 9 and rustls to 0.23.46), had no conflicts. The one onto a6e3388e (after #239 and the 0.23.0 release) had one conflict, a textual one in tests/e2e/testnet.rs: the harness now sets the fetch concurrency override and then wires pointers with #239's with_pointer_service. The tests, gates, simulation and benchmark below were re-run on this head; the mutation checks and the "before" builds are from earlier heads.

Local risk simulation (12-node local network, 8 fetch slots, 5 s fetch timeout). "Before" is a build with the limit removed and no silent-source handling, which schedules exactly as before this change.

scenario	this branch	before
fill: 48 keys held by 3 answering holders	0.36 s	10.08 s
a non-answering holder first among 3 sources, 2 answering: 24 keys	5.1 s	20.1 s
a non-answering sole holder: 16 keys	15.0 s	10.15 s

The last row is the documented one-timeout penalty: with C ≥ 2 fetch slots, T × (1 + ceil((N − 2) / C)) against T × ceil(N / C). These are the figures recorded in ADR-0024. Re-run on this head while other builds shared the machine, the three scenarios took 0.37 s, 5.1 s and 15.2 s.

Unit tests: cargo test --features test-utils --lib passes: 1,237 tests, 48 of them new (one is the opt-in scale benchmark), in scheduling.rs and mod.rs. They cover:

• the share per answering source and preference order;
• a startable key behind 300 blocked keys starting in the same pass;
• the nearest parked key winning whatever its source, including through a ninth source;
• retry parking with tried sources and per-owner reservations kept across park and restart;
• the queue-full fallback to verification;
• tried sources never merged back;
• departures re-queueing or purging parked keys;
• the source limit across park, retry and merge;
• dequeue_fetch returning the global nearest key;
• bootstrap drain held open by a parked key;
• silent sources: fetched from past the share, an answering source with room preferred, parked keys starting when a full source falls silent, the verdict the same in every order of outcomes, marks set, refreshed by newer failures, cleared and expired, a late-learned failure still marking for a whole lifetime, the table bound keeping every mark and in-flight guard, and a sole silent holder's sixteen keys starting in exactly three waves of 2, 8 and 6;
• an answer reported when it arrives keeping a source within its share when an earlier request to it times out while the answered fetch is still storing its chunk (known only from the outcome, the same answer comes too late and the source is sent every waiting key);
• re-advertisements keeping a queued or in-flight key's retry entry within the hint bound and with its capacity owner, also after it returns to verification;
• a 4,000-step randomised run of queue operations, re-advertisements of keys that carry retry entries, and source contacts, checking the queue, index, mark and retry-entry invariants after every step;
• the dispatch summary timing each attempt to when its task finished, and counting cancelled attempts separately.

New e2e tests (tests/e2e/fetch_per_source_share.rs):

• 12 keys directed to one holder whose admitted requests are held by a test-only gate: the requester recorded at most 2 in-flight fetches to it. With the per-source limit removed, it recorded 8 and the test failed.
• 16 keys directed to a holder that answers nothing, 5 s fetch timeout: the requester went to 8 in flight once the first requests timed out and drained the keys in 15.1 s (limit 20 s, four timeouts). With silent-source handling removed, the test failed.
• Both tear the network down with the holder's gate still closed and requests held at it. Engine shutdown releases them; without that release, teardown hit its 120 s limit and the test failed.

Mutation checks: removing silent-source handling fails 16 unit tests and the e2e test. Ignoring silence when waking parked keys, marking a source despite a later answer, clearing a mark on any answer, not refreshing a mark on newer failures, or re-marking on older failures each fail at least one unit test. Ignoring answers reported before their fetch ends fails the early-answer test, an unbounded retry-entry merge fails the hint-bound test, and not releasing the serve gate at shutdown fails the e2e teardown.

CI test suites, run locally on this head (each CI test command, with CI's RUSTFLAGS="-D warnings"):

• cargo test --lib --features test-utils: 1,237 pass;
• cargo test --lib --no-default-features: 1,192 pass;
• --test e2e --features test-utils -- --test-threads=1: 121 pass, 3 ignored (main's live_testnet tests, which this change does not touch);
• chunk_store_crash_safety 2 (its 1 ignored test is the crash test's child process), storage_scale 2, webrtc_direct_devnet 3, its ignored five-node devnet test 1, poc_commitment_audit_attacks 19, poc_audit_handler_live 16, poc_bootstrap_stall 3: all pass;
• the shipped-binary failpoint check: no failpoint in a default-feature build.

Outside CI's list, poc_d1_bounded_queues (7) and pointer_convergence (17) pass. poc_price_floor_live fails enforced_floor_rejects_cheapest_of_k_real_settlement in exactly the same way on main 2598bcf4; this change touches no payment code.

Scale benchmark at the fetch queue's hard bound (131,072 parked keys, release build): one park pass 89 to 99 ms / 207 to 216 ms / 754 to 960 ms over repeated runs, and peak RSS 197 MiB / 342 MiB / 1.13 GiB, at 1 / 8 / 35 sources per key, within the thresholds declared before it ran. The exception is dropping the queues at 35 sources per key: 274 to 315 ms over repeated runs, against the 250 ms first declared. The scheduler before the silent-source handling took 226 to 295 ms, and the same time in alternating runs on one machine. It happens once at shutdown, outside any lock, so its limit is now 1 s. Re-run on this head, once through cargo and twice as the bare test binary for peak RSS: one park pass 74 to 89 ms / 181 to 193 ms / 652 to 676 ms, peak RSS 189 to 195 MiB / 342 to 348 MiB / 1.13 GiB, departures at most 89 ms, and dropping the queues at 35 sources per key 211 to 220 ms, all within the thresholds.

Gates:

• cargo clippy --all-targets --all-features -- -D warnings on stable and on 1.99.0, and cargo clippy -- -D warnings;
• cargo fmt --check;
• cargo doc and cargo doc --all-features with -D warnings;
• cargo +1.95.0 check --all-targets --all-features --locked (MSRV);
• RUSTFLAGS="-D warnings" cargo build --release --no-default-features and cargo check --lib --no-default-features --locked;
• scripts/adr-governance.py;
• cargo audit: reports two advisories added to the RustSec database after the previous head was audited: rustls-webpki 0.103.15 (RUSTSEC-2026-0382) and webpki-root-certs 1.0.9 (RUSTSEC-2026-0383). This change does not touch Cargo.lock, which is identical to main's, so main fails the same way; chore(deps): move rustls-webpki to 0.103.16 and webpki-root-certs to 1.0.10 for two RustSec advisories #258 moves both crates to fixed versions.

Dev testnet: not run yet. T2 needs it before merge, ideally with a batch of fresh nodes joining a populated network, comparing drain time, stored bytes, fetch admission drops, trust events and host load against the release.

Release: this should be tested on a dev testnet before it ships, so it is not proposed for the next release unless Chris decides otherwise.

New dependency

none

ADR

https://github.com/grumbach/ant-node/blob/fix/fetch-per-holder-cap/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md

Mitigation / rollback

Revert the commit. The change is confined to requester-side fetch scheduling, with no stored state and no wire change, so a node on either side of it interoperates with every other.

[dirvine]

Provisional review — no severe introduced defect confirmed

Reviewed WithAutonomi/ant-node at e1c7b2e6fbc990037df1a19fb905a019ab6cc832 against merge base f8008ef4e75f4b95a9cbb18442af0718a20171f5. This is a review comment, not an approval or merge/release sign-off.

Review participation

2/3 models reviewed; 4/6 independent role reviews completed. GLM 5.3 through OpenRouter and local Qwen 3.8 completed separate correctness and adversarial reviews. Their returned model identifiers matched the requested models and all four bounded responses ended with finish_reason=stop. Both found no severe introduced defect in the evidence supplied. Local DS4 Flash did not complete: the original native run exhausted its overall deadline; separate bounded replacement requests for both scopes timed out. DS4 endpoint discovery worked, but endpoint availability is not a completed review. No fallback was counted in its place. There is no full-team consensus; formal approval is withheld.

What the patch does

The cap is two fetches per answering holder, not an unconditional limit of two. The dispatcher and retry path select another startable holder or park the key instead of sending a third request to a holder still regarded as answering. Parked keys remain in the queued payload/capacity budget. Source departure, retries, verified-repair requeue, cancellation and panic handling were checked against the worker lifecycle and local tests.

The silent-holder exception is deliberate in ADR-0024 (WithAutonomi/ant-node). A holder that does not answer can be fetched from above its share, within the existing global worker concurrency limit. Contact ordering, expiry and bounded health-table overflow determine that exception. Actual wire contact is recorded separately from payload validity and local persistence, so a local write failure is not itself evidence of a silent holder.

This fits the reported V2-1514 failure mechanism: answering holders drop the third concurrent request without a reply, which turns admission shedding into timeout work. However, shedding itself can produce silence, and health-table overflow treats untracked holders as silent. This is a mitigation, not proof that all admission drops or retries disappear. It also changes upgraded requesters; it cannot retroactively throttle the old external 0.22.0 cohort. The cohort figures in the thread are author-reported evidence, not measurements freshly reproduced during this review.

Executed verification

• cargo test --locked --features test-utils --lib replication::: 673 passed, 0 failed.
• cargo test --locked --features test-utils --lib: passed, exit 0.
• cargo test --locked --features test-utils --test e2e fetch_per_source_share:: -- --nocapture: both new tests passed.
• cargo fmt --all -- --check: passed.
• scripts/adr-governance.py: passed, 20 ADR files checked.

The new e2e tests cover an answering holder staying within its share and a silent holder draining through the exception. They do not replace a mixed-holder join-wave test or independently prove production admission-drop rates. No source edits, branch pushes, staging provisioning or live-network writes were made.

Remaining gates and watch-outs

1. T2 dev/staging testnet is unchecked in the PR body and was not run here. Before release, supply the ADR's join-wave evidence. Check admission drops, completed fetches/bytes, bootstrap drain, oldest_parked, source-health overflow, queue-lock wait/hold and memory. The documented worst-case parked index is large; a bounded allocation is not automatically a suitable fleet allocation.
2. Security Audit is red. I independently ran cargo-audit against head and base lockfiles using the same advisory database. Both have byte-identical lockfiles and fail with the same quinn-proto 0.11.17 advisories: RUSTSEC-2026-0343, RUSTSEC-2026-0338, RUSTSEC-2026-0340. This is an existing dependency problem, not introduced by this patch; it still needs tracking. The other exact-head GitHub checks passed.
3. DS4's two review scopes remain missing. Complete those before treating this as a full-panel approval.

I rejected unsupported reviewer hypotheses rather than copying them into findings. In particular, the proposed stale FuturesUnordered concurrency count is not a reachable oversubscription case: pushing the retry future updates the collection before the next fill pass. Reviewer-invented split filenames were not accepted as source citations. Table-overflow bypass is an explicit design trade-off, not a newly discovered universal cap violation. No responding reviewer supplied a confirmed severe blocker; that does not turn the missing reviewer into agreement.

[dirvine]

REQUEST_CHANGES — T2 testnet evidence is still missing

Reviewed WithAutonomi/ant-node at head 31f028a2b84013841504783b32ae750842f9c83d, against base a6e3388e79d0907ef678bcb6ec0fcaa12822e148.

Blocking request: complete the stated before-merge gate

The PR description still says:

Dev testnet: not run yet. T2 needs it before merge, ideally with a batch of fresh nodes joining a populated network, comparing drain time, stored bytes, fetch admission drops, trust events and host load against the release.

This changes live replication scheduling, including the deliberate cap bypass for a source classified as silent. The local tests check the mechanics; they do not measure the join-wave behaviour or load on a populated network. The requested T2 evidence is therefore still absent. ADR-0024 also calls for this comparison before release (lines 233–237); the stricter before-merge requirement comes from the PR description, not from relabelling the ADR.

Minimum change: attach the approved dev-testnet run and baseline comparison to this PR. Identify the revisions, topology and join wave; provide drain time, stored bytes/completeness, fetch admission drops, trust events and host load. Show that the run completes without a fetch/parking stall or unacceptable load regression. The local e2e tests and scheduler simulation are not substitutes for this run. If the team intends to move this gate to release-only, record the authorised human decision explicitly; I have not waived it or started a testnet.

No severe introduced code defect was confirmed in this re-review. This verdict is for the unmet, explicitly documented gate, not a claim that a missing reviewer is a code defect.

Checks I ran on this exact head

Runtime: macOS 26.4, rustc/cargo 1.96.0. Detached checkout remained clean.

• cargo test --locked --features test-utils --lib: 1,236 passed, none failed.
• cargo test --locked --features test-utils --test e2e fetch_per_source_share:: -- --nocapture --test-threads=1: 2 passed.
  • Answering-holder case: maximum in flight to the holder 2, with global concurrency 8 and 12 keys.
  • Silent-holder case: maximum 8, drained after 15.123578667 s, with a 5 s timeout. This verifies the intentional exception, not a zero-admission-drop guarantee.
• cargo fmt --all -- --check, git diff --check a6e3388e...HEAD, and python3 scripts/adr-governance.py: passed.

The current GitHub checks pass except Security Audit. Cargo.lock is byte-identical to the base revision; local cargo audit --no-fetch --json reports seven existing advisories. This is an inherited dependency/audit issue, not a dependency change introduced here. It remains red; I am not describing CI as fully green. The additional runs listed by the author are not counted as runs I performed.

Review coverage and dissent

Partial panel: 2/3 named models contributed — GLM 5.3 and Qwen 3.8. No full-team consensus. Both correctness/evidence and adversarial roles have substantive responses. The source was split into scheduler and engine scopes: 6/12 original scoped passes completed. Focused GLM/Qwen follow-ups connected the caller/callee seams; those are not extra independent votes.

• GLM 5.3 (z-ai/glm-5.3): substantive engine reviews and a focused adversarial scheduler/engine retry. Full scheduler correctness still failed with incomplete/length termination after retry. Its focused review found no confirmed defect in the supplied seams and explicitly marked the T2 gate CHANGES_REQUIRED.
• Qwen 3.8 (qwen38-27b): completed both scopes and both roles. Its correctness follow-up withdrew the proposed infinite re-parking loop and unbounded silent-source dispatch concerns: nearest_parked filters out blocked sources, and engine global concurrency bounds the deliberate bypass. Its adversarial follow-up also withdrew the proposed lost-wakeup defect: the notified future is enabled before reading the gate flag.
• DS4 (deepseek-v4-flash): no substantive completed review. Scheduler/other attempts and the narrowed retry hit the model-lock queue timeout; the original correctness-engine request exceeded its response deadline. These are missing coverage, not agreement.

Residual dissent: Qwen's adversarial follow-up still returned CHANGES_REQUIRED, but its remaining counter concern was described in its own explanation as a design choice, not a demonstrated fault. I do not accept that as an additional blocker: unreachable counts unanswered attempts, not new marks; silent_peak is a per-report-window peak seeded from the current count. GLM independently confirmed those semantics. I have not converted Qwen's verdict into an approval or counted omitted source in an individual scope as proof of a defect.

The intended limits remain important: silent sources may receive more than two concurrent fetches, subject to global concurrency; older requesters can still overshoot; and the ADR documents substantial worst-case parked-index memory. The missing join-wave run must test the network effect. No merge, deployment or release approval is implied.