feat: add DataMapIndex for 64-bit random access

[mickvandijke]

Summary

Adds DataMapIndex, random access to a DataMap's content with u64 positions on every platform, and builds DecryptionStream on it.

The browser client (wasm32) reads files larger than 4 GiB by fetching chunks asynchronously and decrypting them one at a time. DecryptionStream's range API couldn't serve that: its offsets are usize sums, which overflow on 32-bit targets, and its fetch callback is synchronous. The client therefore had to rebuild the chunk index and call decrypt_chunk itself, with a hard-coded KDF level and its own copy of the source-hash ordering. With this change, self_encryption owns that contract again.

• DataMapIndex (new, no I/O): new(&DataMap) checks that the chunk indices are exactly 0..n and that the size fits in u64. It provides file_size() -> u64, infos() in index order, chunk_range(i) -> Range<u64>, chunks_overlapping(Range<u64>) -> Range<usize> (binary search), and decrypt_chunk(i, &content). decrypt_chunk takes the KDF level from the map and checks the result against the declared src_size and src_hash. Decompression stops one byte past the declared size, so a crafted chunk cannot expand beyond what the map states.
• DecryptionStream now holds a DataMapIndex instead of its own chunk_infos/src_hashes/child_level. get_range uses u64 arithmetic and binary search instead of usize sums and linear scans. Public signatures are unchanged.
• Fix: DecryptionStream now fails when the fetch callback omits a chunk. Before, get_range skipped the missing chunk and joined the rest, which returned wrong bytes, and iteration paired fetched chunks with the batch by position, so it ended early as if the content were complete. Both paths now share one batch fetch that looks chunks up by index.
• decrypt_chunk decompresses through brotli::Decompressor, which the bounded variant shares. Output is unchanged.

Compatibility

Additive, so the version is bumped to 0.36.2: existing 0.36 users (ant-node) pick it up without a major bump. Behaviour changes, all for maps that encrypt never produces:

• DecryptionStream::new* fails up front for maps whose chunk indices are not 0..n. Before, such maps failed or misbehaved during decryption.
• Every chunk DecryptionStream decrypts is checked against its declared size and BLAKE3 source hash. Every map 0.35+ can decrypt uses content_hash source hashes, so honest maps are unaffected.
• On 32-bit targets, DecryptionStream::file_size saturates at usize::MAX instead of overflowing. DataMapIndex::file_size gives the exact size.

Verification

• cargo test --release: all passing. New unit tests cover reassembly, half-open overlap ranges, offsets past 4 GiB, rejected index layouts, declared-size over- and under-runs, a crafted map with a wrong source hash, and out-of-range indices. The old test_chunk_boundary_underflow_reproduction read private DecryptionStream fields, so it was ported to the index. test_omitted_chunk_fails_instead_of_truncating covers a callback that succeeds without one chunk; on e8aa39ee its iteration check fails.
• cargo clippy --all-targets --all-features -- -D warnings and cargo fmt --check are clean; cargo check --workspace (Node.js bindings) builds.
• Criterion read_* benchmarks are unchanged between master and this branch (read_10_megabytes 21.05 ms vs 21.01 ms, read_100_megabytes 124.1 ms vs 123.9 ms).
• Used by feat(wasm)!: support large-file downloads and disk streaming ant-client#214 for its browser range reads, pipeTo and complete downloads. Its 819 native library tests and 221 generated-WASM tests pass against this branch.

🤖 Generated with Claude Code

[dirvine]

Hermes paired review — provisional, not approval

Reviewed #447 at e8aa39ee1337c11f52628356d97fa86ac0d1007d with WithAutonomi/ant-client#214 at 773ea5dac9b7c68efcf1f9c0a73a298e9fa46d3c.

Code verdict: No new blocking defect found in the inspected changes. The index keeps file positions in u64, validates contiguous chunk indices, derives keys from the map's child level, and verifies plaintext size and source hash. The bounded Brotli reader stops at the declared size plus one byte. The range reader now rejects a missing fetched chunk instead of joining an incomplete result.

Checks I ran: cargo test --release passed: 97 tests including integration and documentation tests; 1 ignored; 0 failed. cargo fmt --all -- --check passed. The paired ant-client also passed 819 native ant-core library tests and 221 generated-WASM tests against this exact Git revision. GitHub reported all 17 checks successful on this head. I did not independently run the native 32-bit or Node.js suites; those results are from CI.

Review-team participation: 1/3 completed. Qwen 3.8 (qwen38-27b) completed an independent paired review and found no new blocker. GLM 5.3 (huggingface, zai-org/GLM-5.3) failed with HTTP 402/no remaining credits. DS4 (deepseek-v4-flash) timed out. An earlier oversized local panel was stopped without completed local opinions. No missing member has been replaced or counted as agreement. This is not full consensus, so this review does not approve or authorise merge.

Paired release gate: ant-client#214 still requires this version, self_encryption 0.36.2, to be published and its temporary [patch.crates-io] entry removed. The crates.io version endpoint returned HTTP 404 during this review. Publication is a separate human-authorised action.

[dirvine]

Clean re-review — provisional, not an approval

Reviewed head e8aa39ee1337c11f52628356d97fa86ac0d1007d against master / merge base eb95108a6eb8f1012ff2499c1cbb830aed8392a8 in a detached, clean worktree. This replaces the earlier assessment with a fresh source review and fresh test execution.

Source verdict: no blocking defect introduced by this diff was confirmed. This is a COMMENT, not an APPROVE. The prescribed review panel is incomplete.

Actual review participation

• GLM 5.3, OpenRouter z-ai/glm-5.3: independent correctness and adversarial reviews completed.
• Local Qwen 3.8, qwen38-27b: independent correctness and adversarial reviews completed.
• Local DS4, deepseek-v4-flash: both roles exceeded their deadlines. Both bounded retries also timed out. No DS4 review is counted.

Participation is 2/3 models, 4/6 independent seats. Reconciliation answers are not extra votes. There is no full consensus. After reconciliation, GLM and Qwen support the source changes, subject to the integration conditions and the cautions described below.

Findings adjudicated against the baseline

1. KDF compatibility claim: not introduced here. src/utils.rs has the same Git blob on the merge base and head. The first-pass claim that this PR changed the KDF context was therefore rejected. An unchanged KDF does not, by itself, establish compatibility with every deployed network version.
2. Sequential fetch silently accepts missing callback results: real, but inherited. I executed the same standalone Rust probe against the base and head. Each encrypted 6,000 bytes into a three-chunk map, then supplied an empty successful fetch result. Both returned zero output bytes and zero errors. This needs a separate fix and regression test; it is not a regression caused by #447.
3. Unbounded decompression in the legacy full-decrypt path: inherited. The baseline already has it. The new indexed path uses bounded decryption and rejects output beyond the declared size; this concern does not establish a new defect in that path.
4. Reversed range_inclusive yielding one byte: inherited. The helper is in the unchanged src/utils.rs blob. Treat it as a separate hardening issue, not as a blocker introduced by this diff.

Verification I ran

• cargo test --release: 97 passed, 0 failed, 1 ignored, including integration and doc tests.
• cargo fmt --check: passed.
• cargo clippy --all-targets --all-features -- -D warnings: passed with PYO3_PYTHON=/Users/studio1/.local/bin/python3.13. The initial attempt selected an unsupported local Python version; the supported-Python rerun succeeded.
• Base/head missing-fetch probe: same inherited failure reproduced on both revisions.
• Latest head check: unchanged; 17 CI checks succeeded. GitHub still reports REVIEW_REQUIRED and merge state BLOCKED.

Integration and evidence limits

The paired client PR needs the released self_encryption 0.36.2. The crates.io version endpoint still returns 404. The normal order is to land/release this library change, then remove ant-client #214's temporary Git [patch.crates-io] override and update its lockfile against that published release. Publication is not a claim that #447 itself needs that client-side patch removed.

I did not run a production-network compatibility test or a real-browser/live-node download-to-disk test above 4 GiB. Those remain separate evidence gaps. No merge or release action was taken. This is Hermes's provisional technical assessment, posted through the authenticated GitHub account, not a human sign-off.

[dirvine]

Hermes-assisted current-head review: APPROVE

Reviewed 869b2f9 against eb95108. This supersedes my provisional reviews of the previous head.

No blocking defect found in this library change. GLM 5.3 (z-ai/glm-5.3), local Qwen 3.8 (qwen38-27b), and local DS4 Flash (deepseek-v4-flash) each completed independent correctness and adversarial reviews: 3/3 models, 6/6 roles, all APPROVE. Each final receipt has the exact head, actual requested model/provider, normal termination, and the same evidence SHA-256 f13528d40338a4533553b57312b626d4f81f234c12cbfd90efe39e5620cd8b96. No unresolved material dissent remains. Earlier failed/truncated attempts are not votes.

Verified evidence

• All 17 GitHub checks pass, including 32-bit tests, cargo-deny, and dry-run publication. The head and check results were re-read before submission.
• Local cargo test --release --locked: 98 passed, 0 failed, 1 ignored. Format, Clippy with warnings denied, and workspace check pass.
• DataMapIndex validates contiguous indices, checks cumulative u64 size, and uses ordered offsets for range lookup. Indexed chunk decryption checks the declared plaintext size and source hash.
• The missing-fetch regression is fixed at this head. fetch_and_decrypt errors when a requested chunk is absent. Both range reads and iteration use it; test_omitted_chunk_fails_instead_of_truncating passes.
• The KDF source is unchanged. An executed native probe against published crates.io 0.36.1, not just the PR base, passes five deterministic fixtures: 3, 3,072, 6,000, 4,190,209, and 12,570,641 bytes. It verifies published-to-current full and indexed decryption, plus current-to-published full decryption. This is compatibility evidence for those fixtures, not an exhaustive corpus test.

Limits and release hand-off

• A real-browser/live-node download above 4 GiB was not run. Native offset tests, 32-bit CI, and the 12.6 MB compatibility fixture do not prove that end-to-end Client/SDK path.
• Decompression is bounded by the map-declared chunk size, with an extra byte to detect overrun. It is not an application-wide resource quota. Legacy full decryption remains unbounded; range reads still materialise overlapping chunks. Fetch batch size is not an output-memory cap. chunk_range retains its documented panic contract for invalid indices.
• These limits are not new blocking library defects. This approval does not approve ant-client #214 or waive its dependency, security, or browser-integration gates.
• self_encryption 0.36.2 is not yet published on crates.io (version endpoint returned 404). The maintainer can proceed with the normal human merge/release process. Client/SDK work must then use the published crate and regenerate its lockfile rather than retain the temporary Git patch.

No merge, publication, or deployment was performed by this review.