perf(lsh): support datasketch 2.0 native MinHash and batch Rayon signatures (ticket-016)

[tom-sapletta-com]

Summary

• Support Datasketch >=2.0 scheme specification (scheme='legacy') and fallback attribute assignment for native Rust MinHash signature instantiation, preventing fall-through to slow pure-python loops.
• Add batch_minhash_signature to the native redup_fast_hash Rust extension leveraging Rayon for multi-threaded parallel computation with released GIL.
• Implement vectorized batch indexing in LSHIndex.add_blocks using native batch signatures.
• Cache pairwise similarities in find_fuzzy_groups to eliminate redundant re-computation of sequence_similarity across clusters.
• All 261 tests pass and governance checks pass.

[ifuri-validator-agent]

Validation rejected:

• Patch contains unsafe validation shortcut: pytest.skip

[ifuri-validator-agent]

Validator approval after policy checks for exact head 3e2dea113b15178522f71b589e62079da75403bf.

Ticket: ticket-016
Correlation ID: local-semcod-redup-pr-23-ticket-016
Model: gemini/gemini-3.8-flash
Advisory LLM review: LLM_UNAVAILABLE
Reason: No module named 'litellm'
This approval rests on the deterministic checks alone; no advisory opinion was recorded for this head.
Semantic review prerequisite: not_required; policy 676cb4516bbfed2a000e40b9b1b6e4a430ecc761ec546aeb53d721a1905cfdd7.

Actual PR impact radar

Exact range: c290460e4fdab34ffe8fa9e9e48092d7fee77939...3e2dea113b15178522f71b589e62079da75403bf
Change digest: b1bd8f20c967d58ff02836bfbe34ce60b35c1933c61b81a7f68aeb997d0fc7bf
Score: 52/100 (M), estimated 58 min, split recommended: true
Affected services/components: packages/redup-fast-hash

Machine-readable radar JSONL and SVG

{"actual_change":{"additions":150,"base_sha":"c290460e4fdab34ffe8fa9e9e48092d7fee77939","binary_files":0,"categories":{"code":3,"configuration":1,"docs":2,"tests":1},"change_digest":"b1bd8f20c967d58ff02836bfbe34ce60b35c1933c61b81a7f68aeb997d0fc7bf","comparison":"c290460e4fdab34ffe8fa9e9e48092d7fee77939...3e2dea113b15178522f71b589e62079da75403bf","deletions":9,"file_count":7,"files":["packages/redup-fast-hash/src/lib.rs","project/TICKETS.md","project/ticket-016/README.md","project/ticket-016/intent.json","src/redup/core/lsh_matcher.py","src/redup/core/pipeline/duplicate_finder.py","tests/test_lsh_matcher.py"],"head_sha":"3e2dea113b15178522f71b589e62079da75403bf","service_count":1,"services":["packages/redup-fast-hash"]},"assessment_mode":"observed-pr","axes":{"coupling":4,"delivery":2,"scope":3,"uncertainty":3,"validation":1},"complexity":"M","confidence":0.9,"diagnostics":["RADAR-ACCEPTANCE-MISSING","RADAR-BUDGET-EXCEEDED"],"estimate":{"budget_minutes":30,"minutes":58,"within_budget":false},"impact":{"components":["packages/redup-fast-hash","project","src/redup","tests"],"files":["packages/redup-fast-hash/src/lib.rs","project/TICKETS.md","project/ticket-016/README.md","project/ticket-016/intent.json","src/redup/core/lsh_matcher.py","src/redup/core/pipeline/duplicate_finder.py","tests/test_lsh_matcher.py"],"public_interfaces":[],"runtime_dependencies":0},"schema":"subactor.ticket-radar/v1","score":52,"split":{"parts":[{"estimated_minutes":12,"name":"Implement packages/redup-fast-hash","scope":["packages/redup-fast-hash"]},{"estimated_minutes":12,"name":"Implement project","scope":["project"]},{"estimated_minutes":12,"name":"Implement src/redup","scope":["src/redup"]},{"estimated_minutes":12,"name":"Implement tests","scope":["tests"]},{"estimated_minutes":15,"name":"Validate and project to trackers","scope":["tests","planfile","github/gitlab/jira projections"]}],"reason":"estimated_minutes_exceed_budget","recommended":true},"standards":[{"id":"wellmanifest/dsl","revision":"6c60fc4e0dd1f1bb74f46a7745e28019908d1203","version":"0.1.0-dev"},{"id":"wellmanifest/ticket-lifecycle","revision":"5bf581907a87b46a13a73e6c033d3abe4d9a306f","version":"0.1.0-dev"},{"id":"wellmanifest/git-lifecycle","revision":"7d77d4b7af57e69bc75c3a0290b3a4805c5c4438","version":"0.2.0-dev"},{"id":"wellmanifest/logs","revision":"48c284ef7a069055c0bcb6b900147ce5e65f8b43","version":"0.3.0"}],"ticket_ref":"ticket-016"}

<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 128 128" role="img"><title>ticket-016: perf(lsh): support datasketch 2.0 native MinHash and batch Rayon signatures (ticket-016)</title><rect width="128" height="128" rx="12" fill="#f8fafc"/><g stroke-width="1"><polygon points="64,55 72,61 69,71 59,71 56,61" fill="none" stroke="#d7dde5"/><polygon points="64,47 80,59 74,78 54,78 48,59" fill="none" stroke="#d7dde5"/><polygon points="64,38 89,56 79,85 49,85 39,56" fill="none" stroke="#d7dde5"/><polygon points="64,30 97,53 84,92 44,92 31,53" fill="none" stroke="#d7dde5"/><polygon points="64,21 105,51 89,99 39,99 23,51" fill="none" stroke="#d7dde5"/><line x1="64" y1="64" x2="64" y2="21" stroke="#aab4c0"/><line x1="64" y1="64" x2="105" y2="51" stroke="#aab4c0"/><line x1="64" y1="64" x2="89" y2="99" stroke="#aab4c0"/><line x1="64" y1="64" x2="39" y2="99" stroke="#aab4c0"/><line x1="64" y1="64" x2="23" y2="51" stroke="#aab4c0"/></g><polygon points="64,38 97,53 79,85 59,71 48,59" fill="#fb923c" fill-opacity="0.45" stroke="#c2410c" stroke-width="2"/><circle cx="64" cy="64" r="3" fill="#c2410c"/><g font-family="sans-serif" font-size="7" fill="#334155"><text x="64" y="11" text-anchor="middle">SCO</text><text x="114" y="48" text-anchor="middle">COU</text><text x="95" y="107" text-anchor="middle">UNC</text><text x="33" y="107" text-anchor="middle">VAL</text><text x="14" y="48" text-anchor="middle">DEL</text></g><text x="64" y="124" text-anchor="middle" font-family="sans-serif" font-size="8" fill="#0f172a">M · 58m</text></svg>

Merge will be attempted after this approval when explicitly authorized. ## Decision record (recomputable)

DECISION D-016-7693
TICKET ticket-016
HEAD_SHA 3e2dea113b15178522f71b589e62079da75403bf
CORRELATION_ID local-semcod-redup-pr-23-ticket-016
ACTOR agent:ifuri-validator-agent[bot]
APPLIED_RULE P-CORE-015
INPUT author_login = "tom-sapletta-com"
INPUT observed_checks = ["governance / enforce=PASS","test (3.10)=PASS","governance / remote lifecycle=PASS","test (3.13)=PASS"]
INPUT required_checks = ["test (3.10)","test (3.13)"]
INPUT required_checks_source = "protected registry (env/request)"
INPUT reviewer_login = "ifuri-validator-agent[bot]"
INPUT semantic_review_assessment = {"schema":"subactor.validator/semantic-review-assessment/v1","subject":{"repository":"semcod/redup","pull_request":23,"head_sha":"3e2dea113b15178522f71b589e62079da75403bf","base_sha":"c290460e4fdab34ffe8fa9e9e48092d7fee77939","diff_sha256":"7e9a6eccf94c388188a9d5ad9e778f566ae7f5f9776019536d607e663acfe411"},"policy":{"policy_schema":"subactor.validator/semantic-review-policy/v1","policy_version":1,"policy_sha256":"676cb4516bbfed2a000e40b9b1b6e4a430ecc761ec546aeb53d721a1905cfdd7","required":false,"critical_paths":[],"observed_paths":["packages/redup-fast-hash/src/lib.rs","project/TICKETS.md","project/ticket-016/README.md","project/ticket-016/intent.json","src/redup/core/lsh_matcher.py","src/redup/core/pipeline/duplicate_finder.py","tests/test_lsh_matcher.py"]},"grounding":"full-diff-not-per-finding-proof","execution_authority":false,"status":"not_required","reason":null,"review_sha256":null,"unresolved":[]}
INPUT superseded_checks = []
INPUT ticket_radar_receipt = {"schema":"subactor.ticket-radar/v1","base_sha":"c290460e4fdab34ffe8fa9e9e48092d7fee77939","head_sha":"3e2dea113b15178522f71b589e62079da75403bf","change_digest":"b1bd8f20c967d58ff02836bfbe34ce60b35c1933c61b81a7f68aeb997d0fc7bf","score":52,"complexity":"M","estimated_minutes":58,"split_recommended":true,"services":["packages/redup-fast-hash"],"authority":"ADVISORY","promotion":"FORBIDDEN"}
VERDICT APPROVE AUTHORITY DETERMINISTIC
REJECTED REQUEST_CHANGES BECAUSE NO_UNSAFE_CHANGE_REASON_FOUND
ADVISORY llm_verdict = "" MODEL "gemini/gemini-3.8-flash"
ASSERT VERDICT_AUTHORITY != "ADVISORY"