[dominikpeter]

Refs #3 (Test Embeddings): this PR adds optional code embeddings and the measurements behind them (embedder bake-off, end-to-end results, find-by-description); see eval/REPORT.md.

Related issues: #1 (the benchmark added here was used to measure the comprehension-normalization question), #2 (the harness discovers new models by directory), #4 (ships through the existing auto-release flow).

Summary

Makes the detector measurably better at finding independently written re-implementations, adds SQL and
JavaScript/JSX support, adds three new commands (fragments, find, scan --explain), and adds optional
code embeddings (off by default, any provider). Everything is backed by a new labeled benchmark
(duplicatecode eval-groups) with separate dev and holdout sets. The default copies profile is not retuned.

4 commits (engine, CLI, eval, docs), about +3,400 / -100 lines. The original chronological experiment log was
squashed into these; what was tried, what failed and why is recorded in eval/REPORT.md.

What we built

Detection quality (no embeddings)

• Per-language and per-unit-kind weights (Weights gains file, sql, renormalize): SQL is scored on loose
  n-grams, referenced tables/columns and literals; whole-file units get weights without a name term; function
  units keep the name-aware weights. Features absent on both sides (no literals/calls) are renormalized away in
  the reimpl profile, so identical structure without literals is not penalized.
• IDF-weighted n-gram Jaccard (Idf, weighted_jaccard) and a new containment feature in the vector.
• Whole-file normalization for scripts: unused top-level functions/classes/imports/pure constants are
  dropped, a once-called main() wrapper is inlined, print/console output is kept (it is the program's
  result in a script), sys.setrecursionlimit and input = sys.stdin.readline lines are ignored.
• JS/TS normalization: optional ; and block braces no longer change tokens.
• Profile-aware default thresholds: copies keeps 0.6 / 0.4 / 0.45 (scan / diff / review); reimpl
  now defaults to 0.35 / 0.28 / 0.30 (+0.07 with embeddings), chosen from measured false-positive rates.

New languages

• SQL via tree-sitter-sequel: one unit per statement, clause-level statements, keywords case-folded,
  aliases/CTE names abstracted, table/column/function names kept.
• JavaScript / JSX (.js .mjs .cjs .jsx) handled by the existing TypeScript/TSX grammars.

New commands and options

• duplicatecode fragments <paths>: copied blocks (4+ identical normalized statements, --min-stmts,
  --min-tokens) inside different functions; nested units and constructors are excluded.
• duplicatecode scan --pairs --explain: per pair, the literals/calls only one side has and the source lines
  with no counterpart.
• duplicatecode find "<description>" <paths>: ranks existing units by embedding similarity to a plain-language
  description ("does this already exist?").
• --embed <preset> (minilm, qwen3, potion, openai, cohere), --embed-code, --embed-weight,
  --embed-max-chars.

Optional embeddings (off unless requested)

• Whole-unit embeddings blended as a 12th scoring feature; units without a vector skip it.
• Providers: OpenAI, OpenAI-compatible servers (bearer auth fixed), Azure AI Foundry / Azure OpenAI, Cohere
  (native /v2/embed). Vectors cached per unit text and model; cache key is a stable FNV-1a hash.
• eval/embed_server.py: small local OpenAI-compatible server around embed-anything (CPU), loads whichever
  model a request names, so cached vectors cannot mix models.

Evaluation harness (eval/, see eval/README.md and eval/REPORT.md)

• eval-groups: AUC, TPR at 1%/5% FPR, top-1 and score-at-fixed-FPR for labeled clone groups, dev and holdout.
• Datasets: CodeNet Python/JS (independent solutions), SQL (real queries + sqlglot rewrites/edits), React
  (real TSX components + mutations), function-level LLM re-implementations (the repo's dataset/, split by
  task parity). Fetch/build scripts and an embedder bake-off (embed_bench.py) are included.

What we tested, and the results

Metric: mean of (AUC + TPR@1%FPR) / 2 for the combined score; each set has dev (tuned on) and holdout
(disjoint problems/queries/components/tasks, except CodeNet: its dev and holdout were only separated by a
random seed and overlap, 13 of 60 Python and 4 of 60 JavaScript problems; see the note below the table).

set	before (dev / holdout)	after (dev / holdout)
Python, whole files (CodeNet)	0.559 / 0.510	0.693 / 0.659
JavaScript, whole files (CodeNet)	0.391 / 0.368	0.503 / 0.497
Function level, Python (LLM re-implementations)	0.645 / 0.647	0.698 / 0.706
Function level, TypeScript	0.678 / 0.669	0.823 / 0.728
SQL (real queries, rewrites/edits)	0.836 / n/a	0.989 / 0.993
React (real components, mutations; regression guard)	0.992 / 0.980	0.991 / 0.980

"Before" for the function-level sets is the original weights on the current normalizer (the original build
cannot run the new benchmark), so the true baseline is somewhat lower. SQL "before" is the new parser with the
old weights. SQL and React are mutation-based and saturate; they are guards, not evidence of independent
re-implementation. More than 90% of positives in the CodeNet and function-level sets are independent solutions
(under 50% 4-gram overlap), so those numbers are not inflated by near-copies.

Optional embeddings (MiniLM, --embed minilm, weight 0.35), through the Rust path: function-level Python
0.698 / 0.706 to 0.811 / 0.791, TypeScript 0.823 / 0.728 to 0.836 / 0.901. Embedder bake-off on the
clean function-level and SQL data (eval/REPORT.md): all-MiniLM-L6-v2 +0.083 mean gain, Qwen3-Embedding-0.6B
+0.081 (about 5x slower on CPU), bge-small +0.052, bge-base +0.042, jina-v2-base +0.018, potion-base-8M +0.013.
Several models (ModernBERT, bge-m3, jina code models) do not load in embed-anything. CodeNet embedding scores
are not quoted: likely training-set contamination.

New commands

• fragments: injection benchmark (renamed blocks of real functions pasted into others). Recall at the default
  4 statements / 30 tokens: blocks of 6: Python 0.93, TypeScript 0.975; blocks of 8: 0.97 / 0.97; blocks of 4:
  0.80 / 0.84 (the old 5/40 setting found only 0.43 / 0.33). Sampled reports on a real repo were genuine
  duplicates. A bug found by reading the output (a one-line comprehension counted as a 4-statement block) is
  fixed and covered by a test.
• find: 50 task descriptions over 168 implementation units, right unit ranked first 94% (MiniLM) / 96%
  (Qwen3), recall@3 100%; with 1,379 unrelated real units added: 82% first, 96% in top 3, 100% in top 10
  (MiniLM).
• Thresholds: score at 1% / 0.1% FPR on unrelated pairs is stable across datasets and dev/holdout (about 0.25 /
  0.34; with MiniLM 0.33 / 0.42).

Engineering checks

• Unit tests 24 -> 51 (SQL normalization, dead-code pruning, main() inlining, script noise, cache/embedding
  pipelines incl. Cohere request/response shapes, credential and key selection, cache recovery, fragments,
  explain, prefilter agreement with brute force for every profile, deeply nested input). cargo fmt --check,
  cargo test --workspace --locked, and the CI wheel smoke test (maturin build, install, scan dataset) pass
  locally on Linux.
• The existing bench (function-level LLM benchmark) compared against a build of main: combined top-1 improves
  on most rows (Python strict 94% -> 99%, stricthard 79% -> 83%; TypeScript strict 95% -> 97%, stricthard 85% ->
  88%); a few hard rows are 2-3 points lower (Python hardhard 70% -> 67%, loosehard 70% -> 67%; TypeScript
  hard~hard 85% -> 83%).
• Clippy is clean for new code (pre-existing warnings in bench.rs and the cache loader are untouched).
• Scan time on an 11k-unit repository, best of two runs, original main vs this branch: default scan 4.7 s -> 4.0 s,
  scan --min-name 0 13.3 s -> 6.9 s, --profile reimpl --min-name 0 4.0 s -> 12.2 s (about 3x slower: at its
  weights the k-gram prefilter cannot prune), fragments 2.2 s.
• Holdout caveat: on the holdout problems that are NOT in dev, Python scores 0.648 (0.659 on the full holdout)
  and JavaScript 0.497 (unchanged).

Review and fixes

The branch went through a full BMS code review (architecture gate, security, two correctness reviewers, agent
docs, performance, Python scripts, duplication) with a verification pass on every blocking/should-fix candidate;
refuted candidates were dropped. Fixed in this PR (each with a regression test where code is involved):

• Credentials: an Azure token is no longer sent to non-Azure endpoints (a keyless OpenAI-compatible or Cohere
  endpoint gets no credential); the API key is chosen from the endpoint's own provider (an OpenAI key can no
  longer reach a Cohere endpoint); no redirects are followed (an api-key header cannot be replayed to another
  host); the key is redacted from Debug.
• Uploads: the local presets accept only a loopback endpoint; the destination host is printed before any
  request (with a warning for plain http to a remote host); units that cannot match (--min-tokens,
  --skip-tests, boilerplate) are not embedded; the embedding cache is owner-only and keyed by a 128-bit hash.
• Cache: a corrupt or truncated tail is cut off before the next append (it used to make every later record
  unreadable, so the same texts were re-fetched on every run); empty or non-numeric embeddings are rejected
  instead of being stored as hits; oversized entries no longer desync the file.
• Scoring: --embed-code no longer changes the copies renormalization; the embedding term is dropped
  whenever a unit has no vector; the k-gram prefilter no longer loses pairs that share no 4-gram but score above
  the threshold; scan computes only the features with weight, which made the default scan faster than main.
• SQL and JS: plain SQL queries are named like their file (every SELECT was named query, which made any two
  queries look alike by name in the default profile); .js/.mjs/.cjs use the TSX grammar so React components in
  plain JS files are parsed.
• Normalization and output: attribute reads inside functions are no longer treated as script noise; main()
  inlining keeps an else branch and multi-line strings; compound statements report their header lines only
  (--explain, fragments); extraction runs on a large stack so deeply nested input cannot abort a scan.
• CLI: fragments and find with several roots keep the root in file names; --explain pairs units by
  full place instead of file:line.
• Benchmarks and docs: the eval scripts were hardened (argument parsing, byte-offset bug in the TypeScript
  fragment generator, path filters, unsafe paths from remote data, local server validation and model allowlist,
  pinned versions); the repo's own agent skill, --help text and crate descriptions now describe SQL/JS, the new
  commands and the embedding privacy implications.

Not fixed here (listed in eval/REPORT.md): --profile reimpl is slower than main; SQL scoring compares every
same-family unit; per-language rules are still inlined in units.rs and whole-file pruning rewrites source text
instead of the token stream; Unit.text copies source for every unit.

Behavior changes to review

1. scan/diff/review now also read .js .mjs .cjs .jsx and .sql files. More units, so more
   findings, which matters for --fail-on-found CI users. There is no language filter yet.
2. Default copies scoring is unchanged for Python: on a real internal Python repository a scan is identical
   to main (209 pairs, 0 score differences). C# token rules are untouched and its tests pass, but it was not
   re-checked against real C# code.
3. TypeScript default results shift slightly because ; and block braces no longer count as tokens: on a
   real TSX frontend, 316 pairs are common to both builds, 31 of them change score (largest change 0.118), 6
   pairs from main drop out and 2 appear. Net effect on the function-level benchmark is positive (a no-;/brace
   ablation scored 0.8064 / 0.7099 vs 0.8225 / 0.7282), but it is a change.
4. Credentials and uploads: keyless OpenAI-compatible and Cohere endpoints now get no Authorization header
   (previously an Azure token could be attached); local --embed presets refuse a non-loopback
   DUPLICATECODE_EMBED_ENDPOINT; --embed* prints where the text is sent before sending it. Plain SQL queries
   are named after their file, so SQL pairs now need the same file name to get a name match.
5. Engine library API: Weights has new fields, N_FEATURES is 12 (was 10), Scores has new fields and
   Unit carries text, vec and statement data. Embedding is opt-in; Unit.text costs one copy of each unit's
   source in memory even when embeddings are off.
6. New dependency: tree-sitter-sequel (SQL grammar). No other runtime crates; the Python scripts in
   eval/ use inline uv dependencies and are not part of the wheel.

Known limitations

• Test sets are small (12-13 tasks per language at function level) and mostly built by us; the headline gains
  are directional, and the embedding gain varies by set (TypeScript dev is flat). The embedded text includes
  docstrings, so part of the embedding gain is intent matching.
• SQL and React sets are mutation-based; SQL fragments are not benchmarked; the fragments benchmark covers
  Python and TypeScript only. There are no labeled real-repository clone pairs; real-repo checks were manual
  samples.
• Qwen3 on CPU is impractical (about 3.5 hours estimated for a first scan of an 11k-unit repository); embedding
  runs here were CPU-only. Hosted models (OpenAI, Cohere) are wired in but were not benchmarked or exercised
  against a live endpoint (no key available): Cohere is covered by request/response unit tests only.
• macOS and Windows CI is untested locally (the cache permission code is cfg(unix)).
• The CodeNet dev/holdout overlap above also means the earlier "disjoint" description of those two sets in the
  evaluation docs was wrong; eval/fetch_codenet.py --exclude-from builds a disjoint holdout for future runs.
• The PyPI embed extra, a .duplicatecode.toml, --embed-optional and --allow-upload are proposals, not
  built (listed in the README).

Decisions for the owner

• Version: bumped to 0.3.0 in this PR (new features). Per the README, merging a version bump to main
  tags and publishes automatically, so merge only when a release is wanted.
• Whether JS/SQL should be scanned by default or behind a language option (see behavior change 1).
• Whether eval/ (scripts, report, about 700 lines of Python) belongs in this repo or a separate one.
• Whether reimpl should become the default profile for find-style use, or stay opt-in.

Reviewing

Suggested order: eval/REPORT.md (what was measured and what failed) -> index.rs (scoring) -> units.rs
(normalization, SQL, pruning) -> embed.rs -> fragments.rs / explain.rs -> main.rs / groups.rs.
Reproduce: eval/README.md. The experiment journal (kept and discarded ideas, with numbers) is in
eval/REPORT.md.

🤖 Generated with Claude Code