Skip to content

feat(bench,dataset): the MR9/MR10 cycle - recipes, corrected labels, the eval grids - #165

Merged
log0u7 merged 9 commits into
mainfrom
feat/lab-eval-twins
Nov 15, 2026
Merged

log0u7 merged 9 commits into
mainfrom
feat/lab-eval-twins

Conversation

@log0u7

@log0u7 log0u7 commented Nov 15, 2026

Copy link
Copy Markdown
Owner

Summary

The MR9/MR10 cycle lands: the datasets, the bench grids, the eval2 manifest.

  • the MR9 facts recipe: complete source material (12000-char excerpts), Prolog
    one-predicate-per-line completions matching the bench prompt verbatim, class
    balance via minority over-sampling (uaf x3, fmtstr x2), group-key split so
    no over-sampled copy leaks from train to test
  • the ORW labels corrected to sof (both orw targets are unbounded-copy
    stack overflows, the printf there prints a fixed string) and the ground
    truth loader parsing Prolog completions
  • the 10 MR6 eval twins and the 10 eval2 (harder) twins join the final
    benchmark grid, mapped to ports, with the ingest manifest carrying their
    verified-tier entries and the cases/ examples carrying their shapes

Bench evidence (runs in experiments/final-bench-results/): mr9 199/290 vs
208/290, mr10-full 198/290 vs 207/290, mr10-fixed 210/290 vs 217/290 with
the clean-11 moving from 40/110 (0.364) to 50/110 (0.455) - shellcode-eval
0/10 to 10/10 after the system-side repairs.

Tests (mandatory)

The dataset validation tests (Prolog grammar, contamination gate, class
balance) run green; the bench runs are data, not code.

Linked Issue

No issue: rationale = the MR9/MR10 cycle the articles document, landed as
data and bench plumbing.

Quality Checklist

  • I wrote the failing test first (red), then made it pass (green)
  • I ran the lint check (zero offenses)
  • I ran the test suite (all green)
  • I ran the build
  • I updated CHANGELOG.md under [Unreleased]
  • This change is focused and avoids unrelated behavior changes
  • I updated docs when user-facing workflow, tool, or config behavior changed

The clean verdict was statistically dead: 19 lab challenges, 18
contaminated by design (sft-mr6/v2 + grpo-translated cover 18/19),
one clean target (uaf_heap-001). The 10 eval twins (lictor
feat/lab-eval-twins, never in any training dataset) fix that: one
twin per capability class (fmtstr, orw, ret2plt, ret2func, shellcode,
ret2libc, callme, srop) plus two UAF class-transfer targets
(uaf-eval-001/002). LICTOR_LAB_PORTS grows by 10 entries (ports
19980-19999), 10 verified case files and 10 manifest rows land. The
contamination audit keeps them clean by construction: no
sft-*-facts-<twin> record exists. Clean list: 1 -> 11 challenges.
Regression tests: the twins map to their ports and no two challenges
share a port.
…lass balance

Three measured defects of the mr8-20 recipe (bench v6: trained-mr8
collapsed every UAF challenge to vuln=sof while the base model stayed
correct):

1. TRAIN/EVAL PROMPT MISMATCH - sft-mr8-facts was built with the older
   build_sft_facts.FACT_PROMPT; the bench evaluates against the newer
   brainforge.experiments.pilot.FACT_PROMPT. The ingest now raises
   EXCERPT_CHARS 1500 -> 12000 so the case-file material is the full
   source (the UAF free() call sits past the 1500-char mark), and
   assemble_sft_mr9.py embeds the BENCH prompt verbatim.

2. FORMAT CONFLICT - mr8 completions taught the JSON envelope while the
   prompt demands one-predicate-per-line Prolog style. MR9 completions
   are plain Prolog facts. The dataset validator now accepts a Prolog
   fact block as the final assistant payload: the gate's grammar mirrors
   the bench scorer's _PROLOG_RE, so a completion passes exactly when
   the scorer can parse it (TDD: 4 new tests in test_validation.py).

3. CLASS COLLAPSE - 13/20 sof records vs 3 uaf taught majority-label
   emission. MR9 over-samples class-minority targets (uaf x3, fmtstr
   x2) for a sof 13 / fmtstr 8 / uaf 9 mix, and unions the 219 MR6-v2
   agent-loop rows so the fine-tune cannot forget the pipeline behaviour
   (the 20-record pool alone caused catastrophic forgetting).

The re-ingest also regenerated the corpus case files from the lictor lab
(32 targets, ret2csu re-added as a manual manifest entry: binary-only,
no service.c) and the manifest grew 75 -> 78 entries. The bench still
loads the same 29 live challenges: callme, callme_train, ret2libc_train
and srop_train carry no lab port and are skipped.
The orw targets are sof (unbounded strcpy/memcpy on char small[8], the
printf calls are on FIXED strings): VULN_TRUTH orw/orw-002 change from
fmtstr to sof. The MR10 dataset is the MR9 recipe rebuilt with the
corrected labels: assemble_sft_mr9.py is parametrized (--id-prefix,
--date) and the majority votes no longer fill the vuln fact (VULN_TRUTH
is authoritative; the MR6 vote carried the wrong orw label). orw-002
also gets the seccomp=true override. datasets/sft-mr10-facts/train.jsonl:
247 records, 20 unique facts ids.

The bench ground truth loader silently dropped every MR9/MR10 record
(json.loads failed on the Prolog completions), so expected_vuln kept
the stale MR8 labels. It now parses plain-Prolog completions too
(vuln("sof").); first-wins over the sorted scan puts the corrected
MR10 dataset ahead of MR8/MR9.

Two stale tests updated to match the committed manifest: the 10 MR6
eval twins (commit 1028b7c) and the 3 MR8 training twins (commit
f5a4676) are manifest-covered now.
The trained-mr9 model is trained on the MR9 facts dataset; the
contamination audit must scan it. The MR10 facts dataset is included
proactively for the MR10 bench. The clean list is unchanged (11
challenges: the MR9/MR10 facts targets are the same as MR8 facts).
The default TRAINED_MODEL moves to trained-mr9.
The ingest manifest gains 10 verified-tier entries (one per eval2
twin, the expected flag read from each ORACLE) and the cases/ examples
carry the eval2 shapes. Coherent with the port mapping commit: the
manifest is what the bench consumes, the cases are what the operator
sees.
The glob pattern ctf-lictor-*-eval2-001.json missed the uaf twins'
double-suffix names (uaf-eval2-001-001.json): the manifest referenced
them, the disk cases were uncommitted, and the manifest/cases
regression guard caught the orphans.
…s absent

The facts datasets are gitignored artifacts (datasets/* with only
.gitkeep and eval/ committed): the test passed locally where the data
exists and failed on CI where it does not. The skip guard names the
reason - a live-data check, not a unit test.
@log0u7
log0u7 merged commit 13c3b6c into main Nov 15, 2026
8 checks passed
@log0u7
log0u7 deleted the feat/lab-eval-twins branch November 15, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant