Repository navigation
feat(bench,dataset): the MR9/MR10 cycle - recipes, corrected labels, the eval grids - #165
Merged
Merged
Conversation
The clean verdict was statistically dead: 19 lab challenges, 18 contaminated by design (sft-mr6/v2 + grpo-translated cover 18/19), one clean target (uaf_heap-001). The 10 eval twins (lictor feat/lab-eval-twins, never in any training dataset) fix that: one twin per capability class (fmtstr, orw, ret2plt, ret2func, shellcode, ret2libc, callme, srop) plus two UAF class-transfer targets (uaf-eval-001/002). LICTOR_LAB_PORTS grows by 10 entries (ports 19980-19999), 10 verified case files and 10 manifest rows land. The contamination audit keeps them clean by construction: no sft-*-facts-<twin> record exists. Clean list: 1 -> 11 challenges. Regression tests: the twins map to their ports and no two challenges share a port.
…lass balance Three measured defects of the mr8-20 recipe (bench v6: trained-mr8 collapsed every UAF challenge to vuln=sof while the base model stayed correct): 1. TRAIN/EVAL PROMPT MISMATCH - sft-mr8-facts was built with the older build_sft_facts.FACT_PROMPT; the bench evaluates against the newer brainforge.experiments.pilot.FACT_PROMPT. The ingest now raises EXCERPT_CHARS 1500 -> 12000 so the case-file material is the full source (the UAF free() call sits past the 1500-char mark), and assemble_sft_mr9.py embeds the BENCH prompt verbatim. 2. FORMAT CONFLICT - mr8 completions taught the JSON envelope while the prompt demands one-predicate-per-line Prolog style. MR9 completions are plain Prolog facts. The dataset validator now accepts a Prolog fact block as the final assistant payload: the gate's grammar mirrors the bench scorer's _PROLOG_RE, so a completion passes exactly when the scorer can parse it (TDD: 4 new tests in test_validation.py). 3. CLASS COLLAPSE - 13/20 sof records vs 3 uaf taught majority-label emission. MR9 over-samples class-minority targets (uaf x3, fmtstr x2) for a sof 13 / fmtstr 8 / uaf 9 mix, and unions the 219 MR6-v2 agent-loop rows so the fine-tune cannot forget the pipeline behaviour (the 20-record pool alone caused catastrophic forgetting). The re-ingest also regenerated the corpus case files from the lictor lab (32 targets, ret2csu re-added as a manual manifest entry: binary-only, no service.c) and the manifest grew 75 -> 78 entries. The bench still loads the same 29 live challenges: callme, callme_train, ret2libc_train and srop_train carry no lab port and are skipped.
The orw targets are sof (unbounded strcpy/memcpy on char small[8], the
printf calls are on FIXED strings): VULN_TRUTH orw/orw-002 change from
fmtstr to sof. The MR10 dataset is the MR9 recipe rebuilt with the
corrected labels: assemble_sft_mr9.py is parametrized (--id-prefix,
--date) and the majority votes no longer fill the vuln fact (VULN_TRUTH
is authoritative; the MR6 vote carried the wrong orw label). orw-002
also gets the seccomp=true override. datasets/sft-mr10-facts/train.jsonl:
247 records, 20 unique facts ids.
The bench ground truth loader silently dropped every MR9/MR10 record
(json.loads failed on the Prolog completions), so expected_vuln kept
the stale MR8 labels. It now parses plain-Prolog completions too
(vuln("sof").); first-wins over the sorted scan puts the corrected
MR10 dataset ahead of MR8/MR9.
Two stale tests updated to match the committed manifest: the 10 MR6
eval twins (commit 1028b7c) and the 3 MR8 training twins (commit
f5a4676) are manifest-covered now.
The trained-mr9 model is trained on the MR9 facts dataset; the contamination audit must scan it. The MR10 facts dataset is included proactively for the MR10 bench. The clean list is unchanged (11 challenges: the MR9/MR10 facts targets are the same as MR8 facts). The default TRAINED_MODEL moves to trained-mr9.
The ingest manifest gains 10 verified-tier entries (one per eval2 twin, the expected flag read from each ORACLE) and the cases/ examples carry the eval2 shapes. Coherent with the port mapping commit: the manifest is what the bench consumes, the cases are what the operator sees.
The glob pattern ctf-lictor-*-eval2-001.json missed the uaf twins' double-suffix names (uaf-eval2-001-001.json): the manifest referenced them, the disk cases were uncommitted, and the manifest/cases regression guard caught the orphans.
…s absent The facts datasets are gitignored artifacts (datasets/* with only .gitkeep and eval/ committed): the test passed locally where the data exists and failed on CI where it does not. The skip guard names the reason - a live-data check, not a unit test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The MR9/MR10 cycle lands: the datasets, the bench grids, the eval2 manifest.
one-predicate-per-line completions matching the bench prompt verbatim, class
balance via minority over-sampling (uaf x3, fmtstr x2), group-key split so
no over-sampled copy leaks from train to test
sof(both orw targets are unbounded-copystack overflows, the printf there prints a fixed string) and the ground
truth loader parsing Prolog completions
benchmark grid, mapped to ports, with the ingest manifest carrying their
verified-tier entries and the cases/ examples carrying their shapes
Bench evidence (runs in experiments/final-bench-results/): mr9 199/290 vs
208/290, mr10-full 198/290 vs 207/290, mr10-fixed 210/290 vs 217/290 with
the clean-11 moving from 40/110 (0.364) to 50/110 (0.455) - shellcode-eval
0/10 to 10/10 after the system-side repairs.
Tests (mandatory)
The dataset validation tests (Prolog grammar, contamination gate, class
balance) run green; the bench runs are data, not code.
Linked Issue
No issue: rationale = the MR9/MR10 cycle the articles document, landed as
data and bench plumbing.
Quality Checklist
CHANGELOG.mdunder[Unreleased]