Repository navigation
chore(bench): enforce clean bench protocol - #152
Merged
Merged
Conversation
- Add bench preflight (alias, template/props, latency) - Pass attempt cap and protocol metadata in final bench - Add health probes and cap per attempt - Add max_tokens to pilot facts and env-configurable packs dir - Add nssm setup script and update runbook/CHANGELOG
- ask_facts disables Qwen3 thinking via chat_template_kwargs: thinking mode left content empty, so every attempt ran on lictor harvest alone - per-model workdir (base/trained): the trained arm no longer overwrites the base arm's run.json/target.facts/trail - record the LLM fact count in trajectory metadata and report attempts_without_llm_facts in results - rewrite preflight: root/v1 URL normalization, template detection, 100-column formatting
Free-form fact lines were not reproducible across models: the base
emitted value(...) shapes, the trained student emitted valueless
predicate("name") lines and degenerated to max_tokens, so the bench
measured lictor's harvest alone.
- ChatRequest gains response_format, forwarded by the OpenAI-compatible
provider (llama.cpp constrains decoding to the schema)
- ask_facts asks for one {"facts": [{name, value}]} object with an enum
of allowed names, a value pattern per domain, and maxItems = 16 so a
repeating model still closes the array
- _facts_from_json maps the answer onto the fact grammar and keeps
clean_facts as the domain/allow-list filter; the line parser stays as
a fallback for providers without constrained decoding
- preflight fails when an endpoint ignores response_format, so the bench
cannot silently measure the parser
The enriched probe showed both models classify every overflow challenge as fmtstr (2/6 correct): the prompt never explained what sof, fmtstr and uaf mean in source terms. The prompt now defines each class by its code pattern, caps vuln at one emission, and forbids guessing when no pattern matches (a wrong class sends the engine down a failing path).
Single-turn records aligned with the constrained JSON schema: the enriched FACT_PROMPT in, the ground-truth facts out. The 8 training- eligible lab targets carry the majority-vote vuln class (fmtstr 2, sof 3, uaf 3) plus the majority LLM facts mined from the mr6-v2 trajectories. The 72 empty-facts-step records are dropped: an empty completion teaches the model to stay silent, the exact failure the bench counters.
The majority-vote assembler collapsed every trajectory into one record per target (17 records). The union extractor mines every (target, facts) step from four historical sources instead: rollout trajectories, twin trajectories, the consolidation mix, and the mr6-v2 dataset. - Ground-truth vuln class from VULN_TRUTH, never from the trajectory. - Twin alias uaf_heap -> uaf_heap-002 keeps the eval target untouched. - Empty-facts steps are dropped and counted (381): an empty emission teaches the model to emit nothing. - Dedup key includes the prompt material: two targets with identical facts are two distinct samples. Result: 58 records (was 8), reproducible byte-for-byte from committed sources. Runbook in docs/how-to/expand-facts-dataset.md covers the full chain: extraction, QLoRA training, GGUF export, nssm serving, benchmark.
Keep both layers: the main recipe (mode full/model, clean-only verdict) and the protocol fixes (attempt cap, json_schema facts, per-arm workdir, preflight). The mode/attempt-cap conflicts in run_final_bench.sh and final_bench.py compose, no behavior dropped. test_model_only_pilot asserts the JSON prompt the protocol now enforces.
log0u7
added a commit
that referenced
this pull request
Nov 15, 2026
The original MR9 run (2026-11-11) was destroyed by a worktree cleanup before its results were ever committed. This run is the committed replacement, executed under the clean bench protocol (#152). Mode full, 10 reps, 29 challenges x 2 models: base 168/290 (0.58), trained-mr9 169/290 (0.58) clean-only: 40/110 = 0.364 both sides - matches the recorded MR9 verdict; the protocol rule 5 (completed run = committed run) is what makes this run survive the next cleanup.
log0u7
added a commit
that referenced
this pull request
Nov 15, 2026
Rerun of the MR10 bench destroyed with the worktree remove --force incident (2026-11-15). Same protocol as the MR9 restore (#167): mode full, 10 reps, attempt cap 600, seed 7, served trained-mr10 (NSSM alias verified by preflight). Verdict (clean-only, 11 challenges): 40/110 vs 40/110 (0.364 vs 0.364), identical to the destroyed MR10-full verdict. Per-challenge table identical to the MR9 rerun: trained-mr9 and trained-mr10 (different GGUFs) produce identical eval outcomes; the only trained edge left is uaf_train_b +1 (9/10 -> 10/10, contaminated). The destroyed runs' globals (198/207) pre-date the attempt cap (#152) and the system repairs; the clean-only verdict is the reproducible contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enforce a clean, reproducible protocol for the final benchmark:
template availability, endpoint latency, and
json_schemaconstraineddecoding. A failed preflight refuses the run instead of burning GPU hours on
a misconfigured endpoint.
chat_template_kwargspassthrough. The run-20261106 smoke test loggedthe LLM emitted no usable factson every attempt: thinking mode returnedthe text in
reasoning_contentand leftcontentempty, so flags cameonly from lictor's deterministic harvest and the bench measured nothing
about the model.
not reproducible across models: the base emitted
value(...)shapes, thetrained student emitted valueless
predicate("name")lines and degeneratedto
max_tokens. The request now asks for one{"facts": [{"name": ..., "value": ...}]}object with an enum of allowednames, a value pattern per domain, and
maxItems = 16so a repeating modelstill closes the array.
clean_factsstays as the domain and allow-listfilter; the line parser remains a fallback for providers without constrained
decoding.
workdir/base,workdir/trained).pilot.run_pilotwritesrun.json,target.factsandtrail-*.jsonlunder
workdir/<binary>; with one shared directory the trained armoverwrote the base arm's record.
attempts_without_llm_factsis reported per model inresults.jsonand
results.md, so an empty-fact bench cannot pass for a model score.max_tokenscap on fact generation, configurableAUGURE_PACKS_DIR.bf-bench-base/bf-bench-trainedandbf-trainrepair guidance, plus runbook and CHANGELOG updates.
Tests (mandatory)
Red-green, failing tests written first:
tests/unit/test_pilot.py::test_ask_facts_disables_thinking_generationfailed before the fix (
request.extra_bodywasNone) and checks thechat_template_kwargspayload and themax_tokenscap.tests/unit/test_pilot.py::test_ask_facts_requests_a_constrained_json_schemafailed on
request.response_formatbeingNoneand pins the schema shape(
name/valuerequired,maxItems = 16).tests/unit/test_pilot.py::test_ask_facts_parses_constrained_json_into_predicatesand
::test_ask_facts_drops_json_facts_outside_the_value_domainspin themapping from the JSON answer onto the fact grammar, including domain drops.
tests/unit/test_pilot.py::test_trajectory_metadata_records_the_llm_fact_countfailed with
KeyError: 'facts'and now pins the metadata contract.tests/unit/test_bench_preflight.py(new file) fails withoutcheck_constrained_outputand covers: schema honoured, prose answer(endpoint ignored the schema), HTTP error, thinking disabled in the probe,
and both endpoints included in
run_preflight.tests/unit/test_bench_lab.py::TestEvidenceSeparation::test_each_model_gets_its_own_evidence_directoryfailed (both arms shared the root workdir) and now asserts disjoint
base/trainedevidence directories.tests/unit/test_bench_lab.py::TestEvidenceSeparation::test_counts_attempts_where_the_llm_emitted_no_factsfailed with
KeyError: 'attempts_without_llm_facts'and now pins theper-model counter.
Linked Issue
No issue exists. Rationale: the MR6 benchmark run produced numbers that were
not valid measurements (no model facts, overwritten evidence), so the protocol
must fail loudly before the next bench and keep the two arms' evidence apart.
Quality Checklist
CHANGELOG.mdunder[Unreleased]Bench validation (mr7-58, 10 reps, git cc880d1)
scripts/run_final_bench.sh mr7-58-clean-facts 10on the clean-only set of12 challenges (7 contaminated ids excluded per the manifest):
run-20261106-mr7-58-clean-facts-10rep: base 61/120 (0.508) vs trained68/120 (0.567), a net +7 clean solves over the base arm.
run-20261106-135649-345930, Qwen3-1.7B QLoRA4bit, 20 epochs, loss 1.195 -> 0.0735) moved the trained arm from
60/120 (pilot, 8 records) to 68/120.
shellcode-002improved from 1/10 (base) to 10/10 (trained): theconstrained facts finally steer augure's technique selection.
attempts_without_llm_factsis 0/190 on both arms: every attempt nowcarries model-emitted facts, so the score measures the model, not lictor's
deterministic harvest.
ret2func-0029/10 ->8/10 and
srop-0011/10 -> 0/10. The net trend remains positive.Bench validation (mr6-v2-clean-schema, 1 rep, git cc880d1)
scripts/run_final_bench.sh mr6-v2-clean-schema 1(JSON-schema factvalidation smoke run, clean-only set of 12 challenges):
constrained_outputcheck: both base and trained arms honourresponse_formatwithFACT_JSON_SCHEMA.srop-001. Consistent with the mr7-58 trend.contamination: LEAKEDis expected for the 7 mined challenge ids.