Skip to content

chore(bench): enforce clean bench protocol - #152

Merged
log0u7 merged 10 commits into
mainfrom
chore/bench-protocol-clean
Nov 15, 2026
Merged

log0u7 merged 10 commits into
mainfrom
chore/bench-protocol-clean

Conversation

@log0u7

@log0u7 log0u7 commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Summary

Enforce a clean, reproducible protocol for the final benchmark:

  • Preflight before any attempt: distinct servers, served aliases, chat
    template availability, endpoint latency, and json_schema constrained
    decoding. A failed preflight refuses the run instead of burning GPU hours on
    a misconfigured endpoint.
  • Pilot fact requests disable Qwen3 thinking through the llama.cpp
    chat_template_kwargs passthrough. The run-20261106 smoke test logged
    the LLM emitted no usable facts on every attempt: thinking mode returned
    the text in reasoning_content and left content empty, so flags came
    only from lictor's deterministic harvest and the bench measured nothing
    about the model.
  • Fact output is constrained to a JSON schema. Free-form fact lines were
    not reproducible across models: the base emitted value(...) shapes, the
    trained student emitted valueless predicate("name") lines and degenerated
    to max_tokens. The request now asks for one
    {"facts": [{"name": ..., "value": ...}]} object with an enum of allowed
    names, a value pattern per domain, and maxItems = 16 so a repeating model
    still closes the array. clean_facts stays as the domain and allow-list
    filter; the line parser remains a fallback for providers without constrained
    decoding.
  • Per-model evidence directories (workdir/base, workdir/trained).
    pilot.run_pilot writes run.json, target.facts and trail-*.jsonl
    under workdir/<binary>; with one shared directory the trained arm
    overwrote the base arm's record.
  • attempts_without_llm_facts is reported per model in results.json
    and results.md, so an empty-fact bench cannot pass for a model score.
  • Per-attempt wall-clock cap, health probe, protocol metadata in results,
    max_tokens cap on fact generation, configurable AUGURE_PACKS_DIR.
  • nssm setup script for bf-bench-base / bf-bench-trained and bf-train
    repair guidance, plus runbook and CHANGELOG updates.

Tests (mandatory)

Red-green, failing tests written first:

  • tests/unit/test_pilot.py::test_ask_facts_disables_thinking_generation
    failed before the fix (request.extra_body was None) and checks the
    chat_template_kwargs payload and the max_tokens cap.
  • tests/unit/test_pilot.py::test_ask_facts_requests_a_constrained_json_schema
    failed on request.response_format being None and pins the schema shape
    (name/value required, maxItems = 16).
  • tests/unit/test_pilot.py::test_ask_facts_parses_constrained_json_into_predicates
    and ::test_ask_facts_drops_json_facts_outside_the_value_domains pin the
    mapping from the JSON answer onto the fact grammar, including domain drops.
  • tests/unit/test_pilot.py::test_trajectory_metadata_records_the_llm_fact_count
    failed with KeyError: 'facts' and now pins the metadata contract.
  • tests/unit/test_bench_preflight.py (new file) fails without
    check_constrained_output and covers: schema honoured, prose answer
    (endpoint ignored the schema), HTTP error, thinking disabled in the probe,
    and both endpoints included in run_preflight.
  • tests/unit/test_bench_lab.py::TestEvidenceSeparation::test_each_model_gets_its_own_evidence_directory
    failed (both arms shared the root workdir) and now asserts disjoint
    base/trained evidence directories.
  • tests/unit/test_bench_lab.py::TestEvidenceSeparation::test_counts_attempts_where_the_llm_emitted_no_facts
    failed with KeyError: 'attempts_without_llm_facts' and now pins the
    per-model counter.

Linked Issue

No issue exists. Rationale: the MR6 benchmark run produced numbers that were
not valid measurements (no model facts, overwritten evidence), so the protocol
must fail loudly before the next bench and keep the two arms' evidence apart.

Quality Checklist

  • I wrote the failing test first (red), then made it pass (green)
  • I ran the lint check (zero offenses)
  • I ran the test suite (all green)
  • I ran the build
  • I updated CHANGELOG.md under [Unreleased]
  • This change is focused and avoids unrelated behavior changes
  • I updated docs when user-facing workflow, tool, or config behavior changed

Bench validation (mr7-58, 10 reps, git cc880d1)

scripts/run_final_bench.sh mr7-58-clean-facts 10 on the clean-only set of
12 challenges (7 contaminated ids excluded per the manifest):

  • run-20261106-mr7-58-clean-facts-10rep: base 61/120 (0.508) vs trained
    68/120 (0.567), a net +7 clean solves over the base arm.
  • The 58-record SFT run (run-20261106-135649-345930, Qwen3-1.7B QLoRA
    4bit, 20 epochs, loss 1.195 -> 0.0735) moved the trained arm from
    60/120 (pilot, 8 records) to 68/120.
  • shellcode-002 improved from 1/10 (base) to 10/10 (trained): the
    constrained facts finally steer augure's technique selection.
  • attempts_without_llm_facts is 0/190 on both arms: every attempt now
    carries model-emitted facts, so the score measures the model, not lictor's
    deterministic harvest.
  • Known regressions (strict per-challenge metric): ret2func-002 9/10 ->
    8/10 and srop-001 1/10 -> 0/10. The net trend remains positive.

Bench validation (mr6-v2-clean-schema, 1 rep, git cc880d1)

scripts/run_final_bench.sh mr6-v2-clean-schema 1 (JSON-schema fact
validation smoke run, clean-only set of 12 challenges):

  • Preflight passed on both endpoints, including the new
    constrained_output check: both base and trained arms honour
    response_format with FACT_JSON_SCHEMA.
  • Verdict: base 6/12 (0.500) vs trained 7/12 (0.583), +1 GAIN on
    srop-001. Consistent with the mr7-58 trend.
  • contamination: LEAKED is expected for the 7 mined challenge ids.

log0u7 added 10 commits November 6, 2026 07:32
- Add bench preflight (alias, template/props, latency)
- Pass attempt cap and protocol metadata in final bench
- Add health probes and cap per attempt
- Add max_tokens to pilot facts and env-configurable packs dir
- Add nssm setup script and update runbook/CHANGELOG
- ask_facts disables Qwen3 thinking via chat_template_kwargs: thinking
  mode left content empty, so every attempt ran on lictor harvest alone
- per-model workdir (base/trained): the trained arm no longer overwrites
  the base arm's run.json/target.facts/trail
- record the LLM fact count in trajectory metadata and report
  attempts_without_llm_facts in results
- rewrite preflight: root/v1 URL normalization, template detection,
  100-column formatting
Free-form fact lines were not reproducible across models: the base
emitted value(...) shapes, the trained student emitted valueless
predicate("name") lines and degenerated to max_tokens, so the bench
measured lictor's harvest alone.

- ChatRequest gains response_format, forwarded by the OpenAI-compatible
  provider (llama.cpp constrains decoding to the schema)
- ask_facts asks for one {"facts": [{name, value}]} object with an enum
  of allowed names, a value pattern per domain, and maxItems = 16 so a
  repeating model still closes the array
- _facts_from_json maps the answer onto the fact grammar and keeps
  clean_facts as the domain/allow-list filter; the line parser stays as
  a fallback for providers without constrained decoding
- preflight fails when an endpoint ignores response_format, so the bench
  cannot silently measure the parser
The enriched probe showed both models classify every overflow challenge
as fmtstr (2/6 correct): the prompt never explained what sof, fmtstr
and uaf mean in source terms. The prompt now defines each class by its
code pattern, caps vuln at one emission, and forbids guessing when no
pattern matches (a wrong class sends the engine down a failing path).
Single-turn records aligned with the constrained JSON schema: the
enriched FACT_PROMPT in, the ground-truth facts out. The 8 training-
eligible lab targets carry the majority-vote vuln class (fmtstr 2,
sof 3, uaf 3) plus the majority LLM facts mined from the mr6-v2
trajectories. The 72 empty-facts-step records are dropped: an empty
completion teaches the model to stay silent, the exact failure the
bench counters.
The majority-vote assembler collapsed every trajectory into one record
per target (17 records). The union extractor mines every (target, facts)
step from four historical sources instead: rollout trajectories, twin
trajectories, the consolidation mix, and the mr6-v2 dataset.

- Ground-truth vuln class from VULN_TRUTH, never from the trajectory.
- Twin alias uaf_heap -> uaf_heap-002 keeps the eval target untouched.
- Empty-facts steps are dropped and counted (381): an empty emission
  teaches the model to emit nothing.
- Dedup key includes the prompt material: two targets with identical
  facts are two distinct samples.

Result: 58 records (was 8), reproducible byte-for-byte from committed
sources. Runbook in docs/how-to/expand-facts-dataset.md covers the full
chain: extraction, QLoRA training, GGUF export, nssm serving, benchmark.
Keep both layers: the main recipe (mode full/model, clean-only verdict)
and the protocol fixes (attempt cap, json_schema facts, per-arm workdir,
preflight). The mode/attempt-cap conflicts in run_final_bench.sh and
final_bench.py compose, no behavior dropped. test_model_only_pilot
asserts the JSON prompt the protocol now enforces.
@log0u7
log0u7 merged commit cdf3719 into main Nov 15, 2026
8 checks passed
@log0u7
log0u7 deleted the chore/bench-protocol-clean branch November 15, 2026 12:47
log0u7 added a commit that referenced this pull request Nov 15, 2026
The original MR9 run (2026-11-11) was destroyed by a worktree cleanup
before its results were ever committed. This run is the committed
replacement, executed under the clean bench protocol (#152).

Mode full, 10 reps, 29 challenges x 2 models:
  base 168/290 (0.58), trained-mr9 169/290 (0.58)
  clean-only: 40/110 = 0.364 both sides - matches the recorded MR9
  verdict; the protocol rule 5 (completed run = committed run) is what
  makes this run survive the next cleanup.
log0u7 added a commit that referenced this pull request Nov 15, 2026
Rerun of the MR10 bench destroyed with the worktree remove --force
incident (2026-11-15). Same protocol as the MR9 restore (#167):
mode full, 10 reps, attempt cap 600, seed 7, served trained-mr10
(NSSM alias verified by preflight).

Verdict (clean-only, 11 challenges): 40/110 vs 40/110 (0.364 vs 0.364),
identical to the destroyed MR10-full verdict. Per-challenge table
identical to the MR9 rerun: trained-mr9 and trained-mr10 (different
GGUFs) produce identical eval outcomes; the only trained edge left is
uaf_train_b +1 (9/10 -> 10/10, contaminated). The destroyed runs'
globals (198/207) pre-date the attempt cap (#152) and the system
repairs; the clean-only verdict is the reproducible contract.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant