feat(sleep): add adversarial candidate probes

[Bogdan (Dan) Baciu (bogdanbaciu21)]

What Problem This Solves

A candidate skill can improve its held-out validation score while depending on the exact wording or framing of harvested requests. It can therefore pass the ordinary gate and still break when the same request arrives with a harmless surface change.

The first revision of these probes scored only the candidate, so blocking mode could not distinguish brittleness the candidate introduced from prompt-frame sensitivity already present in the backend, and a single stochastic sample could mark a row brittle at the default margin.

Why This Change Was Made

This revision compares probe sensitivity with the baseline and adds a repeated-rollout consistency screen before any probe result can block adoption.

Identical source and probe pairs are scored under both the current documents and the candidate documents, each score is the mean of a configured number of repeated rollouts, and a row is brittle only when the candidate's probe minus source gap worsens beyond the margin relative to the baseline gap and the worsening holds in a strict majority of rollout indices. Blocking additionally requires at least two rollouts, so one stochastic sample can never reject a candidate; advisory runs may use one. The baseline documents are required arguments, so no caller can silently compare against an unintended baseline. Request-frame transformations are restricted to a defensible semantic-preservation contract: only explicitly politeness-marked requests are reframed, and ability, permission, and desire questions are never touched.

Project Fit

• Improves SkillOpt-Sleep by making candidate brittleness visible and attributable before adoption.
• Supports stable gains across varied request phrasing while preserving existing behavior when the feature is disabled.

User Impact

Operators can see which harmless request variation broke a staged candidate, and whether the breakage is candidate-introduced or pre-existing backend sensitivity. Advisory mode adds evidence without changing gate decisions; explicit blocking mode rejects only candidate-introduced degradation, after operators calibrate dream_adversarial_margin on their task mix and set dream_adversarial_rollouts to at least two.

Proof

The deterministic proof covers the pre-registered contract scenarios, each comparing the candidate arm versus the baseline arm on identical source and probe pairs.

1. A planted literal-only rule passes the ordinary held-out gate, is flagged by the probes, remains accepted in advisory mode, and is rejected in blocking mode; its evidence rows retain all four aggregated scores and show the gap change of -1.0 is candidate-introduced.
2. A backend that is equally frame-sensitive with and without the candidate produces a zero gap change and is never flagged, so pre-existing sensitivity cannot block adoption.
3. A candidate that improves both the source and probe scores while keeping the pre-existing gap produces a zero gap change and is never flagged, so the preserved gap does not reject that improved candidate.
4. A single-rollout fluctuation under the candidate produces a negative mean gap change but only half the rollout indices worsen, so the majority rule keeps the row stable. This deterministic test does not measure provider token usage.
5. A surface-robust candidate produces identical held-out gate results, artifacts, and acceptance decisions with probes disabled and enabled, while train/holdout isolation is covered separately.

Regression tests also pin train, validation, test, and provenance isolation; target-only routing in dual-backend operation; the three-variant, 256-probe, and eight-rollout resource bounds; strict configuration validation including the blocking rollout floor; non-finite and zero-probe fail-closed behavior; redaction; and Markdown-safe reporting.

These are deterministic contract tests, not a live-provider performance claim, so no stochastic lift or confidence interval is claimed.

Academic Support

• Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh, "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList," ACL 2020, arXiv:2005.04118. CheckList motivates capability-focused behavioral tests and invariance checks instead of relying on aggregate accuracy alone; this design applies that principle to candidate-skill gating.
• Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits, "Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment," AAAI 2020, arXiv:1907.11932. TextFooler demonstrates that meaning-preserving surface changes can flip model behavior; this supports probing candidates with bounded request-frame variants.
• Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr, "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting," ICLR 2024, arXiv:2310.11324. Its prompt-format sensitivity results motivate the wrapper, delimiter, and boundary perturbations exercised by these probes.
• Alexander Pan, Kush Bhatia, and Jacob Steinhardt, "The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models," ICLR 2022, arXiv:2201.03544. Its analysis of proxy misspecification supports candidate-specific stress tests, a baseline-relative attribution rule, and an advisory-first rollout before probe results can block adoption.

Testing

Current head 4d93cad055f9210438867d02c8e7c24236902040

The current head fixes the Markdown renderer to read all four baseline/candidate score fields and gap_change. Four new real-cycle cases cover brittle and robust candidates in advisory and blocking mode. Each checks the structured scores and the exact numeric row in the generated report.md; all four failed on the prior head 67b2f16441cf before the renderer fix.

For the planted brittle candidate, the report now renders the baseline source/probe scores as 0.000 / 0.000, candidate source/probe as 1.000 / 0.000, and gap change as -1.000, with status brittle.

Fork-runner validation below asserts actual_sha == expected_sha == 4d93cad055f9210438867d02c8e7c24236902040 before testing. These are contributor-run checks, not official upstream CI.

Platform	Python	Complete suite	Focused integration	Strict docs
Ubuntu	3.10	1,547 passed; 12 skipped; 353 subtests	207 passed; 2 skipped; 6 subtests	n/a
Ubuntu	3.11	1,547 passed; 12 skipped; 353 subtests	207 passed; 2 skipped; 6 subtests	n/a
Ubuntu	3.12	1,547 passed; 12 skipped; 353 subtests	207 passed; 2 skipped; 6 subtests	passed
macOS arm64	3.12	1,547 passed; 12 skipped; 353 subtests	207 passed; 2 skipped; 6 subtests	passed
Windows	3.12	1,502 passed; 57 skipped; 353 subtests	207 passed; 2 skipped; 6 subtests	passed

Local Linux/Python 3.12 validation also passed: python3 -m pytest tests/test_adversarial_dream.py -q reported 51 passed; python3 -m pytest -q reported 1,547 passed, 12 skipped, 353 subtests; strict docs and git diff --check passed. No test was deselected in the complete-suite runs and no failure was masked. The Windows count difference is reported as platform/optional-dependency skips.

The exact-head upstream CI run is action_required and awaits repository approval. No official upstream CI success is claimed.

Historical evidence: superseded head 84bbde9, not current validation

The matrix below was collected on fork runners against 84bbde9ac191, the candidate reviewed before the multi-skill rollout-forwarding fix. It is retained for history only. It does not describe the current head and it is not official CI.

Platform	Python	Complete suite	Focused integration	Strict docs
Ubuntu	3.10	1,538 passed; 12 skipped; 353 subtests	202 passed; 2 skipped; 6 subtests	n/a
Ubuntu	3.11	1,538 passed; 12 skipped; 353 subtests	202 passed; 2 skipped; 6 subtests	n/a
Ubuntu	3.12	1,538 passed; 12 skipped; 353 subtests	202 passed; 2 skipped; 6 subtests	passed
macOS arm64	3.12	1,538 passed; 12 skipped; 353 subtests	202 passed; 2 skipped; 6 subtests	passed
Windows	3.12	1,493 passed; 57 skipped; 353 subtests	202 passed; 2 skipped; 6 subtests	passed

Every job in that historical matrix asserted exact candidate 84bbde9ac191 before testing. No test was deselected and no failure was masked. The Windows difference is explicit platform and optional dependency skips, not failures.

Regression tests also pin train, validation, test, and provenance isolation; target-only routing in dual-backend operation; the three-variant, 256-probe, and eight-rollout resource bounds; strict configuration validation including the blocking rollout floor; non-finite and zero-probe fail-closed behavior; redaction; and Markdown-safe reporting.

Limitations & Negative Results

• These probes cover request-frame brittleness, not semantic correctness or a general adversarial guarantee; any fixed perturbation suite is an incomplete proxy.
• The aggregation rule uses means plus sign-consistency across paired rollouts; it is a screening decision, not a significance test. The paired evalkit provides separate paired statistical utilities; this probe decision does not run those tests.
• No live paid-provider result is claimed; the committed evidence deterministically exercises the real gate, evidence, and report plumbing.
• Enabled probes replay each eligible source and probe pair once per rollout under both arms, so the replay cost is rollouts times two times the pair count; the factor, probe, and rollout caps bound it.
• Blocking remains opt-in because the margin should be calibrated on each operator's task distribution from advisory evidence first.

Reproduce It Yourself

Linux or macOS, shell in a fresh working directory:

git clone --no-checkout https://github.com/microsoft/SkillOpt.git SkillOpt-review-263
cd SkillOpt-review-263
git fetch origin refs/pull/263/head
git checkout --detach 4d93cad055f9210438867d02c8e7c24236902040
python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev,docs]'
.venv/bin/python -m pytest tests/test_adversarial_dream.py -q
.venv/bin/python -m pytest -q
.venv/bin/python -m mkdocs build --strict

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Ready for review.

This is one commit and nine files on the current base, at exact head f1b2c651781eaf1f80d307b727218b68a84d4f9c. The PR body now includes the design rationale, deterministic proof, four directly relevant academic references, exact native test receipts, limitations, and reproduction commands.

The focused integration slice passed on Ubuntu, macOS, and Windows: 177 passed, 2 optional skips, and 6 subtests on each runner. The complete suites also passed with zero failures: Ubuntu and macOS 1,513 passed; Windows 1,468 passed. Strict docs passed on Python 3.12.

The CLA check passed. The upstream CI run is marked action_required because the fork workflow is waiting for repository approval; no CI job failed.

[Yifan Yang (Yif-Yang)]

Thanks for the unusually thorough test matrix and for making the feature opt-in. The direction is useful, but the blocking decision is not yet sound enough to gate adoption.

The main issue is attribution. evaluate_adversarial_probes() evaluates only the candidate skill/memory and flags candidate_probe_score - candidate_source_score < -margin. It never evaluates the same source/probe pair against the baseline skill/memory. Consequently, blocking mode cannot distinguish brittleness introduced by the candidate from prompt-frame sensitivity already present in the backend. A candidate that materially improves both variants, or simply preserves an existing source/probe gap, can be rejected even though adopting it is strictly better.

Please make blocking baseline-relative: evaluate baseline and candidate on identical source/probe pairs, then base rejection on a paired change such as the candidate gap worsening relative to the baseline gap. Evidence should retain all four scores so the decision is auditable. At minimum, add regressions where (1) baseline and candidate are equally frame-sensitive and adoption is not blocked, (2) the candidate improves both scores but retains a gap and is not blocked, and (3) only candidate-introduced degradation is blocked.

There are two related validity problems:

• Each source and probe is currently evaluated once through replay_batch(). With a stochastic provider and the default margin=0, any one-sample fluctuation can mark a row brittle. Please use paired repeated rollouts with a documented aggregation/confidence rule, or keep this evidence strictly advisory until such a rule is implemented and calibrated.
• _strip_polite_frame() treats every leading Can/Could/Would you ... as removable boilerplate. That is not generally meaning-preserving—for example, Can you swim? becomes Swim?. Please restrict transformations to cases with a defensible semantic-preservation contract and add negative tests for ability/permission/question forms.

Until baseline-relative comparison, stochastic robustness, and conservative transformations are covered, please remove or disable the blocking path and keep the probes advisory-only. The deterministic mock tests and cross-platform green suite verify plumbing, but they do not validate the blocking signal itself.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Got it. Will do. These are incredibly helpful comments thank you so much for the time and care you took to provide them. It will take me a bit to make these changes, will reply when complete at a high quality level.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Thanks again for the review. All three issues are addressed at head 84bbde9ac19113bf4dee29dbfbf0755aacde6b36 (one new commit on top of the reviewed head, so the delta is inspectable).

1. Blocking is now baseline-relative. evaluate_adversarial_probes() scores identical source/probe pairs under both the current (baseline) documents and the candidate documents, and rejection is based on the paired change: a row is brittle only when the candidate gap worsens beyond the margin relative to the baseline gap. The baseline documents are required arguments so no caller can silently compare against an unintended baseline. Evidence rows retain all four aggregated scores plus the per-rollout samples. The three regressions you named are in: equal frame sensitivity in both arms is not blocked; a candidate that improves both scores but retains the gap is not blocked; only candidate-introduced degradation is blocked.

2. Stochastic validity: each task/arm is now replayed dream_adversarial_rollouts times (distinct sample ids so caching cannot collapse the samples), scores aggregate by mean, and brittleness additionally requires the worsening to hold in a strict majority of paired rollout indices. Blocking mode requires at least two rollouts (config validation rejects blocking at one), so a single-sample fluctuation can never reject a candidate; a regression pins exactly that case. The rule is documented in the docstring and docs as a screening decision, not a significance test; the retained per-rollout samples can be pushed through the paired evalkit for full inference, and the docs now say to calibrate the margin from advisory evidence before enabling blocking.

3. _strip_polite_frame now has an explicit semantic-preservation contract: only a leading "please" or a politeness-marked modal request ("can/could/would you please ...") is reframed; bare modal questions are never touched, and the first-person desire pattern ("I need/want you to ...") is removed entirely. Negative tests cover ability, permission, desire, and question forms ("Can you swim?", "Would you like some tea?", "I want you to know this matters."), plus positive tests for the politeness-marked forms.

Validation at the exact head across five native runners (Ubuntu 3.10/3.11/3.12, macOS arm64, Windows): full suite 1,538 passed on Ubuntu and macOS and 1,493 on Windows with zero failures; the focused slice is 202 passed everywhere; strict docs pass.

On your closing point: we considered removing the blocking path entirely and keeping probes advisory-only. We kept blocking opt-in behind the three conditions you set (baseline-relative comparison, repeated-rollout consistency, conservative transformations), plus the rollout floor, and advisory remains the default. If you would still prefer advisory-only until the margin has been calibrated on a public scenario, we are happy to disable blocking in this PR and propose it separately with that calibration.

[Yifan Yang (Yif-Yang)]

Thanks for fixing baseline-relative comparison, repeated rollouts, and conservative request transformations. Re-reviewing 84bbde9ac191, the single-skill path now carries the rollout setting, but multi-skill fanout drops it.

run_sleep_cycle() forwards dream_adversarial_rollouts to its first dream_consolidate() call, but omits it from the consolidate_groups() call around cycle.py:1009-1040. Each group therefore uses the default of 1.

I reproduced this using the real cycle/group pipeline and the PR's deterministic robust backend, with:

multi_skill_fanout=True
dream_adversarial=1
dream_adversarial_blocking=True
dream_adversarial_rollouts=2

The ordinary consolidation runs, but the skill group is reported as status="failed" with ValueError: dream_adversarial_blocking requires dream_adversarial_rollouts >= 2. With advisory mode, the same omission silently reduces the requested sample count to one. The current fanout plumbing test disables adversarial execution inside its spy, so it does not catch this failure (the full shipped suite passes: 1541 passed, 9 skipped).

Please forward the rollout setting through the grouped call and per-group diagnostic config, and add a real multi-skill cycle regression with blocking enabled and at least two rollouts. This can be fully offline; no new paid-model experiment is necessary for this wiring fix.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Thanks for the precise reproduction. I fixed the multi-skill omission and added the real offline regression you requested at head 67b2f16441cf87f539f0769498f6a4ef6e05c8fb.

• run_sleep_cycle() now forwards dream_adversarial_rollouts through consolidate_groups(), so each group receives the configured sample count.
• Per-night diagnostics now retain the configured rollout count alongside the other adversarial settings.
• The new end-to-end regression runs a real multi-skill cycle with blocking enabled and two rollouts against the deterministic robust backend. It asserts that the routed skill group completes as consolidated and is accepted. Against pre-fix head f1466003e1dc, the same test fails because the group is reported as failed, reproducing the dropped rollout setting.
• The existing fanout propagation test now also pins the rollout argument, and the diagnostics/report test exercises and records two rollouts.

Proof at 67b2f16441cf87f539f0769498f6a4ef6e05c8fb: focused slice 48 passed; complete suite 1543 passed, 12 skipped, 353 subtests; strict documentation build and git diff --check passed.

No paid-provider run was added because this is the fully offline fanout wiring defect from your review. Ready for re-review.

[Yifan Yang (Yif-Yang)]

Re-reviewed 67b2f16441cf. The missing multi-skill forwarding is fixed: the independent real-cycle reproduction with blocking enabled and two rollouts now completes with a consolidated skill group. The adversarial selection including that independent regression passes 48 tests locally on Linux/Python 3.11.

This is the right kind of evidence for the wiring defect; I am not requesting a paid-provider run merely to verify argument propagation. Keeping the configured rollout count in diagnostics is also useful.

Please refresh the validation summary in the PR description: its multi-platform table still says every job asserted 84bbde9ac191..., which is an earlier head. Keep that historical evidence, but label it as such and separate it from the new 67b2f16441cf... receipt. Historical local/fork runs should not be read as official CI success for the final head.

The stated limitations on request-frame probes, proxy coverage, rollout cost, and opt-in blocking remain important. The focused propagation blocker is resolved; final review and the exact-head upstream CI gate remain. That CI run currently awaits maintainer approval, so this comment is not merge approval.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Thanks. The validation summary is refreshed; the head is unchanged at 67b2f16441cf, since this is a description-only edit.

• The multi-platform matrix is now under a Historical evidence heading scoped to 84bbde9ac191, the candidate you reviewed before the rollout-forwarding fix, and labeled as fork-runner evidence rather than official CI.
• A separate Current head section reports only 67b2f16441cf: complete suite 1,543 passed, 12 skipped, 353 subtests; adversarial selection (-k adversarial) 47 passed, 2 skipped. Both come from a single local Linux / Python 3.12 run, and the section says so rather than implying a matrix.
• I did not re-run the strict docs build on this head in that environment, so the description claims no docs result for 67b2f16441cf instead of carrying the old one forward.
• The section states plainly that upstream CI has not run on this head and that the workflow is action_required pending maintainer approval, so nothing there reads as official CI success.

The stated limitations on request-frame probes, proxy coverage, rollout cost, and opt-in blocking stand as written. Nothing else changed.

[Yifan Yang (Yif-Yang)]

Thank you for addressing the multi-skill rollout handling and documenting the limits of the probe evidence.

Re-reviewed 67b2f16441cf. The real multi-skill blocking regression now passes with two rollouts, and the validation description correctly separates historical evidence. Local results: 206 passed, 2 skipped in the adversarial/cycle/staging selection; 1,542 passed, 13 skipped in the full suite; strict docs pass.

One output mismatch remains at skillopt_sleep/cycle.py:409-411: the Markdown renderer still reads source_score, probe_score, and delta, but the evaluator now emits baseline/candidate scores and gap_change.

An independent real-cycle regression produces correct structured evidence:

baseline source/probe: 0 / 0
candidate source/probe: 1 / 0
gap_change: -1

Its report.md row is nevertheless:

| train | train_adversarial_request-frame | request-frame | — | — | — | brittle |

Please update the table columns and field names to expose the baseline-relative evidence, including gap_change, and assert the numeric row through the real cycle/report path. This is a rendering defect; the structured decision values are present. No paid-provider experiment is needed to verify this correction. Exact-head upstream CI remains pending approval.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Welcome back! Will do!

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Fixed the report mismatch in 4d93cad055f9210438867d02c8e7c24236902040, one new commit on the reviewed head.

The table now shows baseline source/probe, candidate source/probe, and gap change. Four real-cycle regression cases cover brittle and robust candidates in advisory and blocking mode, asserting both the structured evidence and the exact numeric row in report.md. All four fail before the rendering fix and pass after it.

python3 -m pytest tests/test_adversarial_dream.py -q -k cycle_report_renders_baseline_relative_probe_scores
67b2f16441cf + the new tests, before renderer fix: 4 failed, 47 deselected
4d93cad055f92, after renderer fix: 4 passed, 47 deselected

The reproduced brittle row now reads:

| Source | Probe | Variant | Baseline source | Baseline probe | Candidate source | Candidate probe | Gap change | Status |
| `train` | `train_adversarial_request-frame` | request-frame | 0.000 | 0.000 | 1.000 | 0.000 | -1.000 | brittle |

Validation on that exact head: 51 adversarial tests pass locally; the five fork jobs (Ubuntu Python 3.10/3.11/3.12, macOS arm64 3.12, Windows 3.12) pass the focused integration slice and full suite. The full suite reports 1,547 passed on Ubuntu/macOS and 1,502 passed on Windows, with platform/optional skips reported separately. Strict docs pass on all three Python 3.12 platforms.

The PR description now separates this receipt from the historical matrix. These are fork-runner results; upstream CI for this head still awaits repository approval. No paid-provider experiment was added for this rendering correction.

This rendering fix is ready for re-review. Upstream workflow approval and the final merge decision remain with the project.