fix(criteria): search the whole command in command_executed and bump harness SDKs for Opus 5.5

[bai-uipath]

Unblocks moving the claude-code nightly to Opus 5.5 (UiPath/skills#3514). Three changes:

1. command_executed searches the whole command

Before, the criterion only searched the first 2,000 characters of each command. Opus 5.5 often batches a task into one long Bash call and runs the checked command at the very end:

cat > agent.json <<'EOF'
{ ... 2,400 characters of agent definition ... }
EOF
uip agent refresh && uip agent validate

Criterion on the command above	Before	After
command_executed: uip agent validate	not found, task fails	found, task passes
max_count: 0 on a retired command at the end of a long script (same for exclude_pattern)	not seen, check passes	seen, check fails

The per-search length bound against ReDoS stays; the rest of the command is searched in bounded windows aligned to shell lines.

Only Bash commands are windowed. Other tools keep the leading 2,000 characters of their JSON params, so a long Write/Edit body is not read as a command.

Re-grading published runs:

Run	Fail → pass	Pass → fail
Opus 5.5 full suite (2026-09-23)	35	0
Sonnet 5 nightly (2026-09-23)	0	0

2. Harness SDKs and CLIs move to current releases

Package	Was (age on 2026-09-23)	Now	Why this version
Claude Code CLI (agent image)	2.1.177 (102 days)	2.1.281	the CLI claude-agent-sdk 0.2.159 bundles
claude-agent-sdk	0.2.124 (65 days)	0.2.159	latest
openai-codex	0.144.4 (68 days)	0.156.1	latest
openai-codex-cli-bin	0.144.4 (70 days)	0.156.1	latest
google-antigravity	0.1.8 (62 days)	0.1.18	latest
Pi CLI	0.84.4 (26 days)	0.87.1	latest
anthropic	1.0.0 (34 days)	1.8.0	latest
harbor	0.22.0 (32 days)	0.23.0	latest
litellm	1.98.0 (32 days)	1.102.0	newest past the 48h gate; 1.102.1 clears 2026-09-25
uipath	2.10.31 (182 days)	2.14.24	newest past the 48h gate; 2.14.25 clears 2026-09-25
openai	2.54.0	2.54.0	unchanged: litellm requires <3

What this fixes in practice:

• Cost. Claude Code computes the claude-code cost itself. The old CLI billed Sonnet 5 at Sonnet 4 rates ($3/$15) and Opus 5.5 at Opus 5 rates ($5/$25), so costs read about 1.5x high for Sonnet 5 and 1.65x high for Opus 5.5. 2.1.281 bills both at list: Opus 5.5 $4/$20, Sonnet 5 $2/$10.
• Reported CLI version. The SDK runs the CLI it bundles, not the image's. Before, Docker runs reported 2.1.177 while 2.1.216 actually ran. The image pin now equals the bundled version, so the reported version is the one that ran.

Pricing check on this branch, hello_date on Opus 5.5 over Bedrock (passes, score 1.0):

Tokens	Count	Opus 5.5 list, per MTok	Cost
Uncached input	6	$4.00	$0.0000240
Output	373	$20.00	$0.0074600
Cache write	22,262	$5.00	$0.1113100
Cache read	43,997	$0.20	$0.0087994
Total			$0.1275934, the exact cost Claude Code reported

3. claude-agent-sdk is exempt from CI's 48-hour package-age gate

• Why: new-model support ships in the Claude Code CLI bundled inside this SDK, faster than the gate allows. Example: Opus 5.5 pricing first shipped in Claude Code 2.1.280 on 2026-09-22. With the gate, CI could only install claude-agent-sdk 0.2.157, which bundles 2.1.277 and bills Opus 5.5 at a $5/$25 fallback (about 1.6x list).
• Precedent: openai-codex and its CLI binary are already exempt for the same reason. The trade-off is the same too: a new SDK release reaches CI without the 48-hour wait.
• Scope: PR checks, Harbor E2E, release and TestPyPI publish. The gate only runs on the CI host, not inside docker build, so image-only installs (Antigravity, Pi, the Claude Code npm package) were never gated.
• Downstream: the pyproject floors stay at claude-agent-sdk>=0.2.157 and openai-codex>=0.155.1. skills and coder_eval_uipath CI don't exempt these packages, so a higher floor would break their installs of this release until the new versions age in. The lockfile and the agent image still get the versions above.

🤖 Generated with Claude Code

[github-actions]

Claude finished @bai-uipath's task in 1m 25s —— View job

---

Code Review for PR #196: Fix command_executed criterion and update harness SDKs

Working on review...

Review Checklist:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Review the full PR diff
• Review each changed file in detail
• Check cross-file consistency
• Perform "what's missing" analysis
• Format and post final review

[uipreliga]

Review: coder_eval — pr:196 (branch vs origin/main, 14 files)

PR #196 — fix(criteria): search the whole command in command_executed and bump harness SDKs for Opus 5.5 (author: bai-uipath, base: main)

Change class: complex — changes the command_executed search algorithm (single leading window → multi-window logical-line packing), which alters verdicts of min_count/max_count/exclude_pattern gates, plus harness SDK / CLI pin bumps and CI safe-chain exclusions

The PR is in good shape: types, contracts and tests are clean, and no finding is critical or high. The real risk is that the new windowed whole-command search in command_executed.py can quietly give a different verdict past 2000 characters. Quoted calls inside bash -lc wrappers are missed, anchors match at window edges, and a match that crosses a window boundary is dropped. Fix the window semantics and add the user-facing migration note before merge.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Correctness & Logic	9.4 / 10	0	0	1	1	Regex anchors (^, $) match at window edges past the 2000-char cap, so anchored patterns mean something different in long commands
2. Code Quality & Simplicity	9.7 / 10	0	0	0	3	_search_windows builds packed windows from offset 0, then discards them and prepends a raw leading slice; the reason is not stated
3. Types & Contracts	10 / 10	0	0	0	0	—
4. Test & Validation Health	10 / 10	0	0	0	0	—
5. Security	9.9 / 10	0	0	0	1	Search-window fan-out makes regex work scale with agent-controlled command length, and the early-stop watcher repeats that work on every tool-call event
6. Architecture & Design	9.4 / 10	0	0	1	1	Fourth hand-synced copy of the SAFE_CHAIN exclusion expression, with no single source or parity check
7. Error Handling & Resilience	9.5 / 10	0	0	1	0	Quote normalization is off in later windows of a wrapped bash -lc command, and no test covers normalization of later windows
8. Interface, Docs & Compatibility	9.4 / 10	0	0	1	1	The user guide does not record that command_executed now changes verdicts past 2000 chars (max_count/exclude_pattern can flip pass to fail)

Overall Score: 9.7 / 10 · Weakest Axis: Correctness & Logic at 9.4 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 4 · 🔵 7 across 8 axes reviewed.

Blockers (0 🔴 Critical · 0 🟠 High)

None.

Non-blocking, but please consider before merge

1. [Axis 1] Regex anchors (^, $) match at window edges past the 2000-char cap, so anchored patterns mean something different in long commands (src/coder_eval/criteria/command_executed.py:131) — Keep the anchors bound to the real command. Search the raw text as pattern.search(cmd_text, s, e) for each window (s, e), not on the slice cmd_text[s:e]. Without MULTILINE, '^' then matches only at the true start of the string, and endpos still bounds the ReDoS work. That fixes '^' but not '$', because '$' still matches at endpos. To also bind '$', search only the last window (e == len(cmd_text)) with endpos, or put each window's end on a real line boundary and document that '$' can match there. Apply the same treatment to the normalized haystacks. They are new strings built per window, so either normalize only the window at offset 0 or document that anchors on normalized forms refer to the start and end of the window. Add tests that pin ^git$ (and a '$'-anchored pattern) against a command longer than 2000 characters whose bare 'git' comes after the cap, for both command_pattern and the exclude_pattern / max_count=0 gate. No shipped task under tasks/ or experiments/ uses an anchored pattern, so this is Medium.
2. [Axis 6] Fourth hand-synced copy of the SAFE_CHAIN exclusion expression, with no single source or parity check (.github/workflows/harbor-e2e.yml:33) — The same expression is now in four workflows (harbor-e2e.yml:33, pr-checks.yml:46, publish-testpypi.yml:41, release.yml:78), and nothing checks it. The follow-up commit af239b9 was needed because harbor-e2e had NO copy, not because two copies disagreed. So a guard must catch a missing copy as well as a different one. Two ways to fix this. (a) Move the whole value, including claude-agent-sdk, into one repo-level source, for example a repo variable vars.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS or a small composite action that exports it, and reference that source from every workflow. (b) Add a doc-surface lint class in tests/test_custom_lint.py with two checks. First, every workflow that runs uv sync/uv pip install on a uipath-* runner sets SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS, except workflows on an explicit exemption list such as docker-publish.yml. Second, each value is byte-identical to the pr-checks.yml value. If you keep the copies and add no check, put an entry in .claude/harness-candidates.md.
3. [Axis 7] Quote normalization is off in later windows of a wrapped bash -lc command, and no test covers normalization of later windows (src/coder_eval/criteria/command_executed.py:153) — _match_haystacks now calls normalized = _normalize_shell(window) on each window from _search_windows. Only the first window (cmd_text[:cap]) holds the wrapper's opening quote. A later window starts on a line boundary inside the -lc script, so it holds only the closing quote. shlex.split then raises ValueError, _normalize_shell returns None without any log, and that window keeps only its raw haystack. Codex commands arrive in this wrapped shape (tests/test_command_executed.py:810 uses '/bin/bash -lc "uip is resources run list ...'). I reproduced this with the local .venv. With pattern uip\s+or\s+users\s+list, the 30-char bash -lc 'uip or "users" list' matches (True). The same command placed after a 60-line heredoc inside the same wrapper (4301 chars) does not match (False). The per-window normalize results for the matching case were [(2000, None), (1988, ok), (389, None)], and the last window is the one that holds the command. So a quoted retired call in a max_count: 0 gate, or a quoted exclude_pattern token, now slips through past the bound. It is caught when the command is short, and when the command is not wrapped (the unwrapped heredoc variant matched True). This is the dodge that test_negative_assertion_not_dodged_by_quoting guards for short commands. It is not a regression against main, which never searched past 2000 chars. But the notes now say 'the whole command is searched', and the degradation is silent. Fix: unwrap the -c/-lc wrapper once on the full text, strip the outer quote, then window the inner script. Another fix: when a later window's shlex fails, retry it with the wrapper's quote character prepended. Add a TestSearchWindows case that places a quoted command after a long heredoc inside bash -lc '...', for both max_count: 0 and exclude_pattern. Also consider a logger.debug when a window fails to normalize, so a missed quoted match can be diagnosed.
4. [Axis 8] The user guide does not record that command_executed now changes verdicts past 2000 chars (max_count/exclude_pattern can flip pass to fail) (src/coder_eval/criteria/command_executed.py:105) — _search_windows (line 105) makes command_pattern, exclude_pattern and the max_count gate see the whole command. Before, they saw only cmd_text[:_MAX_PATTERN_SEARCH_LEN]. The PR body says this change is not purely additive: "max_count: 0 on a retired command at the end of a long script ... not seen, check passes -> seen, check fails". The only place in the repo that records this is .claude/notes/contracts.md, which is not auto-loaded and is not a user doc. docs/TASK_DEFINITION_GUIDE.md § command_executed (lines 976-1000) already has a rule for this case. For shell normalization it says "normalization is not purely additive ... Cross-repo suites that hand-encoded quote tolerance in their patterns should re-baseline." This change gets no matching sentence. Add one line to that section: the whole command is searched in bounded windows, so an exclusion or a max_count gate can now catch a call late in a long heredoc script, and suites should re-baseline. The generated plugin reference (make plugin-reference) takes its text from the model Field descriptions (src/coder_eval/models/criteria.py:888 onward), and those also do not mention it. Verified with grep -n -i '2000\|truncat' docs/TASK_DEFINITION_GUIDE.md plugins/coder-eval/reference/criteria.md: no hit for command_executed. Kept at Medium because the old 2000-char cap was not documented either. The problem is the missing migration note for a change that flips verdicts, not a doc that is now wrong.

Nits

1. [Axis 1] Packed windows do not overlap, so a multi-line or DOTALL match across a window boundary is silently missed and no test pins it (src/coder_eval/criteria/command_executed.py:123) — _WINDOW_OVERLAP applies only when one logical line is longer than the cap is chopped. When lines are packed and flushed at bounds.append((start, end)) (line 123), the next window starts exactly at end and nothing overlaps. A DOTALL pattern that spans two short lines, which the criterion's own comment says is intended ('foo.*--body must span them'), matches the same text inside the first 2000 chars but misses it later. Reproduced: body (1998 chars) + 32 filler lines + "git commit -m x\n" + "git push --force origin main\n" gives windows [2000, 1997, 29], with the commit line ending window 2 and the push line starting window 3. git commit.*git push --force returns False, but it returns True for the same three lines in a short command. With max_count: 0 this is a false pass of a forbidden-sequence gate, and on exclude_pattern it is a missed exclusion. This is not a regression, because pre-PR that text was never searched. But .claude/notes/contracts.md line 99 now says 'the whole command is searched'. Fix: start each packed window a few lines (up to _WINDOW_OVERLAP chars) before the previous window's end, or reword the note and docstring to say that a match that crosses a window boundary is not guaranteed. A related packing quirk: _LOGICAL_LINE_END = re.compile(r"(?<!\\)\n") (line 28) treats \\\n (an escaped backslash followed by a real newline) as a continuation, and it splits a CRLF continuation \\\r\n. Both only move boundaries, so they matter only through this cross-boundary gap.
2. [Axis 2] _search_windows builds packed windows from offset 0, then discards them and prepends a raw leading slice; the reason is not stated (src/coder_eval/criteria/command_executed.py:131) — return [cmd_text[:cap]] + [cmd_text[s:e] for s, e in bounds if s > 0] gives the function two modes. The packing loop runs from start = end = 0, then if s > 0 drops the first packed window and puts the mid-line cut cmd_text[:cap] in its place. So the characters between the first line boundary and cap are searched twice, and a reader cannot tell from the code whether the special first window matters. Neither the docstring (lines 106-114) nor contracts.md says why the first window is different. The likely reason is to keep every verdict the old single-window code gave, including matches in the normalized form that cross a line boundary. Either state that in one line (in .claude/notes/contracts.md, per the docstring-is-contract rule), or remove the special case: return [cmd_text[s:e] for s, e in bounds]. The one-pass version is also better for _normalize_shell, because every window then starts and ends on a line boundary, and a cut through a quoted string makes shlex fail.
3. [Axis 2] New module comments repeat what the constant names and the _search_windows docstring already say (src/coder_eval/criteria/command_executed.py:24) — Lines 24-25 (# Overlap between the pieces of one logical line longer than the bound, so a match up to this long is never split across two pieces.) and line 27 (# A newline not escaped by a trailing backslash: where one logical shell line ends.) say the same thing as the names _WINDOW_OVERLAP and _LOGICAL_LINE_END. The _search_windows docstring (line 110 'a backslash-continued line stays whole', line 112 'overlapping by _WINDOW_OVERLAP') says it again. CLAUDE.md says 'default to ZERO comments'. Delete the comment on line 27. Cut the one on lines 24-25 to the part the code cannot show, which is the guarantee 'a match of up to 256 chars is never split', or move that guarantee into the docstring.
4. [Axis 2] Rewritten antigravity workspace test duplicates the test above it (tests/test_antigravity_agent.py:150) — The old test called the real google.antigravity workspace_only policy. The rewrite test_resolved_workspaces_cover_skill_reads now only asserts any(skill_md.is_relative_to(w) for w in workspaces). That is already implied by test_resolve_workspaces_includes_workdir_and_skill_roots (line 137), which asserts _resolve_workspaces(...) == [str(tmp_path / "work"), str((repo / "skills").resolve())]: SKILL.md is under repo/skills by construction. The other assert, assert not skill_md.is_relative_to(workdir.resolve()), only checks the fixture. Following 'delete before you guard', delete the new test, or move its docstring (why skill roots must be in workspaces) onto the line-137 test. With it gone, the new from pathlib import Path import is unused and can go too.
5. [Axis 5] Search-window fan-out makes regex work scale with agent-controlled command length, and the early-stop watcher repeats that work on every tool-call event (src/coder_eval/criteria/command_executed.py:150) — Before this PR a command cost at most one 2000-char window. Now for window in _search_windows(cmd_text): (line 150) yields about len/1744 windows for a long single line, plus one normalized haystack per window. The work in each window is still bounded, so this is not a ReDoS regression. But the total cost is now (command length) x (pattern cost per window). live_verdict (line 245 onward) re-runs _matching_commands over the whole accumulated trajectory, and the regex search is not memoized (only _normalize_shell has @lru_cache, line 56). Measured on the local venv: a 5 MB one-line command ("a 'b' " * 833333) gives 3823 haystacks. A super-linear task-authored pattern (a|b|\s)*z took 17.2 s for one full scan, and early stop repeats that scan on each tool-call event. A benign literal pattern took 0.003 s. The patterns are task-authored and therefore trusted, and the agent controls only the length, so the impact is a slow evaluation, not a security boundary crossing. Fix: add a total cap on searched characters per command (for example, the leading window plus the trailing N windows, or a total of about 64 KB), or memoize the per-command match result across early-stop re-scans. Document the cap in .claude/notes/contracts.md next to the new windowing paragraph. CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:N/I:N/A:L
6. [Axis 6] The PR bumps the Pi CLI pin, but the Pi parser vocabulary and docstrings still say they were captured from 0.84.4 (docker/Dockerfile:42) — docker/Dockerfile:42 changes ARG PI_VERSION=0.84.4 to ARG PI_VERSION=0.87.1. The stream reducer this binary feeds still says it was built against 0.84.4: src/coder_eval/agents/pi_agent.py:8 (Three grammar facts that are not obvious from the event names (``pi`` 0.84.4)), pi_agent.py:152 (# The full recognized Pi vocabulary (from pi 0.84.4)., above _RECOGNIZED_EVENTS) and models/agent_config.py:365 (Confirmed against ``pi --help`` on Pi 0.84.4). A clean exit that recognizes no event in _RECOGNIZED_EVENTS is scored as a crash, so vocabulary drift in 0.87.1 would change docker-driver Pi results. I did not check 0.87.1's event names, which is why this is Low. The Claude Code pin has a parity test (tests/test_image_from_dockerfiles.py). The Pi pin is only checked for being an exact version. To fix, confirm the 0.87.1 event grammar against a fresh capture, then update the version cited in pi_agent.py and agent_config.py. If the grammar changed, refresh the tests/test_pi_agent.py fixture. The other option is to keep PI_VERSION at 0.84.4 in this dependency-bump PR.
7. [Axis 8] docs/agents/ANTIGRAVITY.md still says google-antigravity is pinned to 0.1.8 after the bump to 0.1.18 (pyproject.toml:134) — This PR changes the extra to "google-antigravity==0.1.18", (pyproject.toml:134). docs/agents/ANTIGRAVITY.md:30 still says "This pulls in google-antigravity (pinned to 0.1.8)". Update that doc line in the same change. Checked with grep -rn '0\.1\.8' docs/: this is the only stale pin. The other bumped pins (2.1.177, 0.144.4, 0.22.0, PI_VERSION 0.84.4) have no stale mentions in docs/, README.md, plugins/ or .claude/notes.

What's Missing

Parallel paths & mirrors:

• 🔵 The google-antigravity 0.1.18 bump changed the workspace policy, but src/coder_eval/agents/antigravity_agent.py still describes the old mechanism. I checked the 0.1.18 wheel. hooks/policy.py:534 workspace_only() now runs del workspaces and returns marker deny policies with no when predicate. local_connection_config.py no longer adds workspace_only in front of the other policies (no prepend logic is left). The path check now runs inside localharness. The rewritten test docstring in tests/test_antigravity_agent.py says this. The agent does not: antigravity_agent.py:278 (_resolve_workspaces: "Workspace roots for the harness's workspace_only file-tool policy") and :344-345 ("File tools are confined to workspaces by the auto-prepended workspace_only policy") still describe the 0.1.8 Python-side policy. Change both so they say the harness enforces workspaces itself. (trigger: pyproject.toml) (restates: Axis 2: Rewritten antigravity workspace test duplicates the test above it)
• 🔵 harbor 0.22.0 -> 0.23.0 changed the ATIF models, but the vendored copy and its validation record did not follow. I diffed the two wheels. harbor 0.23.0 adds ATIF-v1.8 to Trajectory.schema_version, and it adds an audio content part (ContentPart.type: Literal['text','image','audio'], source: ImageSource | AudioSource). src/coder_eval/harbor/atif_models.py:39 still has type: Literal['text','image'] under extra='forbid'. Its docstring says the image variant exists "so ATIF documents produced by other agents still parse", but a v1.8 audio part from a harbor 0.23 producer is now rejected. The pin references are also stale. tests/test_atif_models.py:5-14 says "pinned to 0.22.0" and "Last validated: harbor 0.22.0". atif_models.py:70 and packager.py:39 (_HARBOR_SCHEMA_VERSION comment) still cite harbor 0.22.0. Run the re-validation steps in tests/test_atif_models.py against 0.23.0, then either add the audio variant or document that it is not supported, and update the version references. (trigger: pyproject.toml)

Tests:

• 🔵 Since 0.1.18, no test checks that workspaces plus policies=[policy.allow_all()] still lets the agent read SKILL.md from a skill root outside the workdir. The old test called the SDK's real workspace_only predicate. In 0.1.18 that predicate is gone (the path check runs in the localharness binary), and the rewrite checks only list membership, which the test at line 137 already covers. The one path left that can fail this way is a skill read denied by the harness. Add a gated live or integration check (skipped when the extra or GEMINI_API_KEY is absent) that does one skill read under 0.1.18, or record in docs/agents/ANTIGRAVITY.md that this path has no test. (trigger: tests/test_antigravity_agent.py) (restates: Axis 2: Rewritten antigravity workspace test duplicates the test above it)
• 🔵 The PR says "The SDK runs the CLI it bundles, not the image's ... The image pin now equals the bundled version". Nothing enforces that. tests/test_image_from_dockerfiles.py:463 checks only that docker/Dockerfile and docker/Dockerfile.runtime agree with each other. No test compares CLAUDE_CODE_VERSION with claude_agent_sdk._cli_version.__cli_version__ (2.1.281 for the locked 0.2.159). So the next SDK bump can again make Docker runs report one CLI version while another one runs, which is the 2.1.177-vs-2.1.216 case this PR fixes. The pyproject floor >=0.2.157 bundles 2.1.277, so a downstream install at the floor already differs from the image label. Add a test that compares the installed SDK's bundled CLI version with the Dockerfile ARG. Skip it when the SDK is absent. (trigger: docker/Dockerfile)

Rollout impact:

• 🔵 The release note will not show the effect on scores and costs. The repo uses squash merges, and semantic-release builds CHANGELOG.md from the PR title (fix(criteria): ...), so the release gets one patch line. That line will not say that the Claude Code CLI 2.1.177 -> 2.1.281 bump lowers claude-code costs by about 1.5-1.65x for Sonnet 5 / Opus 5.5, and that the codex, antigravity and Pi harnesses changed too. docs/agents/CLAUDE_CODE.md:147 calls a CLAUDE_CODE_VERSION bump "a score-affecting change", and the PR shows 35 fail-to-pass flips. The PR body explains this, but dashboards and the evalboard that compare cost and pass rate across the release see only the changelog. Put a short summary of the harness bumps and the cost and verdict shift in the squash commit body or in a BREAKING CHANGE/notes footer, so it reaches CHANGELOG.md. (trigger: pyproject.toml)

Guardrails & Automation

Static checks (lint / type / repo-local):

• [repo-local-check] CE067 SAFE_CHAIN exclusion parity. Add a doc-surface lint class TestCE067SafeChainExclusionParity to tests/test_custom_lint.py, with its logic in a new module tests/lint/safe_chain_parity.py. Follow the CE035 pattern (tests/lint/workflow_outputs.py): the module docstring states the invariant, the scope and the blind spots, and has a Rationale: .claude/notes/lint-rules.md § CE067 pointer. The rule walks .github/workflows/*.yml. For every job whose runs-on is a uipath-* runner and that has a step running uv sync or uv pip install, it has two checks. (1) Presence: the job or step env must set SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS. (2) Parity: the value must be byte-identical to the value in pr-checks.yml, which is the reference copy. An EXEMPT dict maps a workflow file to its reason, for example docker-publish.yml: it leaves the variable out on purpose, as its comment at line 49 says. A stale EXEMPT entry, meaning a workflow that no longer exists or no longer installs packages, is also a finding. Record the defect in .claude/notes/lint-rules.md. The alternative that deletes the pattern instead of guarding it is one repo variable or a composite action that exports the value. If the team takes that fix, the check (2) is not needed, but check (1) still applies. Prevents: Finding 7 (fourth hand-synced copy of the SAFE_CHAIN exclusion, at harbor-e2e.yml:33, pr-checks.yml:46, publish-testpypi.yml:41 and release.yml:78). The presence check also catches the missing copy that af239b9 had to fix: harbor-e2e.yml had no exclusion list, so it blocked claude-agent-sdk 0.2.159. A parity-only check would not have caught that.
• [repo-local-check] CE068 pin-citation parity. Add a lint class TestCE068PinCitationParity in tests/test_custom_lint.py, backed by a table PIN_CITATIONS in a new tests/lint/pin_citations.py. Each key is a pin source: an exact == pin in pyproject.toml (read with tomllib) or an ARG <NAME>_VERSION= in docker/Dockerfile, parsed the same way tests/test_image_from_dockerfiles.py:500 already parses PI_VERSION. Each value is the list of files that cite that version in prose, plus a regex that finds the cited version. Seed entries: google-antigravity -> docs/agents/ANTIGRAVITY.md (pattern google-antigravity.*?pinned to .?(\d+\.\d+\.\d+)). PI_VERSION -> src/coder_eval/agents/pi_agent.py (both the module docstring and the _RECOGNIZED_EVENTS comment, pattern pi (\d+.\d+.\d+)``), src/coder_eval/models/agent_config.py (Pi (\d+\.\d+\.\d+)), and the tests/test_pi_agent.py capture fixture's recorded version, if it has one. The rule fails when a cited version differs from the pin. A second assert fails when a cited file no longer matches its regex, so a citation cannot drop out of the table without notice. Put the rationale in .claude/notes/lint-rules.md: the rule cannot verify that Pi 0.87.1 kept the event grammar. It only makes a pin bump touch every place that says 'verified against X', so the person who bumps the pin must re-verify or revert the bump. Scope and blind spot for the docstring: it covers only the citations in the table, and a new prose citation needs a new entry. Prevents: Finding 11 (docs/agents/ANTIGRAVITY.md:30 still says google-antigravity is pinned to 0.1.8 after the bump to 0.1.18). Finding 8 (the PI_VERSION bump to 0.87.1 in docker/Dockerfile:42 while pi_agent.py:8, pi_agent.py:152 and agent_config.py:365 still cite 0.84.4). The check fails the same PR that bumps the pin, and the Pi event grammar re-verification is then an explicit step.

Guardrail improvements (not statically reachable):

• Add a metamorphic parity test for windowed search to tests/test_command_executed.py, under TestSearchWindows. The test has a fixed corpus of (pattern, command, expected verdict) cases that are correct on a short command. For each case, it builds long variants that put the command after a filler prefix: a heredoc body, packed short lines, and a single long line. The prefix lengths are chosen so the command lands at each window boundary shape: the first window at offset 0, just past _MAX_PATTERN_SEARCH_LEN, exactly on a packed-window flush, and inside the overlap of a chopped long line. The test asserts that the long variant gives the same verdict as the short one, through command_pattern, exclude_pattern and the max_count: 0 gate. The corpus must contain these cases: ^git$ and a $-anchored pattern after a long heredoc (finding 1), git commit.*git push --force across two short lines at a packed boundary (finding 2), and uip\s+or\s+users\s+list against bash -lc '<long heredoc>\nuip or "users" list' (finding 9). A known limitation that is kept on purpose goes in as an expected-divergence case with a comment that points to .claude/notes/contracts.md. That way the notes' claim that 'the whole command is searched' is checked by a test and not only written in prose. Why not static: The defects are about regex semantics over windows and shlex-normalized strings that are built at runtime: ^/$ binding at slice edges, DOTALL spans across non-overlapping windows, and a wrapper quote that is lost per window. Each line of code is correct on its own, so no AST or linter rule can see that a haystack slice changes what a pattern means. Only running the matcher can show it. Prevents: Finding 1 (anchors match at window edges), finding 2 (no overlap between packed windows gives false passes of a forbidden-sequence gate), finding 9 (per-window quote normalization fails in later windows of bash -lc), and the related test-gap findings grouped under them (tests/test_command_executed.py:1113, :1143). It would also keep finding 3's special first window honest, because removing that window must not change any verdict in the corpus.
• Add a cost-bound test for command_executed. Assert a hard upper bound on the number of haystacks that _search_windows and _match_haystacks make for one command, for example len(list(_search_windows('a ' * 2_500_000))) <= N, with N taken from a named cap constant. Also assert that a live_verdict re-scan over a trajectory that already holds a very long command does not repeat the per-command regex work, either through memoization or a total searched-character cap. Assert on counts, not on wall-clock time, so the test is deterministic. Record the cap in .claude/notes/contracts.md beside the windowing paragraph. Why not static: The defect is total work that scales with the length of agent-controlled input and repeats on each early-stop event. That is a runtime property of the data volume and the call frequency, not a code pattern. The code has no unbounded-loop construct that a rule could flag: every window is bounded, and only the number of windows is not. Prevents: Finding 6 (search-window fan-out makes regex work scale with command length, and live_verdict repeats it on every tool-call event).
• Add a process guardrail for criterion changes that flip verdicts. Put an entry in .claude/harness-candidates.md, and a checklist line in the PR template or in the /coder-eval-code-review rubric: 'If this change can move an existing task from pass to fail or from fail to pass for the same trajectory, add a "not purely additive ... re-baseline" sentence to the criterion's section in docs/TASK_DEFINITION_GUIDE.md, and to the model Field description so that make plugin-reference carries it.' docs/TASK_DEFINITION_GUIDE.md:998, the shell-normalization paragraph, is the model. Do not promote it to a CE rule. Why not static: Only semantic judgment can tell whether a change alters verdicts. A rule of the form 'a diff to criteria/X.py must touch the X section of the guide' would fire on every refactor, and it would still not check that the right sentence was added. So the boundary is a recorded decision, not an omission. Prevents: Finding 10 (the user guide and the plugin reference do not say that command_executed now searches past 2000 chars, so max_count and exclude_pattern gates can flip).
• No mechanical guard for the review-quality findings. The redundant comments at command_executed.py:24-27 and the duplicate test at tests/test_antigravity_agent.py:150 stay review-time checks. make docs-budget already caps comment volume and run length, and it cannot judge whether a comment repeats a name. A 'unique coverage per test' check through pytest-cov dynamic contexts could flag tests that cover no line of their own, but assertion-only tests like the one at line 150 cover the same lines as their neighbours by design, so the signal would be mostly false positives. If the redundant test is deleted, ruff F401 already removes the unused from pathlib import Path that is left over. Why not static: To decide whether a comment says more than the identifier, or whether one test's assertions are implied by another's, needs semantic judgment about meaning and logical implication. No available linter rule or repo-local AST check can decide this without heavy false positives. Prevents: Finding 4 (module comments repeat constant names and docstring), finding 5 (rewritten antigravity workspace test duplicates the line-137 test), finding 3 (the unexplained special first window in _search_windows is covered only indirectly, by the metamorphic test above).

Top 5 Priority Actions

1. Fix the per-window shell normalization at src/coder_eval/criteria/command_executed.py:153. Unwrap the bash -c/-lc wrapper once on the full text and then window the inner script, or retry a failed window with the wrapper's quote prepended. Add tests for both max_count: 0 and exclude_pattern with a quoted call after a long heredoc, because today that call slips past a forbidden-command gate with no log.
2. Bind the window edges to the real command at src/coder_eval/criteria/command_executed.py:123 and :131. Today ^ and $ match at every window edge, and packed windows do not overlap, so an anchored or DOTALL pattern can give a false hit or a false pass past 2000 characters. Overlap the packed windows by up to _WINDOW_OVERLAP, search with pos/endpos, and state the remaining $ and normalized-haystack limits (or drop the special first window). Pin all of this with tests for commands longer than 2000 characters.
3. Confirm the Pi 0.87.1 event grammar against a fresh capture before bumping docker/Dockerfile:42. If it changed, update _RECOGNIZED_EVENTS and the cited version in src/coder_eval/agents/pi_agent.py:8/:152 and models/agent_config.py:365, and refresh the tests/test_pi_agent.py fixture. The other option is to keep PI_VERSION at 0.84.4, because a clean exit with no recognized event is scored as a crash.
4. Add a re-baseline note to the command_executed section of docs/TASK_DEFINITION_GUIDE.md (next to line 998) and to the model Field descriptions (src/coder_eval/models/criteria.py:888 onward, then make plugin-reference). The note must say that the whole command is now searched, so exclude_pattern and max_count can flip a pass to a fail. In the same change, fix the stale google-antigravity 0.1.8 pin in docs/agents/ANTIGRAVITY.md:30 (pyproject.toml:134 now pins 0.1.18).
5. Remove the fourth hand-synced SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS copy (.github/workflows/harbor-e2e.yml:33, pr-checks.yml:46, publish-testpypi.yml:41, release.yml:78). Use one repo-level source, or add a doc-surface lint that catches both a missing copy and a copy that differs, and also cap the total searched characters per command in command_executed.py:150 so that early-stop re-scans cannot grow with agent-controlled command length.

---

Change class: complex — changes the command_executed search algorithm (single leading window → multi-window logical-line packing), which alters verdicts of min_count/max_count/exclude_pattern gates, plus harness SDK / CLI pin bumps and CI safe-chain exclusions

Stats: 0 🔴 · 0 🟠 · 4 🟡 · 7 🔵 across 8 axes reviewed.
Verification: 8 medium+ finding(s) adversarially re-checked · 1 dropped as false positives (13%) · 6 corrected in place · 9 low passed through unverified.

[uipreliga]

Fix what you agree with and 🚢