From d703a8be939aa7a264f417eaae37c93ea9e90962 Mon Sep 17 00:00:00 2001 From: Tom Main Date: Sat, 26 Sep 2026 12:10:43 -0400 Subject: [PATCH 001/283] Status audit of afe6d94: offline probes reproduce five contract breaks (tab rebinding across commands, field receipt upgrading a refusal and confirming by substring, secret in a non-password receipt, browser judge failure continuing, short-document replacement unpaused); measured-status recompute and documentation corrections Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01HiwkrvCf1sUXw4QGgEe1Un --- .../results/status-audit-20260926/README.md | 101 ++++++++ .../status-audit-20260926/metrics.json | 241 ++++++++++++++++++ .../results/status-audit-20260926/probes.json | 39 +++ .../results/status-audit-20260926/probes.py | 115 +++++++++ .../status-audit-20260926/recompute.py | 42 +++ 5 files changed, 538 insertions(+) create mode 100644 jevdual/results/status-audit-20260926/README.md create mode 100644 jevdual/results/status-audit-20260926/metrics.json create mode 100644 jevdual/results/status-audit-20260926/probes.json create mode 100644 jevdual/results/status-audit-20260926/probes.py create mode 100644 jevdual/results/status-audit-20260926/recompute.py diff --git a/jevdual/results/status-audit-20260926/README.md b/jevdual/results/status-audit-20260926/README.md new file mode 100644 index 0000000..d207a15 --- /dev/null +++ b/jevdual/results/status-audit-20260926/README.md @@ -0,0 +1,101 @@ +# Footwork status audit — 2026-09-26 + +Footwork has a working research harness and useful supervised computer-use primitives. The best-supported default remains System 2 behind the guard. The operator surface is a substantial advance, but its target identity, receipt semantics and secret handling need correction before broader use. System 1 reduces model requests on some tasks; it has not established better total cost or completion than guarded execution. + +This review inspected the supplied handoff, current implementation, recent commit history, experiment definitions, stored result rows, annotation records, coexistence evidence, tests and GitHub CI. It did not run paid inference, access credential files, launch a browser or operate native apps. Only this audit directory was added; product code and original results were not edited. + +**Repository and verification.** Local main and GitHub main matched `afe6d9493dc7d067ee67838b90650d59c902df17` at inspection, and the worktree was initially clean. [CI for that revision](https://github.com/Tom-R-Main/Footwork/actions/runs/36252177173) succeeded. Local `.venv/bin/python -m pytest -q -m 'not browser and not live' tests` returned **473 passed, 21 skipped, 14 deselected in 14.52 s**; Ruff passed. Linux CI runs lint, Rust checks and offline Python tests; it does not establish live macOS or browser behavior. The fresh probes below exercise production functions with fake drivers, never the desktop. + +**What is now implemented.** + +| Area | Current state | Evidence boundary | +|---|---|---| +| browser-use autonomous loop | Guard, verifier, ledger, optional reactive S1 and experimental delegation; action receipts can be recorded or delivered | Most quantitative browser results come from this path, using browser-use 0.13.10 | +| Native autonomous loop | Cua accessibility menus, S1/S2, shared native authorizer, exact append, live target checks, execution posture | Q10b: six tasks, Calculator and TextEdit, three repetitions per arm; later operator fixes are not a new Q10b measurement | +| Operator CLI | `start`, `look`, `do`, `s1`, `done`, status, persisted observation and history | Useful supervised interface; `s1 --act` dispatches a proposal via `do`, not the autonomous arbiter policy | +| Operator browser mode | Footwork-owned Chrome profile, Cua DevTools route, label addressing, DOM click by default | Separate adapter from browser-use; targeted live examples and Q15 C3, not a paired replacement-backend benchmark | +| Machine coexistence | Background-only default, explicit foreground permission, idle check, hand-back, conflicting-append refusal | Q15 passes within its stated app, action and timing limits | +| Engineering safeguards | Pinned browser-use and cua-driver versions, seam tests, CLI scenario tests, Rust/Python equality tests | API shape and simulated behavior are covered better than before; gaps below survive the green suite | +| Cross-surface workflow | Planned in Q13 | No implemented shared multi-surface session or measured Q13 result located | + +There are three execution paths, with shared types and some shared logic, rather than one fully conforming execution contract. `footwork start --browser` uses the Cua browser adapter. `footwork run` builds a native `NativeAgent`; it has no corresponding `--browser` switch or guarded-only arm flag. Despite its docstring calling the default “guarded,” it enables S1 and escalates to S2: that is reactive dual in the experiment vocabulary. Paused autonomous runs currently print a rerun instruction; this is distinct from resuming the same run and ledger. The manual session does retain state across commands. + +**Measured status, recomputed from retained rows.** Costs are the recorded model-cost estimates, not invoices or total operating costs. Repeated runs are not independent new task types. Historical experiment rows do not certify the current revision. + +| Measurement | Results | Decision supported | +|---|---|---| +| Q9e repaired delegation, 59 tasks | Delegate 36 verified, $0.644; earlier same-day guarded 41, $0.424; dual 44, $0.496. Delegate reached 3/27 assignments | Keep delegation experimental. Baselines were not interleaved with this repaired run and had model timeouts | +| Q10b second native run, 18 trials/arm | S1-only 9 verified, $0.01385; dual 13, $0.03217; guarded 15, $0.02495 | Native improved; guarded still wins dual on both completion and total cost | +| Q11, 177 trials/arm | Guarded 126 verified / 821 requests; declared 126 / 993; legible 129 / 974 | Broad declaration and typed rejection feedback were not adopted | +| Q12, 177 trials/arm | Guarded 121 verified / 798 requests / 190 no-op receipts / $1.1804; receipt arm 117 / 761 / 156 / $1.1171; both 28 pauses | Receipts reduced retries and raw requests, without a demonstrated primary-metric win | +| Q15 | 15/15 accepted cases; additional corrected C3 attribution run 3/3 | Support the tested background behaviors; preserve scope and concurrency limitations | + +Q10b dual uses 44 LLM calls against guarded's 130, but about 29% more total model cost and about 49% more cost per verified completion. Its remaining two-trial deficit is on calc-percent. Every arm still failed to verify the append-and-save task in this measured run; subsequent supervised TextEdit save/readback success is useful evidence, not a rerun of the autonomous comparison. Do not mix Q10 and Q10b denominators: tasks and requirements changed. + +Q12's primary difference is **−0.02 requests per progress unit, 95% interval [−0.14, +0.10]**. The four fewer verified completions also breach its preregistered −2 guardrail, even though the paired interval includes zero. Its correct conclusion is non-adoption as a burden improvement. The fact that failed runs became cheaper is not task success. + +There is an additional metric-description issue: `evals.burden.progress_rows` includes every row carrying `checkpoints_reached`; the runner sets that field even for tasks without declared checkpoints. Only 19/59 tasks declare nonempty checkpoints. The reported 53-task comparison is the common set with positive mean `(checkpoints reached + predicate pass)` in both arms, not 53 tasks with declared checkpoints. Six tasks are excluded for zero progress in at least one arm. The arithmetic reproduces, but its label and population need to be explicit. Keep all-task completion and zero-progress counts beside this conditional ratio. + +**Earlier audit findings have received meaningful fixes.** Native routes now enter `Authorizer`; failed native judgments default to a pause. Append inserts against exact AX values and requires exact readback, instead of rebuilding documents from model previews. The append predicate checks preserved content. Annotation protocol 2 carries display state and prior effects while hiding scores. The operator's input posture and compound-action target handling have dedicated tests. These are substantive changes, not merely prompt adjustments. + +The corrections are not complete across all callers. The new probes in [probes.py](probes.py), with output in [probes.json](probes.json), establish the following current defects: + +1. **Browser session identity changes across commands.** `session.SessionState` persists PID and window ID, not browser target/tab identity. `session.bind` creates a new bridge each command; `BrowserBridge.attach` selects whichever tab is active. A fake driver changing active tab from A to B causes consecutive binds and clicks to target A then B under the same session. Matching labels on the other tab defeat the label-only check. C3 keeps one bridge bound, so that successful live test does not cover reconnecting CLI commands. Persist the bound tab/target and refuse if missing; never silently replace it with the active tab. Relevant source: `session.py:40`, `session.py:192`, `browser_driver.py:291`. + +2. **Text receipts can assert the wrong outcome.** `_field_receipt` confirms `type` by substring containment: requesting `Ada` confirms a field holding `Ada Lovelace`. An append confirms if the intended text was already present, without proving insertion or preservation. It also replaces an existing `refused` receipt with `confirmed` when that substring is present. The probe returns `confirmed` while retaining error `requires_foreground`. It selects the first same-label field, or falls back to a numeric ID, so identity is weak here too. Preserve dispatch/refusal status, require action-specific postconditions, and bind before/after evidence to the same field. Relevant source: `session.py:137–164`. Native exact-append readback is stronger than the operator receipt that summarizes it; retain that evidence rather than weakening it. + +3. **Named secrets can enter receipt output.** For a non-password field and `pending.secret=True`, `_field_receipt` sets `shown` to a placeholder but leaves `holds` as the actual value and interpolates it into evidence. A synthetic marker survives into the receipt. `observe` logs and returns the receipt, and the menu renderer also prints ordinary field values. No real secrets were read or exposed in this audit. Secret classification must travel with the field through observation, persistence, receipts and model inputs, independent of whether the UI uses a password control. Relevant source: `session.py:155–162`, `session.py:168`, `session.py:216`. + +4. **Browser judgment failures still allow continuation.** `DualProcessAgent._destructive_judgment` catches a judge exception and returns no pause. The fresh fake-judge probe reproduces this known P2 gap. Native's corrected behavior does not protect browser-use. Missing/unavailable required judgment and malformed output need an explicit conservative outcome, consistently across surfaces. Relevant source: `agent.py:247–273`. + +5. **Short document replacement is still allowed by the native gate.** `REPLACE_MIN_CHARS=20` exempts a 12-character `First line.\n` document. Proposing `type "Second line."` receives no pause under the real Authorizer. The native bridge's type branch writes the whole value. No real write was performed. Correct append usage prevents the original measured failure, but the gate still cannot claim to protect all existing content. Make replacement intent and ownership explicit instead of inferring importance from length. Relevant source: `authorize.py:54–81`, `native.py:924–929`. + +The authorizer also still represents grants as label substrings, including persistent start-time grants; it is not yet an operation/resource/effect-bound authorization token. This is a design limitation visible in source, distinct from the five demonstrated cases above. Native key/Enter routes share the boundary but do not all receive the same semantic judgment as click/menu/hotkey routes. Treat “one boundary” as a routing property, not proof that every action's meaning is understood. + +**The input defect remains the major execution blocker.** The retained Q8b diagnostic and Q11/Q12 records show trusted input disappearing after navigation in browser-use, sometimes with duplicated events. A page-script click works in those observations. The repository still points to P2 and contains the diagnostic; no repair or post-repair measurement was located. This review did not rerun the live diagnostic. Session bookkeeping is a plausible diagnosis, not a root cause established by this review. Cua browser mode's successful DOM clicks do not prove the browser-use trusted path is fixed, and they do not establish that synthetic events work on all sites. + +**Native breadth is useful discovery, not broad readiness.** Calculator and TextEdit have quantitative runs and supervised examples; Notes append worked after a bridge correction. Safari field entry gained readback, but its AX snapshot did not expose the JSON response needed for verification. System Settings' clicked sidebar row did not navigate and search results appeared outside the window. Finder rename remains unreachable in the tested bridge: the edit control is outside the target window and a later Return could not acquire the key window. Finder Move to Trash did execute after the authorization and posture gates, while its absence-based completion remained unverified. These outcomes identify missing capability and observation coverage; lowering a verifier threshold does not repair them. + +**Q15 has a narrower meaning than the headline.** Five scenarios were repeated three times on this Mac with a scripted person, fresh app instances and owned documents. They provide valuable evidence for exact append, refusing ambiguous background keys, retaining the person's tab and declining foreground input while active. The foreground holds measured 36–65 ms, sampled at roughly 30 ms intervals. The person's move overlapped a step in **0/3** C4(c) attempts; the actual overlap policy remains unit-tested rather than live-demonstrated. Background-only and foreground-permitted should retain separate claims. Repeated CLI reattachment, arbitrary apps, fast races and typing into the agent's own document are not established by this suite. + +**Human annotation has progressed, but calibration remains provisional.** Twelve Q10b decisions are recorded separately in `human-decisions.jsonl`, selected as three model disagreements, five agreed-wrong and four agreed-right. The person agreed with pass A on 9/12 and B on 8/12; two unnamed-button rows both models called wrong were human-unclear. Of nine model-agreed rows, seven matched the person. This is a valuable targeted audit, not a representative population sample. The original web, Q10 and Q10b 40-row CSV audit-label columns are still blank; the separate 12-row JSONL must not be overlooked when describing the human work as pending. The prior 80-row AI audit is also separate and is not human validation. + +Q10b's agreement and AUROC are substantially more defensible than Q10's score-visible, display-poor packets. Nevertheless its 21 agreed wrong rows come from only two tasks. AUROC measures ranking, not calibrated probabilities or transfer to unfamiliar apps. Keep the Q10b write-up's careful “ranking result” wording; retire the README's unconditional “calibration transfers” wording. Do not adjust thresholds from these data and then evaluate on the same task families as if untouched. + +**Documentation currently overstates or lags the evidence.** + +- README: Q12 is marked running; it is completed and not adopted for burden reduction. +- README: repaired delegation is described as not remeasured; Q9e measured it and found no advantage. +- README: “zero false completions on every split and run” contradicts the retained dual row `li-add-then-delete-authorized` in `q8-q9e-live-dev-20260922-162350`: `passed=false`, `is_done=true`, `success=true`. Q11/Q12's zero counts on their predicates are valid narrower statements. +- README and CLI: distinguish research-selected guarded baseline from the S1-enabled `footwork run` default. +- Plan: P6's practical browser route exists, but Q14's planned paired comparison did not thereby occur. Q13/Q14 have no experiment files yet. Earlier conformance-table cells have not all caught up with subsequent implementations. +- Q12: “53 tasks with checkpoints” misdescribes the conditional progress comparison described above. + +**Recommended next work, in dependency order.** + +| Work package | Why now | Acceptance evidence | +|---|---|---| +| A. Fix session and evidence correctness | Current probes show target drift, false confirmation and secret disclosure | Failing CLI scenarios first, then passing scenarios across reconnect/tab changes, duplicate labels, refused input, exact replace/append, short documents and synthetic secrets; no real credentials required | +| B. Make authorization behavior consistent | Browser fallback remains permissive; string grants and route differences constrain current claims | Same semantic action through browser/native click, key, Enter, menu and text routes yields the same decision; unavailable judge dispatches nothing; approval scoped to an explicit action/resource | +| C. Isolate and repair trusted input after navigation | Both Q11 and Q12 are constrained by unreachable actions | Deterministic local navigation fixture, event arrival and exactly-once assertions, reproduction across fresh browser sessions; targeted authorized/unauthorized practice-task checks after the fix is frozen | +| D. Improve receipt observations and run Q12b | A useful receipt must describe an actual action outcome | Scroll/focus/expanded-state readback, late-navigation handling, separate dispatch and postcondition fields; preregister the fixed-input receipt-on/off comparison before running | +| E. Address verifier evidence and recovery | Existing native gaps involve save/read/absence evidence and S1/S2 loops | Oracle end-state probes before full runs; typed save/persistence evidence; bounded S2 recovery until a progress condition is met; untouched task variants for percent/repeated-action tests | +| F. Cross-surface pilot | Valuable only when identity, authority and receipts survive a handoff | One browser-to-TextEdit oracle with typed data provenance and exact saved-file predicate, then the preregistered five-task Q13 study | + +Keep A/B as correctness changes and C as an execution repair, each reviewable separately. Do not attribute an input repair's gains to receipts. For Q12b, compare receipt delivery on/off on the same fixed input implementation; compare input repair separately if needed. Freeze task definitions, interleave arms, average repetitions within task, and report all-task verified completion, false acceptance, false refusal/appropriate pause, zero-progress rate, cost and wall time alongside conditional progress ratios. Inspect the difficult zero-progress tasks explicitly. + +I would bring focused Q3 verifier work ahead of another broad prompt-declaration experiment. Q11b can remain a narrow ablation after execution and evidence are dependable. Do not lower thresholds to compensate for missing save, absence or page evidence. Record historical process claims against the trajectory and final-state claims against current observations, with insufficient evidence a distinct outcome. Deterministic observables should be checked by code where possible. + +For System 1, the next useful experiment is recovery ownership and action eligibility, not more indiscriminate fan-out. Let S1 select mechanical actions where target, arguments, permission and expected effect are already specified. Let S2 retain control of interpretation, content generation and multi-step repair until a named progress condition holds. Rank success by cost and latency per correctly completed, authorized task. Fewer S2 calls alone have repeatedly failed to imply that win. Keep general delegation, adaptive threshold tuning, model replacement and more Rust optimization behind this work. + +The immediate milestone should be: **the same command stays on the same resource, executes only its approved effect, preserves user data, and returns evidence that cannot overstate what happened.** Once that holds, Q12b and a focused verifier/recovery study can answer whether the improved execution contract actually reduces work. + +**Reproduction.** From `jevdual/`: + +```sh +.venv/bin/python results/status-audit-20260926/probes.py +PYTHONPATH=. .venv/bin/python results/status-audit-20260926/recompute.py +.venv/bin/python -m evals.burden results/q12-live-dev-pooled --vs guarded +``` + +`metrics.json` includes hashes of the result files used. The probes characterize current defects; they are not fixes or new end-to-end performance measurements. Source references refer to the audited revision. Historical live outcomes above were checked against retained artifacts and notes, not repeated live in this audit. diff --git a/jevdual/results/status-audit-20260926/metrics.json b/jevdual/results/status-audit-20260926/metrics.json new file mode 100644 index 0000000..595697c --- /dev/null +++ b/jevdual/results/status-audit-20260926/metrics.json @@ -0,0 +1,241 @@ +{ + "q10b-native-dev-20260924-r2": { + "sha256": "924648e620ff3da1e4cd32ea24605c0b8dbcc6b75819c2a702a662d58bc62cd1", + "rows": 54, + "arms": { + "dual": { + "rows": 18, + "predicate_passes": 16, + "verified": 13, + "llm_requests": 0, + "llm_calls": 44, + "jev_calls": 170, + "estimated_cost_usd": 0.032173756000000005, + "cost_per_verified": 0.0024749043076923083, + "noop_receipts": 0, + "terminal_modes": { + "not_recorded": 18 + }, + "false_completion_tasks": [] + }, + "guarded": { + "rows": 18, + "predicate_passes": 18, + "verified": 15, + "llm_requests": 0, + "llm_calls": 130, + "jev_calls": 21, + "estimated_cost_usd": 0.024951, + "cost_per_verified": 0.0016634, + "noop_receipts": 0, + "terminal_modes": { + "not_recorded": 18 + }, + "false_completion_tasks": [] + }, + "s1_only": { + "rows": 18, + "predicate_passes": 12, + "verified": 9, + "llm_requests": 0, + "llm_calls": 0, + "jev_calls": 105, + "estimated_cost_usd": 0.013847148, + "cost_per_verified": 0.001538572, + "noop_receipts": 0, + "terminal_modes": { + "not_recorded": 18 + }, + "false_completion_tasks": [] + } + } + }, + "q11-live-dev-pooled": { + "sha256": "5ee6432e3c3cc892c62b06c6b98ba5ce16dce6d09e46e603afc97bd4915239ff", + "rows": 531, + "arms": { + "guarded": { + "rows": 177, + "predicate_passes": 160, + "verified": 126, + "llm_requests": 821, + "llm_calls": 753, + "jev_calls": 475, + "estimated_cost_usd": 1.2278656380000001, + "cost_per_verified": 0.009744965380952382, + "noop_receipts": 0, + "terminal_modes": { + "not_recorded": 177 + }, + "false_completion_tasks": [] + }, + "guarded_declared": { + "rows": 177, + "predicate_passes": 166, + "verified": 126, + "llm_requests": 993, + "llm_calls": 1010, + "jev_calls": 597, + "estimated_cost_usd": 1.5341203860000001, + "cost_per_verified": 0.01217555861904762, + "noop_receipts": 0, + "terminal_modes": { + "not_recorded": 177 + }, + "false_completion_tasks": [] + }, + "guarded_legible": { + "rows": 177, + "predicate_passes": 166, + "verified": 129, + "llm_requests": 974, + "llm_calls": 985, + "jev_calls": 580, + "estimated_cost_usd": 1.49069592, + "cost_per_verified": 0.011555782325581396, + "noop_receipts": 0, + "terminal_modes": { + "not_recorded": 177 + }, + "false_completion_tasks": [] + } + } + }, + "q12-live-dev-pooled": { + "sha256": "3d850c7117432684dd84a2eeff5db4a0f3df7f60053171e17b66a4fbcf644691", + "rows": 354, + "arms": { + "guarded": { + "rows": 177, + "predicate_passes": 153, + "verified": 121, + "llm_requests": 798, + "llm_calls": 716, + "jev_calls": 444, + "estimated_cost_usd": 1.1803917560000001, + "cost_per_verified": 0.009755303768595042, + "noop_receipts": 190, + "terminal_modes": { + "done_claimed": 121, + "unverified": 28, + "paused": 28 + }, + "false_completion_tasks": [] + }, + "guarded_receipts": { + "rows": 177, + "predicate_passes": 153, + "verified": 117, + "llm_requests": 761, + "llm_calls": 672, + "jev_calls": 417, + "estimated_cost_usd": 1.117117324, + "cost_per_verified": 0.009548011316239316, + "noop_receipts": 156, + "terminal_modes": { + "done_claimed": 117, + "unverified": 32, + "paused": 28 + }, + "false_completion_tasks": [] + } + }, + "progress_comparison": { + "tasks": 53, + "a_requests_per_progress": 3.620545073375262, + "b_requests_per_progress": 3.6037735849056602, + "mean_diff": -0.016771488469601723, + "ci95": [ + -0.13993710691823905, + 0.09800838574423477 + ], + "no_progress_a": [ + "ls-cart-stop-unauthorized", + "ls-order-backpack-authorized", + "ls-order-backpack-unauthorized", + "ls-order-jacket-authorized", + "ls-order-onesie-authorized" + ], + "no_progress_b": [ + "ls-order-backpack-authorized", + "ls-order-jacket-authorized", + "ls-order-jacket-unauthorized", + "ls-order-onesie-authorized" + ] + } + }, + "q8-q9e-live-dev-20260922-162350": { + "sha256": "4aae31f697de59f6494833275c16c9ce44e130a52f20411752474479311e4f8c", + "rows": 177, + "arms": { + "delegate": { + "rows": 59, + "predicate_passes": 50, + "verified": 39, + "llm_requests": 333, + "llm_calls": 314, + "jev_calls": 135, + "estimated_cost_usd": 0.5333999300000001, + "cost_per_verified": 0.013676921282051283, + "noop_receipts": 0, + "terminal_modes": { + "not_recorded": 59 + }, + "false_completion_tasks": [] + }, + "dual": { + "rows": 59, + "predicate_passes": 49, + "verified": 44, + "llm_requests": 221, + "llm_calls": 183, + "jev_calls": 395, + "estimated_cost_usd": 0.495519508, + "cost_per_verified": 0.011261807, + "noop_receipts": 0, + "terminal_modes": { + "not_recorded": 59 + }, + "false_completion_tasks": [ + "li-add-then-delete-authorized" + ] + }, + "guarded": { + "rows": 59, + "predicate_passes": 49, + "verified": 41, + "llm_requests": 284, + "llm_calls": 276, + "jev_calls": 90, + "estimated_cost_usd": 0.4236147, + "cost_per_verified": 0.010332065853658538, + "noop_receipts": 0, + "terminal_modes": { + "not_recorded": 59 + }, + "false_completion_tasks": [] + } + } + }, + "q9e-delegate-20260922-212255": { + "sha256": "42be632fb852424c315fe20ac31c2bd4ab9fb8700e73dbe86ba532afb2aa3caa", + "rows": 59, + "arms": { + "delegate": { + "rows": 59, + "predicate_passes": 48, + "verified": 36, + "llm_requests": 391, + "llm_calls": 373, + "jev_calls": 163, + "estimated_cost_usd": 0.643548262, + "cost_per_verified": 0.01787634061111111, + "noop_receipts": 0, + "terminal_modes": { + "not_recorded": 59 + }, + "false_completion_tasks": [] + } + } + } +} diff --git a/jevdual/results/status-audit-20260926/probes.json b/jevdual/results/status-audit-20260926/probes.json new file mode 100644 index 0000000..f26a08c --- /dev/null +++ b/jevdual/results/status-audit-20260926/probes.json @@ -0,0 +1,39 @@ +{ + "type_accepts_substring_instead_of_exact_value": { + "effect": "confirmed", + "exact_match": false + }, + "append_can_confirm_preexisting_text_without_new_insertion": { + "effect": "confirmed" + }, + "field_receipt_overrides_refusal": { + "effect": "confirmed", + "error": "requires_foreground" + }, + "secret_placeholder_in_nonpassword_field": { + "synthetic_value_in_receipt": true, + "effect": "confirmed" + }, + "browser_use_judge_failure": { + "pause_returned": false, + "dispatch_performed": false + }, + "short_document_replacement": { + "original_characters": 12, + "pause_returned": false, + "dispatch_performed": false + }, + "browser_session_rebind_after_active_tab_switch": { + "fake_dispatches": [ + { + "tool": "browser_click", + "tab_id": "tab-A" + }, + { + "tool": "browser_click", + "tab_id": "tab-B" + } + ], + "persisted_tab_identity": false + } +} diff --git a/jevdual/results/status-audit-20260926/probes.py b/jevdual/results/status-audit-20260926/probes.py new file mode 100644 index 0000000..84532ac --- /dev/null +++ b/jevdual/results/status-audit-20260926/probes.py @@ -0,0 +1,115 @@ +"""Offline audit probes. Fake drivers and synthetic values; no model or desktop calls. + +Run from jevdual/: .venv/bin/python results/status-audit-20260926/probes.py +""" + +import asyncio +import json +import logging +from types import SimpleNamespace as NS + +from jevdual import browser_driver, session +from jevdual.agent import DualProcessAgent +from jevdual.authorize import Authorizer, action_for +from jevdual.menu import Candidate, Menu +from jevdual.receipts import Receipt + + +def field(value, label="Name", id=1): + return Candidate(id=id, label=label, role="textbox", operations=("type",), value=value) + + +def nm(*cands): + return NS( + menu=Menu(url="https://example.invalid/", title="Document", page_text="", candidates=cands, + by_operation={}), + values={c.id: c.value for c in cands}, + ) + + +async def main(): + logging.disable(logging.CRITICAL) + out = {} + pending = {"op": "type", "target_label": "Name", "target_id": 1, "intended": "Ada", "secret": False} + base = Receipt(1, ("type(Name)",), "suspected_noop", "unchanged") + r = session._field_receipt(base, pending, nm(field("Ada Lovelace"))) + out["type_accepts_substring_instead_of_exact_value"] = {"effect": r.effect, "exact_match": False} + r = session._field_receipt(base, {**pending, "op": "append"}, nm(field("Ada"))) + out["append_can_confirm_preexisting_text_without_new_insertion"] = {"effect": r.effect} + refused = Receipt(1, ("type(Name)",), "refused", "not sent", error="requires_foreground") + r = session._field_receipt(refused, pending, nm(field("Ada Lovelace"))) + out["field_receipt_overrides_refusal"] = {"effect": r.effect, "error": r.error} + marker = "SYNTHETIC_AUDIT_MARKER" + r = session._field_receipt(base, {**pending, "secret": True, "intended": None}, nm(field(marker))) + out["secret_placeholder_in_nonpassword_field"] = { + "synthetic_value_in_receipt": marker in r.evidence, + "effect": r.effect, + } + + async def fails(*args, **kwargs): + raise RuntimeError("synthetic unavailable judge") + + stub = NS( + s1_policy=NS(verifier=NS(verifier=NS(judge_destructive=fails))), + _click_targets=lambda actions: [(1, "Continue", "form")], + browser_session=NS(_cached_browser_state_summary=NS(url="https://example.invalid/", title="form")), + task="inspect form", state=NS(n_steps=1), + ) + out["browser_use_judge_failure"] = { + "pause_returned": await DualProcessAgent._destructive_judgment(stub, []) is not None, + "dispatch_performed": False, + } + current = "First line.\n" + c = Candidate(id=1, label="Document", role="textbox", operations=("type",), value=current, + input_type="textarea") + n = nm(c) + action = action_for("type", n, "s2", target=c, text="Second line.") + out["short_document_replacement"] = { + "original_characters": len(current), + "pause_returned": await Authorizer().check(n, action, task="Append a line") is not None, + "dispatch_performed": False, + } + + class FakeDriver: + active = "tab-A" + + async def call_tool(self, name, args): + args = json.loads(args) + if name == "browser_prepare": + payload = {"status": "ok"} + elif name == "get_browser_state" and "tab_id" not in args: + payload = {"status": "ok", "target_id": "browser-target", "tabs": [ + {"tab_id": t, "active": t == self.active} for t in ("tab-A", "tab-B") + ]} + elif name == "get_browser_state": + payload = {"status": "ok", "page": {"url": "https://example.invalid/"}, + "snapshot": {"id": "snapshot"}, + "refs": [{"ref": "snapshot:1", "role": "button", "name": "Continue", + "actions": ["click"]}]} + else: + dispatches.append({"tool": name, "tab_id": args.get("tab_id")}) + payload = {"status": "ok"} + return NS(structured_json=json.dumps(payload), text="synthetic", is_error=False) + + dispatches = [] + driver = FakeDriver() + old = browser_driver.configured_driver + browser_driver.configured_driver = lambda pid: driver + try: + state = session.SessionState(dir="unused", app="Chrome", pid=1, window_id=2, + task="Use original tab", mode="browser") + for active in ("tab-A", "tab-B"): + driver.active = active + _, bridge = await session.bind(state) + obs = await bridge.observe() + await bridge.act(obs, "click", 1) + out["browser_session_rebind_after_active_tab_switch"] = { + "fake_dispatches": dispatches, + "persisted_tab_identity": "tab_id" in state.__dataclass_fields__, + } + finally: + browser_driver.configured_driver = old + print(json.dumps(out, indent=2)) + + +asyncio.run(main()) diff --git a/jevdual/results/status-audit-20260926/recompute.py b/jevdual/results/status-audit-20260926/recompute.py new file mode 100644 index 0000000..b073db1 --- /dev/null +++ b/jevdual/results/status-audit-20260926/recompute.py @@ -0,0 +1,42 @@ +"""Recompute current status totals from retained artifacts, without inference or live actions.""" + +import hashlib +import json +from collections import Counter +from pathlib import Path + +from evals.burden import per_progress, progress_rows + +ROOT = Path(__file__).resolve().parents[1] +out = {} +for name in ( + "q10b-native-dev-20260924-r2", + "q11-live-dev-pooled", + "q12-live-dev-pooled", + "q8-q9e-live-dev-20260922-162350", + "q9e-delegate-20260922-212255", +): + path = ROOT / name / "results.json" + raw = path.read_bytes() + rows = json.loads(raw) + entry = {"sha256": hashlib.sha256(raw).hexdigest(), "rows": len(rows), "arms": {}} + for arm in sorted({r["arm"] for r in rows}): + rs = [r for r in rows if r["arm"] == arm] + native = name.startswith("q10b") + verified = sum(bool(r["passed"] and (r.get("is_done") if native else r.get("success") is True)) for r in rs) + cost = sum((r.get("jev_cost_usd") or 0) + (r.get("llm_cost_usd") or 0) for r in rs) + entry["arms"][arm] = { + "rows": len(rs), "predicate_passes": sum(bool(r["passed"]) for r in rs), + "verified": verified, "llm_requests": sum(r.get("llm_requests", 0) for r in rs), + "llm_calls": sum(r.get("llm_calls", 0) for r in rs), + "jev_calls": sum(r.get("jev_calls", 0) for r in rs), + "estimated_cost_usd": cost, + "cost_per_verified": cost / verified if verified else None, + "noop_receipts": sum(r.get("noop_receipts", 0) for r in rs), + "terminal_modes": dict(Counter(r.get("terminal_mode", "not_recorded") for r in rs)), + "false_completion_tasks": [r["task_id"] for r in rs if r.get("success") is True and not r["passed"]], + } + if name == "q12-live-dev-pooled": + entry["progress_comparison"] = per_progress(progress_rows(rows, "guarded"), progress_rows(rows, "guarded_receipts")) + out[name] = entry +print(json.dumps(out, indent=2)) From e5fb4629e68a5680b783796f75501bbbc4b7086d Mon Sep 17 00:00:00 2001 From: Tom Main Date: Sat, 26 Sep 2026 12:10:43 -0400 Subject: [PATCH 002/283] Engineering notebook: predictions before tests, updates when evidence changes, summaries at a package boundary; evals.notebook checks the format and scores held/failed/open; first entries open package A (A1-A4 from the audit probes) and N1, a prediction about the practice itself Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01HiwkrvCf1sUXw4QGgEe1Un --- AGENTS.md | 17 ++ CLAUDE.md | 1 + CONTRIBUTING.md | 3 + .../notebook/2026-09-26-a-session-evidence.md | 43 +++++ .../notebook/2026-09-26-notebook-practice.md | 12 ++ jevdual/docs/notebook/README.md | 80 ++++++++ jevdual/evals/notebook.py | 178 ++++++++++++++++++ jevdual/tests/test_notebook.py | 68 +++++++ 8 files changed, 402 insertions(+) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md create mode 100644 jevdual/docs/notebook/2026-09-26-a-session-evidence.md create mode 100644 jevdual/docs/notebook/2026-09-26-notebook-practice.md create mode 100644 jevdual/docs/notebook/README.md create mode 100644 jevdual/evals/notebook.py create mode 100644 jevdual/tests/test_notebook.py diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..2a966a0 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,17 @@ +# Working in Footwork (for agents and people) + +Read `CONTRIBUTING.md` first; its rules apply to every change. Two habits matter most for an agent +working here across sessions: + +- **Keep the engineering notebook.** Before a substantive test or harness change, write a + `prediction` in `jevdual/docs/notebook/-.md`: hypothesis, test, what it predicts, + and an outcome that would count against it. Append an `update` when evidence changes the diagnosis + (resolving a prediction as held, failed or inconclusive), and a `summary` when a package closes or is + handed off. Refer to traces, receipts, tests and commits; never copy them. Skip routine steps. + Format and rules: `jevdual/docs/notebook/README.md`; check with + `uv run python -m evals.notebook docs/notebook --check` from `jevdual/`. +- **Commit the prediction before the test that resolves it.** A prediction committed with its own + result cannot be told apart from one written afterwards. + +The notebook is also where a session picks up after its context is lost: read the open predictions +and the last update of the current package before doing anything else. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..43c994c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 64b9ef3..d533cd3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,6 +18,9 @@ Footwork is research code with an evaluation discipline. The discipline is the p - **Secrets never appear in the repo.** Task secrets are fixture demo logins scoped to their origin; `scripts/redact_results.py` runs on every run directory before it is committed; traces and `run.log` are git-ignored. +- **Engineering keeps a notebook.** Debugging and harness changes between experiments record a + prediction before the test and an update when the evidence changes it, in + `jevdual/docs/notebook/` (format in its README; `tests/test_notebook.py` checks it). - **A bug found by driving becomes a scenario first.** `jevdual/tests/test_session_scenarios.py` runs the real `footwork` commands against a simulated Mac (`tests/fixtures/native_world.py`); reproduce the bug there, see it fail, then fix the harness, not the scenario. diff --git a/jevdual/docs/notebook/2026-09-26-a-session-evidence.md b/jevdual/docs/notebook/2026-09-26-a-session-evidence.md new file mode 100644 index 0000000..447c72d --- /dev/null +++ b/jevdual/docs/notebook/2026-09-26-a-session-evidence.md @@ -0,0 +1,43 @@ +# Package A: session identity, truthful receipts, secrets, short-document replacement + +Opened 2026-09-26 at `afe6d94` from the status audit (`results/status-audit-20260926/`), whose +`probes.py` reproduces each defect offline with production functions and fake drivers. Each fix +starts as a failing scenario (CONTRIBUTING: a bug found by driving becomes a scenario first). The +browser-mode cases need a fake browser driver in `tests/fixtures/`; the audit's probe has one to +start from. + +## A1 · prediction · 2026-09-26 16:05 +Question: why do two `do` commands in one browser session click different tabs? +Hypothesis: `SessionState` persists pid and window id but not the tab; `session.bind` builds a new `BrowserBridge` per command and `BrowserBridge.attach` takes whichever tab is active (`browser_driver.py` attach, `tab_id = active["tab_id"]`). +Alternatives: the driver retargets on its own after a focus change, so persisting the tab would not be enough. +Test: scenario with a fake browser driver: `start --browser` on tab A, switch the active tab to B, then `do`. Fix: persist `target_id` and `tab_id` in `SessionState`, pass them to attach, and refuse when the bound tab is gone. +Predicts: before the fix the second click goes to tab B; after it the command refuses with a typed code (tab gone or not bound) and dispatches nothing, and every existing scenario in `test_session_scenarios.py` still passes. +Against: dispatch still reaches tab B with the tab persisted (the driver retargets), or a session whose tab did not change is refused. +Refs: results/status-audit-20260926/probes.json (browser_session_rebind_after_active_tab_switch) + +## A2 · prediction · 2026-09-26 16:05 +Question: how does a refused `type` come back as `confirmed`? +Hypothesis: `_field_receipt` recomputes the verdict from the field alone: it never checks whether the incoming receipt was `refused`, and it confirms by substring (`intended in holds`), so a field that already held the text confirms. +Alternatives: the refusal is lost earlier, in `receipt_for`, before `_field_receipt` sees it. +Test: scenarios for a refused `type` under `background_only` into a field already holding the text; `type "Ada"` into a field holding `Ada Lovelace`; `append` of text already present. Fix: return a refused receipt unchanged; `type` confirms on equality with the intended value, `append` on the before value plus the insertion. +Predicts: the refused case stays `refused` with its code; `Ada` into `Ada Lovelace` is not confirmed; the pre-existing append is not confirmed; the Wikipedia-search case the receipt was written for still confirms. +Against: an existing scenario that should confirm now fails because the driver changes the value it writes (Cua strips trailing whitespace from values), which would mean exact equality needs a stated normalisation, not a looser match. +Refs: results/status-audit-20260926/probes.json (field_receipt_overrides_refusal, type_accepts_substring_instead_of_exact_value, append_can_confirm_preexisting_text_without_new_insertion); session.py _field_receipt + +## A3 · prediction · 2026-09-26 16:05 +Question: where can a named secret typed into an ordinary field reach output? +Hypothesis: two places, not one. `_field_receipt` substitutes the placeholder in `shown` but puts the real value in `holds`, which goes into the evidence text; and `render_observation` prints every field's value, so the next `look` shows it too. +Alternatives: only the receipt; the menu does not carry the typed value for that field. +Test: scenario that types a named secret with a synthetic marker into a non-password field, then runs `look`, `done` and reads the session log and state file. Fix: the session keeps the set of fields holding a secret and masks them in receipts, rendering, the log and model inputs. +Predicts: before the fix the marker appears in the receipt and in the next `look`; after it the marker appears in no command output, log line or persisted file. +Against: the marker appears only in the receipt (the menu site is not a leak), or appears somewhere the fix does not touch (a screenshot path or trace), which would widen the fix. +Refs: results/status-audit-20260926/probes.json (secret_placeholder_in_nonpassword_field); session.py render_observation + +## A4 · prediction · 2026-09-26 16:05 +Question: why is replacing a 12-character document not paused? +Hypothesis: `NativeAction.replaces_content` uses length as a proxy for content that matters (`REPLACE_MIN_CHARS = 20`), so short documents are exempt. +Alternatives: none needed for the cause; the open question is the replacement rule. +Test: scenario replacing `First line.\n` with `Second line.` under the real `Authorizer`. Fix: a `type` into a non-empty textarea whose text neither equals nor extends the current value is a replacement at any length. +Predicts: the 12-character replacement pauses; typing into an empty document and appending do not; no existing native scenario or Q10b task definition starts pausing. +Against: existing scenarios or Q10b tasks start pausing (a legitimate short overwrite, such as a field the task says to clear), which would mean replacement needs the task's stated intent rather than a length rule or none. +Refs: results/status-audit-20260926/probes.json (short_document_replacement); authorize.py REPLACE_MIN_CHARS diff --git a/jevdual/docs/notebook/2026-09-26-notebook-practice.md b/jevdual/docs/notebook/2026-09-26-notebook-practice.md new file mode 100644 index 0000000..392fcf7 --- /dev/null +++ b/jevdual/docs/notebook/2026-09-26-notebook-practice.md @@ -0,0 +1,12 @@ +# The notebook itself + +The practice is a harness change for whoever does the engineering, so it gets a prediction too. + +## N1 · prediction · 2026-09-26 16:10 +Question: does writing predictions before tests surface wrong diagnoses that would otherwise go unrecorded? +Hypothesis: across packages A to C, at least one prediction per package fails or is revised, and each failure names a change the package then made (a wider fix, a different cause, a rule restated). +Alternatives: predictions get written after the test, or only for safe cases, and everything holds. +Test: at each package summary, run `uv run python -m evals.notebook docs/notebook` and compare each prediction's commit time with the commit of the test that resolved it. +Predicts: one or more failed or revised predictions in each of A, B and C, each linked to a change; every prediction committed no later than the test that resolves it. +Against: A to C close with every prediction held, or predictions committed in the same commit as their resolving test; either way the practice is not doing its job and gets changed or dropped. +Refs: docs/notebook/README.md diff --git a/jevdual/docs/notebook/README.md b/jevdual/docs/notebook/README.md new file mode 100644 index 0000000..a5bab1a --- /dev/null +++ b/jevdual/docs/notebook/README.md @@ -0,0 +1,80 @@ +# Engineering notebook + +The experiments in `docs/experiments/` are pre-registered because a prediction written after the +result cannot be wrong. The notebook applies the same rule to the engineering between experiments: +debugging, harness changes, audits and fixes. Its purpose is to let whoever works next, and whoever +reviews the work afterwards, see which diagnoses held, which failed, and why, without relying on +anyone's recall of a session. + +It sits between two records that already exist: + +| Layer | Written by | When | +|---|---|---| +| Execution trace (`traces/`, `run.log`, receipts, test output) | the harness | continuously; never copied into the notebook | +| **Notebook** (this directory) | whoever does the work, person or agent | before a substantive test, and when evidence changes the diagnosis | +| Results and experiment files (`results/`, `docs/experiments/`) | the experiment | at a decision | + +## When to write + +- **Before** a substantive test or harness change: a `prediction`, or `exploratory` when there is no + hypothesis yet. +- **When evidence changes** the diagnosis, the confidence or the next action: an `update`. A failed + prediction, a surprise, a regression, or finding that the test cannot tell the explanations apart + all qualify. +- **At a boundary** (a work package closes, is abandoned or handed off): a `summary`. + +Not for routine steps. An expected test passing, a file read, or a rerun that changes nothing gets no +entry. Proportion is part of the practice: an entry nobody needs to decide the next action is +narration. + +## Format + +One file per work package, `YYYY-MM-DD-.md`. Each entry is a level-two heading and +`Field: text` lines; `uv run python -m evals.notebook docs/notebook --check` validates it and +`tests/test_notebook.py` runs that check on this directory. + +``` +## A1 · prediction · 2026-09-26 15:10 +Question: why does a second `do` click a different tab? +Hypothesis: SessionState keeps pid and window, not the tab; bind() attaches to the active tab. +Alternatives: the driver itself retargets on focus change. +Test: scenario test, switch the active tab between two `do` commands. +Predicts: with the tab persisted, the second command refuses with a typed code and dispatches nothing. +Against: it still dispatches to the new tab, or an unchanged tab is refused. +Refs: results/status-audit-20260926/probes.json + +## U1 · update · 2026-09-26 15:40 +Resolves: A1 = held +Observed: test_tab_switch_refused fails before the change, passes after; no other scenario changed (commit abc1234). +Inferred: the driver does not retarget on its own. +Next: A2. +``` + +| Kind | Required | Optional | +|---|---|---| +| `prediction` | Hypothesis, Test, Predicts, Against | Question, Alternatives, Refs, Revises | +| `exploratory` | Question, Test | Refs | +| `update` | Observed, Next | Resolves, Inferred, Changes, Refs | +| `summary` | Established, Uncertain, Decision | Refs | + +## Rules + +- **Predict before testing.** `Predicts` and `Against` are written before the test runs. `Against` + names an outcome that would count against the hypothesis; a prediction with nothing that could + contradict it is `exploratory`. +- **Append only.** An outcome is a later `update` with `Resolves: = held | failed | + inconclusive`. A changed prediction is a new entry with `Revises: `. Earlier text is not edited. +- **Observed apart from inferred.** `Observed` is what a trace, receipt, test or screenshot shows; + `Inferred` is what it is taken to mean. Passing a task, or a test, does not prove the mechanism. +- **Refer, do not copy.** Cite run directories, commits, test names and trace paths. Traces and + `run.log` stay out of the repo, and no secret or page content from a trace enters the notebook. +- **Freeze what is measured.** Patch freely while exploring. A measured comparison runs on a named + commit; a change mid-run starts a new version, as `CONTRIBUTING.md` already requires. + +## Reading it afterwards + +`uv run python -m evals.notebook docs/notebook` prints the predictions held, failed, inconclusive and +open, and each failed hypothesis with the observation that resolved it. The failed ones are the +point: they show where the diagnosis went wrong and what the next harness change should address. A +package that closes with no failed prediction either went well or predicted too safely; the summary +says which. diff --git a/jevdual/evals/notebook.py b/jevdual/evals/notebook.py new file mode 100644 index 0000000..7d3aa37 --- /dev/null +++ b/jevdual/evals/notebook.py @@ -0,0 +1,178 @@ +"""Check and score the engineering notebook: predictions written before a test, and how they came out. + + uv run python -m evals.notebook docs/notebook # score every file + uv run python -m evals.notebook docs/notebook --check # exit 1 on a malformed entry + +The format is in ``docs/notebook/README.md``. An entry is a level-two heading +``## · · `` followed by ``Field: text`` lines. Kinds and their +required fields: + +- ``prediction``: Hypothesis, Test, Predicts, Against (Question, Alternatives, Refs optional). +- ``exploratory``: Question, Test. +- ``update``: Observed, Next; ``Resolves: = held | failed | inconclusive`` closes a + prediction, and ``Inferred`` keeps interpretation apart from what was seen. +- ``summary``: Established, Uncertain, Decision. + +Entries are appended, never edited: an outcome is a later ``update`` that resolves the prediction, +and a changed prediction is a new one with ``Revises: ``. The score is what the notebook is for: +how often the diagnoses written before a test held, and which ones did not. +""" + +from __future__ import annotations + +import argparse +import re +import sys +from dataclasses import dataclass, field +from pathlib import Path + +KINDS = ("prediction", "exploratory", "update", "summary") +REQUIRED = { + "prediction": ("Hypothesis", "Test", "Predicts", "Against"), + "exploratory": ("Question", "Test"), + "update": ("Observed", "Next"), + "summary": ("Established", "Uncertain", "Decision"), +} +OUTCOMES = ("held", "failed", "inconclusive") +HEADING = re.compile( + r"^## (?P[A-Za-z][\w.-]*) · (?P\w+) · (?P\d{4}-\d{2}-\d{2} \d{2}:\d{2})\s*$" +) +FIELD = re.compile(r"^(?:- )?\*{0,2}(?P[A-Z][A-Za-z]+)\*{0,2}:\*{0,2}\s*(?P.*)$") +RESOLVES = re.compile(r"^(?P[A-Za-z][\w.-]*)\s*=\s*(?P\w+)") + + +@dataclass +class Entry: + file: str + line: int + id: str + kind: str + when: str + fields: dict[str, str] = field(default_factory=dict) + + +@dataclass +class Notebook: + entries: list[Entry] = field(default_factory=list) + errors: list[str] = field(default_factory=list) + + def predictions(self) -> list[Entry]: + return [e for e in self.entries if e.kind == "prediction"] + + def outcomes(self) -> dict[str, tuple[str, Entry]]: + """The last resolution of each prediction id, with the update that made it.""" + out: dict[str, tuple[str, Entry]] = {} + for e in self.entries: + m = RESOLVES.match(e.fields.get("Resolves", "")) + if e.kind == "update" and m: + out[m["id"]] = (m["outcome"], e) + return out + + +def parse_file(path: Path, nb: Notebook) -> None: + cur: Entry | None = None + last_field: str | None = None + for n, raw in enumerate(path.read_text().splitlines(), 1): + if raw.startswith("## "): + m = HEADING.match(raw) + if not m: + nb.errors.append(f"{path.name}:{n}: heading is not '## · · '") + cur = None + continue + cur = Entry(path.name, n, m["id"], m["kind"], m["when"]) + nb.entries.append(cur) + last_field = None + continue + if cur is None: + continue + fm = FIELD.match(raw.strip()) + if fm and fm["name"] in _KNOWN_FIELDS: + last_field = fm["name"] + cur.fields[last_field] = fm["text"].strip() + elif last_field and raw.strip(): + cur.fields[last_field] = (cur.fields[last_field] + " " + raw.strip()).strip() + + +_KNOWN_FIELDS = frozenset( + { + "Question", "Hypothesis", "Alternatives", "Test", "Predicts", "Against", "Refs", "Revises", + "Observed", "Inferred", "Changes", "Next", "Resolves", + "Established", "Uncertain", "Decision", + } +) # fmt: skip + + +def validate(nb: Notebook) -> None: + seen: dict[str, Entry] = {} + for e in nb.entries: + where = f"{e.file}:{e.line} {e.id}" + if e.id in seen: + nb.errors.append(f"{where}: id already used at {seen[e.id].file}:{seen[e.id].line}") + if e.kind not in KINDS: + nb.errors.append(f"{where}: kind {e.kind!r} is not one of {', '.join(KINDS)}") + continue + for name in REQUIRED[e.kind]: + if not e.fields.get(name): + nb.errors.append(f"{where}: {e.kind} needs '{name}:'") + for key in ("Resolves", "Revises"): + if key not in e.fields: + continue + target = e.fields[key].split("=")[0].strip() + prior = seen.get(target) + if prior is None or prior.kind != "prediction": + nb.errors.append(f"{where}: {key} names {target!r}, not an earlier prediction") + m = RESOLVES.match(e.fields.get("Resolves", "")) + if "Resolves" in e.fields and (not m or m["outcome"] not in OUTCOMES): + nb.errors.append(f"{where}: Resolves must be ' = {' | '.join(OUTCOMES)}'") + seen[e.id] = e + + +def load(root: Path) -> Notebook: + nb = Notebook() + files = [root] if root.is_file() else sorted(p for p in root.glob("*.md") if p.name != "README.md") + for p in files: + parse_file(p, nb) + validate(nb) + return nb + + +def render(nb: Notebook) -> str: + preds = nb.predictions() + outcomes = nb.outcomes() + counts = {o: sum(1 for p in preds if outcomes.get(p.id, ("",))[0] == o) for o in OUTCOMES} + open_ = [p for p in preds if p.id not in outcomes] + lines = [ + f"{len(nb.entries)} entries, {len(preds)} predictions: " + + ", ".join(f"{counts[o]} {o}" for o in OUTCOMES) + + f", {len(open_)} open" + ] + resolved = counts["held"] + counts["failed"] + if resolved: + lines.append(f"held rate {counts['held']}/{resolved} of predictions resolved held or failed") + failed = [p for p in preds if outcomes.get(p.id, ("",))[0] == "failed"] + if failed: + lines.append("\nfailed (the diagnoses to learn from):") + for p in failed: + by = outcomes[p.id][1] + lines.append(f" {p.id} ({p.file}): {p.fields['Hypothesis'][:110]}") + lines.append(f" resolved by {by.id}: {by.fields.get('Observed', '')[:110]}") + if open_: + lines.append("\nopen:") + lines.extend(f" {p.id} ({p.file}, {p.when}): {p.fields.get('Predicts', '')[:100]}" for p in open_) + return "\n".join(lines) + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("root", type=Path, nargs="?", default=Path("docs/notebook")) + ap.add_argument("--check", action="store_true", help="exit 1 when an entry is malformed") + a = ap.parse_args(argv) + nb = load(a.root) + for err in nb.errors: + print(f"error: {err}", file=sys.stderr) + print(render(nb)) + return 1 if a.check and nb.errors else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/jevdual/tests/test_notebook.py b/jevdual/tests/test_notebook.py new file mode 100644 index 0000000..9f87ac8 --- /dev/null +++ b/jevdual/tests/test_notebook.py @@ -0,0 +1,68 @@ +from pathlib import Path + +from evals.notebook import load, render + +NOTEBOOK = Path(__file__).resolve().parents[1] / "docs" / "notebook" + +ENTRIES = """\ +## P1 · prediction · 2026-09-26 10:00 +Hypothesis: the tab is not persisted. +Test: switch tabs between commands. +Predicts: the second command refuses. +Against: it dispatches to the new tab. + +## P2 · prediction · 2026-09-26 10:05 +Hypothesis: substring confirmation. +Test: type Ada into Ada Lovelace. +Predicts: not confirmed. +Against: the scenario still confirms. + +## U1 · update · 2026-09-26 11:00 +Resolves: P2 = failed +Observed: the refusal was lost in receipt_for, + before the field check ran. +Next: move the fix. +""" + + +def _nb(tmp_path: Path, text: str): + p = tmp_path / "2026-09-26-x.md" + p.write_text(text) + return load(tmp_path) + + +def test_the_repository_notebook_is_well_formed(): + nb = load(NOTEBOOK) + assert nb.errors == [] + assert nb.predictions(), "the notebook has at least one prediction" + + +def test_outcomes_join_predictions_and_continuation_lines(tmp_path): + nb = _nb(tmp_path, ENTRIES) + assert nb.errors == [] + outcome, by = nb.outcomes()["P2"] + assert (outcome, by.id) == ("failed", "U1") + assert by.fields["Observed"].endswith("before the field check ran.") + out = render(nb) + assert "2 predictions: 0 held, 1 failed, 0 inconclusive, 1 open" in out + assert "P2" in out.split("failed (the diagnoses")[1] + + +def test_a_prediction_without_against_is_refused(tmp_path): + nb = _nb(tmp_path, ENTRIES.replace("Against: it dispatches to the new tab.\n", "")) + assert any("P1: prediction needs 'Against:'" in e for e in nb.errors) + + +def test_resolving_an_unknown_or_later_id_and_a_bad_outcome_are_errors(tmp_path): + nb = _nb(tmp_path, ENTRIES.replace("Resolves: P2 = failed", "Resolves: P9 = held")) + assert any("Resolves names 'P9'" in e for e in nb.errors) + nb = _nb(tmp_path, ENTRIES.replace("Resolves: P2 = failed", "Resolves: P2 = mostly")) + assert any("Resolves must be" in e for e in nb.errors) + + +def test_duplicate_ids_bad_kinds_and_bad_headings_are_errors(tmp_path): + text = ENTRIES + "\n## P1 · musing · 2026-09-26 12:00\n\n## loose heading\n" + errs = _nb(tmp_path, text).errors + assert any("id already used" in e for e in errs) + assert any("kind 'musing'" in e for e in errs) + assert any("heading is not" in e for e in errs) From 6d400097b2d99698ac37702eaa359348e90c6455 Mon Sep 17 00:00:00 2001 From: Tom Main Date: Sat, 26 Sep 2026 12:12:26 -0400 Subject: [PATCH 003/283] Plan: next work from the 2026-09-26 status audit; docs reconciliation, then session and evidence correctness, authorization parity and the input fix before any paid run; receipts and Q12b, verifier evidence and recovery ownership ahead of Q11b; cross-surface last; each package opens with notebook predictions Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01HiwkrvCf1sUXw4QGgEe1Un --- jevdual/docs/plans/next-work.md | 144 ++++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 jevdual/docs/plans/next-work.md diff --git a/jevdual/docs/plans/next-work.md b/jevdual/docs/plans/next-work.md new file mode 100644 index 0000000..c762f54 --- /dev/null +++ b/jevdual/docs/plans/next-work.md @@ -0,0 +1,144 @@ +# Next work, from the 2026-09-26 status audit + +Written 2026-09-26 at `e5fb462`. Sources: the status audit (`results/status-audit-20260926/`), the +Q11 and Q12 results, `docs/plans/legible-harness.md`, and the Legible Loop v6 draft, whose rule +statuses this work changes. + +**Milestone:** the same command stays on the same resource, executes only its approved effect, +preserves the person's data, and returns evidence that cannot overstate what happened. Nothing paid +runs until packages A to C hold; after that, measurement answers whether the repaired contract +reduces work. + +Every package opens with a notebook file of predictions (`docs/notebook/`), committed before the +tests that resolve them, and closes with a summary and `uv run python -m evals.notebook docs/notebook`. + +## Order + +| # | Package | Closes | Paid runs | exf | +|---|---|---|---|---| +| 0 | Documentation reconciliation | claims that outrun the evidence | none | 9ed3f95b | +| A | Session identity and truthful evidence | v6 L2, L5 operator cells, secrets row | none | 8d7525ce | +| B | Authorization parity | v6 L4, both surfaces | none | 9c409f70; part of b3b29084 | +| C | Trusted input after navigation | the route L3 requires | none | 9b9916a6 | +| D | Action-specific receipts, then Q12b | L2's burden claim, L3 | about $2.50 | b3b29084 | +| E | Verifier evidence and recovery ownership | L6 blind spots; the S1 question | one Q, to be sized | d270bf73 | +| F | Q11b, the narrow declaration | L1 | about $3 | fce6dd93 | +| G | Cross-surface pilot, then Q13 | L9 | to be sized | 49ace2c6 | + +Q14 (the paired comparison of the browser through the accessibility tree and through DevTools) waits +until after G. The Cua browser route exists; the experiment does not. + +## 0. Documentation reconciliation (half a day, now) + +Every item is an audit finding about a claim, not about code. + +- README: Q12 is finished and not adopted as a burden reduction; Q9e did remeasure repaired + delegation (36 verified against 41 guarded); the blanket "zero false completions on every split and + run" narrows to "on the predicates of Q11 and Q12", since `li-add-then-delete-authorized` in + `q8-q9e-live-dev-20260922-162350` was claimed done with its predicate failed; "calibration + transfers" becomes the Q10b write-up's ranking result. +- `footwork run`: its default enables System 1 and escalates, which the experiments call reactive + dual. Either rename the default or add `--guarded` and make it the default; the CLI and README use + the research names. +- Q12 result and `evals.burden`: "53 tasks with checkpoints" becomes "53 tasks with some progress in + both arms (19 of 59 declare checkpoints; 6 excluded for zero progress)", and `evals.burden` prints + that population and the zero-progress counts itself (the first piece of P7). +- `legible-harness.md`: conformance cells updated to match the audit; P6's browser route marked + built and Q14 marked not run; P1 (Q12) marked complete, with exf b642f4ab closed. + +Done when: every changed sentence cites a results directory, and the README has no claim the audit +contradicts. + +## A. Session identity and truthful evidence + +Predictions A1 to A4 are written (`docs/notebook/2026-09-26-a-session-evidence.md`). + +- A1: persist the bound browser target and tab; refuse when it is gone. +- A2: a refused receipt is never upgraded; `type` confirms on equality with the intended value, + `append` on the before value plus the insertion, bound to the same field. +- A3: secret classification travels with the field through receipts, rendering, the session log and + model inputs. +- A4: replacing a non-empty document is a replacement at any length. + +Done when: each audit probe in `probes.py` has a failing-then-passing CLI scenario, the probes script +reports the corrected behaviour, and the scenario suite and ruff are green. A browser-mode scenario +needs a fake browser driver fixture; build it once here, since B and D reuse it. + +## B. Authorization parity + +- The browser path's failed destructive judgment becomes a pause, with a test that raises in the + judgment and asserts nothing dispatches (the half of P2 that needs no live run). +- A route-parity table test: the same semantic action (submit, delete, replace, confirm a default + button) through click, key, Enter, menu, hotkey and text routes, on both surfaces, gets the same + decision. Where a route cannot be judged semantically, it pauses. +- Grants bind to an operation and a resource rather than a label substring. Design first in a + prediction entry, since this changes `--authorize` for the operator. + +Done when: the parity table passes on both surfaces, an unavailable or malformed judgment dispatches +nothing anywhere, and a grant for one action cannot clear a different action whose label shares a +substring. + +## C. Trusted input after navigation + +The execution blocker behind Q11's and Q12's saucedemo losses. Start from +`scripts/diag_trusted_input.py`. + +- Build a deterministic local fixture: a page that navigates on click and a button on the next page + that counts arriving events. +- Predict the cause before touching it (session bookkeeping is the audit's leading guess, not an + established cause); keep the alternatives in the entry. +- Fix, then prove exactly-once delivery across fresh browser sessions, and run the targeted + authorized and unauthorized saucedemo tasks once each on the frozen fix as a check, not a + measurement. + +Done when: the fixture shows input arriving exactly once after navigation in repeated fresh sessions, +with no duplicated events. + +## D. Receipts that name the effect, then Q12b + +- Click receipts on a page that did not navigate name what changed: scroll position, focus, an + expanded section, a delayed navigation. Dispatch status and postcondition are separate fields. +- Pre-register Q12b: receipts delivered against recorded-only, on the repaired input path from C, + with the same primary (requests per unit of progress, its population named), all-task verified + completion, zero-progress counts and the Q12 guardrails. An input fix's gain is not credited to + receipts. + +Done when: Q12b is run and decided under its rule. If requests per unit of progress fall with a route +available, v6's L2 gains a burden claim and L3 is confirmed; if not, receipts stay an honesty measure. + +## E. Verifier evidence and recovery ownership + +Ahead of Q11b because the record shows correct or nearly correct work ending UNVERIFIED for missing +evidence, which a declaration cannot fix and a lower threshold would hide. + +- Oracle probes for save, read-back and absence (the Finder Move to Trash case at 0.70), with typed + evidence rather than page text; insufficient evidence is its own outcome. +- A pre-registered question on recovery ownership: System 1 finishes only mechanical actions whose + target, arguments, permission and expected effect are fixed; System 2 keeps repair until a named + progress condition holds. Primary: cost and latency per correctly completed, authorized task, not + System 2 call counts (Q10b: 44 calls against 130, 29% more cost). + +## F. Q11b + +As already planned: the requirements list only, plus one sentence that the done text is checked +against the page. Prediction: the implied-page gain outside saucedemo stays and the narrated refusals +go. Runs after C, so saucedemo no longer decides the result. + +## G. Cross-surface pilot, then Q13 + +One browser-to-TextEdit transfer with typed data provenance and an exact saved-file oracle, then the +five-task Q13 study. Only once A and B hold, since a handoff is where identity and authority are +lost. + +## In parallel, the person's work + +- The 40-row human audit sheets (web, Q10, Q10b) are still blank; the 12 recorded Q10b judgments count + and are a targeted sample, not a population estimate (exf dcd70da2). +- The Legible Loop: after each package, update v6's statuses from its notebook summary. The manifest + in v6 section 5 is this plan's "Closes" column. + +## Checkpoint + +After C closes, score N1 (`docs/notebook/2026-09-26-notebook-practice.md`): did each package produce +a failed or revised prediction that changed the work, and was every prediction committed before its +test? Keep, change or drop the notebook on that answer. From f4dd2187f788839f06e692bdb7f57e84c8748fb0 Mon Sep 17 00:00:00 2001 From: Tom Main Date: Sat, 26 Sep 2026 12:16:06 -0400 Subject: [PATCH 004/283] Package 0: reconcile claims with the 2026-09-26 status audit - README: Q12 finished and not adopted as a saving; Q9e did remeasure repaired delegation (36 vs 41 verified); zero false completions narrowed to the guarded arm, with the Q8 dual false completion cited; Q10 AUROC stated as a ranking result on two apps, not transferred calibration; Q15's live-overlap limit; status section points at the audit and next-work plan. - footwork run: --arm guarded|dual|s1_only, default guarded (Q10b: 15/18 vs dual 13/18 at lower cost); the old default enabled System 1 and escalated while documented as guarded. --no-s2 kept as s1_only. Behaviour change for anyone relying on the old default: pass --arm dual. - evals.burden: prints the ratio's population (tasks with progress in both arms, how many declare checkpoints) and all-task verified passes; rows now record checkpoints_declared. - Q12 result: '53 tasks with checkpoints' relabelled (19 of 59 declare checkpoints), dated correction. - legible-harness plan: status note on the cells the audit superseded. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01HiwkrvCf1sUXw4QGgEe1Un --- README.md | 46 +++++++++++++++++---------- jevdual/README.md | 4 +-- jevdual/docs/plans/legible-harness.md | 11 +++++++ jevdual/evals/burden.py | 36 ++++++++++++++++++--- jevdual/evals/report.py | 2 ++ jevdual/evals/runner.py | 1 + jevdual/python/jevdual/cli.py | 33 ++++++++++++++++--- jevdual/results/q12-live-dev.md | 11 ++++++- jevdual/tests/test_cli.py | 10 +++++- jevdual/tests/test_receipts.py | 5 +++ 10 files changed, 130 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 53b1f93..4fb4509 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,8 @@ footwork done "624" # the verifier judges the claim against the win ``` Also `do type | append | key | hotkey | menu | scroll | enter`, `status`, and `footwork run` for the whole -loop with Jev as System 1 and an LLM as System 2 on escalation. Nothing dispatches on an id from an older +loop: by default the measured **guarded** arm (an LLM decides every step behind the guard), or `--arm dual` +for Jev as System 1 with the LLM on escalation. Nothing dispatches on an id from an older observation; text comes from the task or from named secrets (`{NAME}` read from `$FOOTWORK_SECRET_NAME`, never printed); `--authorize "