Skip to content

test(cockpit): run every React cockpit example on aimock + real backends - #1680

Open
blove wants to merge 42 commits into
mainfrom
blove/react-cockpit-aimock-all
Open

blove wants to merge 42 commits into
mainfrom
blove/react-cockpit-aimock-all

Conversation

@blove

@blove blove commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This moves the remaining 40 React cockpit examples onto aimock and real backends, the way the Angular examples already run. It follows the pilot, #1661 (chat/messages). React e2e now replays the shared Angular fixture JSON (with chunkSize and latency) through langgraph dev, the real AG-UI servers, or the runtime services. A recording proxy carries the wire inspection and the labelled fault injection. All authored fixture servers are gone from scripts/react-cockpit/serve.mjs.

This PR combines the closed stacked drafts #1662–#1674, rebased onto main. Each batch is still its own commit, in stack order, so it can be reviewed commit by commit:

Batch Examples Original draft
1 chat/input, chat/theming #1662
2 chat/tool-calls, subagents, interrupts #1663
3 chat/threads, timeline #1664
4a langgraph streaming, interrupts, memory, persistence, subgraphs #1665
4b langgraph client-tools, deployment-runtime #1666
4c langgraph time-travel, durable-execution #1667
5 deep-agents ×5, including the Angular memory e2e on the shared fixtures #1668
3b chat/debug, including the native-session probe #1669
6a ag-ui streaming, interrupts, tool-views, json-render, subagents #1670
6b ag-ui/client-tools #1671
2b chat/generative-ui #1672
6c ag-ui/a2ui, chat/a2ui, including the packed-consumer probe #1673
7 runtimes aws-strands, MAF, mastra #1674

Website previews that drive a backend moved with their examples, in the same commits. The design doc is docs/superpowers/specs/2026-11-18-react-cockpit-aimock-design.md.

Rebase notes

Injected data and dropped cases

aimock can't produce some faults through model output. Those are injected at the recording proxy and labelled as injected data in each spec. Examples:

  • reasoning text (/__inject/reasoning);
  • stream edits, cuts and holds;
  • history and state rewrites.

The real failures stay covered by each example's Python tests.

Three chat/generative-ui cases are unreachable from model output: the prose fallback, a failing KPI tool and malformed trend data. They are dropped from e2e and covered in cockpit/chat/generative-ui/python/tests/test_streaming.py.

The network-free compiled-graph proofs stay in each example's socket-patched python/tests/test_streaming.py, which nx e2e depends on. The full per-batch list is in the design doc.

Verification (local, at the tip)

  • ci-guards suites: all pass, including ci-workflow.spec.mjs (208/208).
  • All changed and scripts/react-cockpit / scripts/react-parity spec files pass (45 files).
  • nx affected -t lint,test,typecheck,type-tests,build: the only stack-specific failure was cockpit-langgraph-subgraphs-react:test. Two batch 4a tests still built child snapshots with values, which main renamed to state; fixed in the last commit. The other failures reproduce on plain main locally (website, growth-research, design-tokens, runtime-bridge, render-repeat-loops, workspace-react build), and chat:test passes when run alone.
  • I didn't rerun the React e2e suites locally after the rebase. They were green per batch before it, and this PR's CI covers them.

Website e2e wall time "after" will come from this PR's CI. "Before" (the last three main runs): chat 7.3–10.2 min, agents 8.6–11.5, control-plane 12.9–18.8, ag-ui 6.5–8.6, render 5.9–6.5.

Supersedes #1662–#1674.

🤖 Generated with Claude Code

blove and others added 30 commits November 19, 2026 11:35
Batch 1 of the React cockpit aimock migration
(docs/superpowers/specs/2026-11-18-react-cockpit-aimock-design.md).

- chat/input and chat/theming React e2e replay the shared Angular fixtures
  through a real `langgraph dev`; holds are proxy controls, not magic prompts.
- Proxy: /__captures (forwarded SSE, saved messages/next from the real
  /state, title before/after each run), openResponses in /__lifetime, and
  /__inject/reasoning (documented injected data: ChatOpenAI over Chat
  Completions never surfaces reasoning; the theming palette checks need it).
- Shared reactAimockGlobalSetup({ example, port }) plus a spec keeping it in
  step with configuration.mjs; nx.json `reactAimockBackend` named input so
  proxy/server changes select every migrated React cap.
- Fixtures: title-model fixture, the shared 3-char fence fixture, and a
  catch-all reply (last) for client-only prompts.
- Delete chat-input/chat-theming fixture servers, the theming fixture spec
  and its CI step; the network-free graph proof stays in each example's
  Python tests, which e2e still depends on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…injection

- `serve.mjs <example> --backend` starts that example's aimock + `langgraph
  dev` (scripts/react-cockpit/aimock-backend.mjs) lazily on the first app or
  API request; Website suites start only the preview servers they own.
  `/__ready` waits out the lazy start.
- Website Theming preview runs on it; its assertions use the proxy's
  requests, captures (saved messages, title) and lifetime.
- /__inject/reasoning is opt-in per test: only the cockpit theming test that
  styles the Reasoning region calls it. Injected because langchain_openai's
  ChatOpenAI does not extract `reasoning_content` from Chat Completions.
- Design doc: website previews, the injected-reasoning rationale and the
  Responses API follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e config spec

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aimock

Batch 2 of the React cockpit aimock migration
(docs/superpowers/specs/2026-11-18-react-cockpit-aimock-design.md).

- Shared Angular fixtures gain React prompts whose tool rounds key on aimock
  toolCallId; subagent child calls key on role prompt + task text; a
  mid-stream child model failure is aimock truncateAfterChunks; a title
  (metadata) failure is a per-prompt error fixture.
- Proxy: /__hold/stream { beforeToolResult | afterChildMessage },
  /__fail-stream, /__child/<role|task|error> (injected, as before), tool and
  approval /__history modes, /__ready.
- Delete the three fixture servers, their wire workers and fixture specs;
  e2e depends on each example's network-free Python graph test directly.
- chat/generative-ui moves to batch 2b: its faults swap graph internals that
  aimock cannot produce.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batch 3 of the React cockpit aimock migration
(docs/superpowers/specs/2026-11-18-react-cockpit-aimock-design.md).

- chat/threads and chat/timeline React e2e replay the shared Angular
  fixtures through a real `langgraph dev`; titles come from the real
  generate_title node, checkpoints/history/state from the real server.
- Proxy: thread-metadata title modes and /__hold/title, checkpoint-state
  modes and /__hold/state, one-shot /__hold/history, mapped checkpoint
  representation, foreign/failure history modes. /__captures no longer
  delays the stream end (the browser closed it first); it awaits captures
  in flight instead.
- Timeline stream evidence comes from /__captures[].sse: DevTools cannot
  return a streamed body the client closes.
- Delete both fixture servers, wire workers and fixture specs.
- chat/debug moves to batch 3b: its fixture also backs the packed-consumer
  native-session proof.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tence/subgraphs e2e on aimock

Batch 4a of the React cockpit aimock migration
(docs/superpowers/specs/2026-11-18-react-cockpit-aimock-design.md).

These React examples previously ran against pure-JS authored servers with no
graph at all; they now run the real graphs under `langgraph dev`, replaying
the shared Angular fixtures, and assertions follow the real graphs' output.

- Proxy: immediate holds (resumes, pre-router), /__interrupt/<unknown|
  multiple>, /__values (next run's values/updates/checkpoints state, with
  optional persist to the real thread), /__history-values, `paused` history
  mode, and /__stream/edit (drop/patch/omit/insertAfter ops; checkpoint ops
  address `values`) for stream shapes a real graph will not produce.
- Fix: the React subgraphs app rejected every real nested route, because a
  real server streams the child's __start__ checkpoint with `values: {}`;
  only present-but-malformed child keys are unsafe now (unit-tested).
- Delete the inline authored streaming handler from serve.mjs and the
  interrupts/memory/persistence/subgraphs fixture servers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tence/subgraphs previews on aimock

The Website previews (and the learning workbench's React session) for the
examples batch 4a moved onto aimock now start their backend through
`serve.mjs <example> --backend` and use the proxy controls:
holds via /__hold/stream (afterText, immediate for the resume,
afterChildMessage) and /__hold/history, lifetime via activeStream /
activeHistory, and /__ready before driving a preview.

Also: serve.mjs reads a static file before answering, so a missing build
output is a 404 instead of a crash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the config spec

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e on aimock

Batch 4b of the React cockpit aimock migration
(docs/superpowers/specs/2026-11-18-react-cockpit-aimock-design.md).

- Both examples run the real graphs under `langgraph dev` on the shared
  Angular fixtures; client tool rounds key on prompt + hasToolResult.
- Proxy: hold/fail the client's real state write (POST /threads/<id>/state),
  /__thread/<id> unrecorded state read, and /__stream/edit list ops
  (append, truncate, duplicateLast, copy) for deployment-runtime's nine
  finalization faults. "No final values" also strips the markers from the
  history read the session falls back to.
- Delete both fixture servers. time-travel and durable-execution move to
  batch 4c (their specs encode the authored checkpoint layout).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eviews on aimock

Both previews start their backend through `serve.mjs <example> --backend`,
wait on /__ready, and use /__hold/stream and activeStream from the proxy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s' aimock backend flags in the config spec

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on aimock

Batch 4c of the React cockpit aimock migration
(docs/superpowers/specs/2026-11-18-react-cockpit-aimock-design.md); all nine
langgraph React examples now run real graphs under `langgraph dev`.

- time-travel reads checkpoint ids from real history and thread reads
  instead of authored ids (each turn saves input, start and done
  checkpoints); a competing tip is a real state write; fork result faults
  edit the stream and the fork's confirmation state read.
- durable-execution: uncertain outcomes end the browser's stream after the
  first draft (optionally withholding the run identity or reporting a known
  run as running) and keep history inconclusive until a one-shot rewrite
  publishes the saved outcome.
- Proxy: /__history-copies, /__state/unsafe, unrecorded thread state writes,
  /__stream/edit { state, anonymous, unlessKey }, /__hold/stream
  { afterPattern, end }, /__run-status/<s>, /__history-values { entry },
  mapped checkpoint extras (maps sent back are stripped before forwarding).
  The edit engine is shared between stream events and state reads.
- client-tools: the slow-handler test waits 15s for the deliberate ~3s
  handler plus a real follow-up run.
- Delete both fixture servers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iews on aimock

- time-travel selects turn A's completed checkpoint from the history page the
  app just loaded (real history saves input, start and done checkpoints).
- durable-execution sets up the uncertain run (anonymous, early end, pending
  history), waits for the app's own history read, then publishes the saved
  outcome with a one-shot history rewrite before holding the status read.
- Restore the proxy's `ready` option lost in the restack (every /api request
  failed with it undefined).
- aimock-backend close() stops the server processes under `uv run`, which
  does not forward signals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… aimock backend flags in the config spec

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…website preview on aimock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bsite preview on aimock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te preview on aimock

The React authored memory fixture also backed the Angular memory e2e. Both
now replay shared aimock fixtures into a real `langgraph dev` with a real
store behind the recording proxy. Store-read failures inside the graph
(fail-initial/model/terminal) become a proxy stream cut (/__stream/cut,
injected); oversize stays a genuine failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n aimock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w on aimock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Angular memory e2e on proxy changes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e them; warm next dev routes

Previews whose Website specs send a message (or use proxy controls) now wait
on GET /__ready as their webServer readiness URL, so aimock + `langgraph dev`
boot before tests rather than mid-run beside other workers' page renders.
Static asset requests no longer start a backend, so render-only previews never
boot one. A globalSetup visits the shared docs route in every workspace mode
once before workers start (`next dev` compiles on first request). The Agents
startup pool follows the deep-agents preview commands batch 5 changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd /__ready readiness URLs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ession probe on aimock (batch 3b)

The debug fixture server and its Python wire are deleted. The cockpit e2e and
the website preview replay shared c-debug.json fixtures into a real
`langgraph dev` behind the recording proxy. The packed-consumer probe runs the
installed @threadplane/langgraph against the same backend: two submits, two
loads, exact checkpoints, code-point analysis of an astral answer, and a
transport omission of the process callback. It now proves the `langgraph dev`
it started is gone after cleanup, in place of the fixture's worker journal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…change

The debug proof step no longer runs the deleted fixture spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…imock + the real AG-UI server

Adds the AG-UI flavor of the backend and proxy: createAimockBackend runs the
example's uvicorn src/server.py for `aimock-ag-ui` backends, the e2e harness
gains a backend-only AG-UI global setup, and scripts/react-cockpit/ag-ui-proxy.mjs
records agent POSTs and holds, fails or edits the next run (protocol fault
shapes are injected; a model failure is real and drops the stream).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aimock + the real native AG-UI endpoint

The real /agent/native endpoint pauses with an AG-UI 1.0 interrupt outcome
and resumes from `resume`; the spec reads the real interrupt id from the
draft run. The proxy gains `omit`, `outcome` (multiple, malformed, expired,
child) and two more snapshot modes for the injected protocol faults.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aimock + the real AG-UI server

The real graph calls the real weather tool; an unknown tool and a call
without a location are real tool failures. The proxy gains snapshot modes for
the injected result faults, and `persist` for a backend that saves results
under another identity (mapped back on the way upstream).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… aimock + the real AG-UI server

The scripted dashboard model of the Python tests becomes shared aimock
fixtures; the spec reads each run's final snapshot from /__captures. The
proxy gains setPath, replace and four more snapshot modes for the injected
faults the authored fixture applied to the real graph's events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…imock + the real AG-UI server

The scripted orchestrator and specialists of the Python tests become shared
aimock fixtures (the orchestrator keys on each specialist's result); the
Booking specialist's failure is a real provider error. The proxy gains
dataPattern on ops, and a held run no longer blocks /__captures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blove and others added 11 commits November 19, 2026 11:35
…kends in the frontend matrix spec

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n aimock + the real AG-UI server

The cockpit scenario of the react-parity client-tools server becomes shared
aimock fixtures; that scenario and the React fixture wrapper, its preparation
script, target, specs and CI steps are removed (the parity server itself
stays for scripts/react-parity verification). The causal checks read what
the model received from aimock's request journal: AG-UI backends get a fixed
aimock port (app port + 2000) and the proxy serves /__model-requests. The
proxy's /__requests entries now carry live events, held and closed, and holds
take beforeType, persist and fromRequest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…I examples into 6c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ite preview on aimock (batch 2b)

The decision, interpretation and title models of the old wire script become
shared aimock fixtures, keyed on each node's system prompt; admission, the
real dashboard tools, assessment and publication run in the real graph.
Missing and invalid decisions and model failures are real model output.
Faults no model output reaches (a throwing tool, malformed tool data, prose
on the action parent) are dropped here: admission validates every argument
and binding before execution and creates an empty parent. The Python tests
that cover them are named in the specs. Saved-turn faults on the
confirmation read are new proxy state modes (injected data).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… + the real AG-UI server

The search, select and modify turns replay the react-parity A2UI model's own
outputs for the flow (captured from its traces); the opening form is the
recorded Angular fixture. The React fixture wrapper, its spec, the
preparation script's --serve mode and its manager stage are removed. The
installed-application proof and the parity server stay unchanged. The proxy
gains /__hold/off for persistent holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…site preview on aimock (batch 6c)

The old wire's model calls become aimock fixtures keyed on each node's full
system prompt, replayed into a real `langgraph dev` behind the recording
proxy. The packed-consumer probe (fixture-test and the Library lane step) runs
the installed package and the compiled admission helpers against the same
backend; its harness tests move to chat-a2ui-native-probe.spec.mjs. The
fixture spec's graph cases (swallowed title failure, one structured-output
retry, fallback after three invalid answers) are reached through tagged model
output and run in the React e2e. The fixture server, its Python wire, and its
fixture, host and readiness specs are deleted. createAimockBackend.close()
now stops uv's whole process tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the real servers (batch 7)

Strands and Microsoft Agent Framework run the example's own uvicorn server and
Mastra the deployments/ag-ui-mastra service (new aimock-mastra backend; the
proxy adds its internal token), each with aimock replaying the model. The
react-parity loopback models become fixtures keyed on the React preset;
scenario tags replace /__reset { scenario }; terminal corruptions and holds
are proxy edits. The native fixture servers, their parity dirs, preparation
targets and CI steps are deleted. The AG-UI proxy records what the browser got
(`sent`) beside what the server streamed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k stack onto main

The rebased stack removes only the authored fixture ownership steps
(client tools, MAF, Strands, Theming, Debug) from the cockpit-e2e job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@blove
blove enabled auto-merge November 19, 2026 20:32
#1637 rewrote the React threads e2e against the authored fixture, which this
branch deletes. The real `langgraph dev` serves thread search and the Store
API natively, so the tests now run against it:
- the proxy logs sidebar directory traffic apart from /__requests and serves
  it at /__directory-requests, as the fixture did;
- an opt-in /__purge clears the run's isolated server before each threads
  test, since the sidebar lists every thread;
- /__title/<mode> rewrites also apply to thread search results, where the
  sidebar reads titles (injected data);
- row lookups follow the server's most-recently-updated-first order, and
  title assertions read the title span, not the updated-at <time>;
- the three title tests #1637 removed stay removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant