AI Vision board: embeddedci.aivision client, pytest fixture, vision_* MCP tools

packages/embeddedci-mcp/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog — `embeddedci-mcp`
 
+## Unreleased
+
+- `vision_*` tools for the AI Vision board (ai-vision-2), on their own connection next to the pod's:
+  `vision_connect`, `vision_disconnect`, `vision_status`, `vision_look` (the screen as an image),
+  `vision_ocr`, `vision_find`, `vision_wait_for`, `vision_compare`, `vision_screen_health`,
+  `vision_calibrate`, `vision_add_reference`, `vision_press`, `vision_tap`. Requires `embeddedci`
+  2.13.8, the release with `embeddedci.aivision`.
+
 ## 2.13.8
 
 - Requires `embeddedci` 2.13.7. `capture_la` takes `trigger_pre_samples`, and its result reports

packages/embeddedci-mcp/README.md

@@ -344,6 +344,30 @@ the pod's second UART, so `uart_open` keeps the console. Needs gateware v48 (`gp
 | `la_step` | Step pulses for step/dir motor drivers: `delay` is half the step period (4 us to 65.535 ms), `steps` up to 65535. | **`la`**, **`steps`**, **`delay`**, `dir_la`, `direction=0` |
 | `command` | Raw firmware JSON command, for anything no tool covers. | **`request`** |
 
+### AI Vision board
+
+For a bench with an AI Vision board (ai-vision-2): a camera on the DUT's screen that reads text,
+finds buttons and presses them through its actuator. Its connection is separate from the pod's, so
+one session drives both. Without `vision_connect`, the first `vision_*` call connects to
+`AIVISION_CONNECTION` (token in `AIVISION_TOKEN`). Boxes and taps are in screen pixels once the
+screen is calibrated.
+
+| Tool | Purpose | Parameters |
+| --- | --- | --- |
+| `vision_connect` | Open the AI Vision connection (closing a previous one) and report its status. | `connection`, `token`, `lease_as` |
+| `vision_disconnect` | Close the connection and release its lease. | none |
+| `vision_status` | Version, calibration, camera lock, actuators, lease, capabilities, with warnings. | none |
+| `vision_look` | The screen as an image (an image content block), or what the camera sees; a `region` zooms in at full sensor resolution. | `space=screen`, `overlay`, `region` |
+| `vision_ocr` | All text on the screen or in a region, in reading order, with boxes. | `region`, `zoom` |
+| `vision_find` | Elements matching text, a kind (`button`, `icon`, `text`) or a template, best first. | `text`, `label`, `template`, `match`, `region` |
+| `vision_wait_for` | Wait until text appears (or with `gone`, disappears); `met: false` on timeout. | **`text`**, `timeout_s=10`, `gone=false`, `match`, `region` |
+| `vision_compare` | Similarity of the screen to a stored reference screen. | **`reference`**, `mask` |
+| `vision_screen_health` | Black, dim, frozen, flickering or stuck pixels over a window. | `window_ms=1000`, `expect_motion=false` |
+| `vision_calibrate` | Map camera pixels to screen pixels: four corners and the screen size, or `method` auto/markers. | `corners`, `screen_size`, `method` |
+| `vision_add_reference` | Store the current screen (or a region, as a template) as a reference. | **`name`**, `kind=screen`, `region`, `overwrite=false` |
+| `vision_press` | Find an element, tap its center, optionally wait for `verify_text`. Fails with the visible text when it is not there. | **`text`**, `verify_text`, `verify_timeout_s=5`, `match`, `actuator` |
+| `vision_tap` | Tap a point in screen pixels (`hold_ms` for a long press). | **`x`**, **`y`**, `hold_ms=0`, `actuator` |
+
 Resources: `benchpod://wiring` (the connected bench's own wiring profile, then LA channels, bias
 resistors and analog paths) and `benchpod://help` (the server instructions).
 

packages/embeddedci-mcp/pyproject.toml

@@ -26,7 +26,7 @@ classifiers = [
 dependencies = [
     # cloud: the embeddedci:<device> destination; discovery: connect("discover") over mDNS;
     # analysis: dominant frequency in capture summaries.
-    "embeddedci[cloud,discovery,analysis]>=2.13.7,<3",
+    "embeddedci[cloud,discovery,analysis]>=2.13.8,<3",
     "mcp>=1.12,<2",
     "anyio>=4",
     "pydantic>=2.7",

packages/embeddedci-mcp/src/embeddedci_mcp/guide.py

@@ -37,6 +37,8 @@
 Units are volts, amps, seconds and hertz. A tool that cannot do what was asked fails with an error that names the cause (for example "FirmwareError: la voltage not set; call set_la_voltage first ..." or "NotConnectedError: ..."). A completed operation with a negative outcome is a normal result: flash returns ok=false with its logs, a UART capture returns matched=false. Policy refusals have their own names: PodLockedError (the pod's LAN policy keeps the command for the cloud or USB), PodLeasedError (a cloud job holds the pod; wait and retry), PermissionDeniedError (the credential needs an owner/admin role or access to the device). UnsupportedFeatureError means this pod cannot do it at all (its firmware or gateware lacks the feature, or it is the digital-only board without the analog front end): check the capabilities in status instead of retrying.
 
 flash runs OpenOCD on the machine hosting this server and reads `file` from that machine's filesystem. To flash a build uploaded to embeddedci.com (GitHub Actions upload-artifact or the pytest build_report fixture), pass `build` (its id, or 'latest') and/or `repo`/`ref` instead of `file`: on an 'embeddedci:<pod>' connection embeddedci.com runs OpenOCD itself, on LAN/USB the file is downloaded and flashed here (needs BENCHPOD_API_KEY or `benchpod login`). Over the cloud the device is shared: connect takes an exclusive lease, released by disconnect or after the server's idle timeout (the next call reconnects).
+
+AI Vision board (ai-vision-2), when the bench has one: a camera on the DUT's screen, driven by the vision_* tools next to the BenchPod ones (a separate connection: vision_connect, or AIVISION_CONNECTION). vision_look returns the screen as an image so you can read it yourself; vision_calibrate once per fixture (corners from vision_look(space='camera')) so boxes and taps are in screen pixels; vision_find / vision_ocr locate text and buttons; vision_press finds an element, taps it with the board's actuator and can verify the next screen (verify_text); vision_wait_for waits on the board; vision_compare checks against a reference stored with vision_add_reference. A press of text that is not on the screen fails with NotFoundError and lists the visible text.
 """
 
 WIRING = """\

packages/embeddedci-mcp/src/embeddedci_mcp/server.py

@@ -2094,3 +2094,10 @@ def wiring_resource() -> str:
               description="The server instructions: session start, typical flows, error contract.")
 def help_() -> str:
     return INSTRUCTIONS
+
+
+# -- AI Vision board ------------------------------------------------------------------------
+
+from . import vision as _vision  # noqa: E402  (registers the vision_* tools on this server)
+
+_vision.register(mcp)