Skip to content

Let a QA reviewer point and talk instead of typing - #34

Merged
savvasioakeim merged 1 commit into
mainfrom
feat/qa-capture-and-dictation
Oct 31, 2026
Merged

savvasioakeim merged 1 commit into
mainfrom
feat/qa-capture-and-dictation

Conversation

@savvasioakeim

Copy link
Copy Markdown
Owner

Writing a QA note meant leaving the app under test, describing a spot in words, and typing it out. This adds two ways in, both landing on the same item's result.

Point and capture

Ctrl+Alt+Q grabs the monitor under the pointer, rings where you were pointing, and attaches a full frame plus a detail crop.

The trigger is an OS-level shortcut rather than a DOM keydown because by definition the app under test has focus, not ours. Alt+Shift was avoided deliberately: it switches keyboard layout on Windows.

Two non-obvious points:

  • Pointer-to-pixel mapping uses the ratio of captured pixels to reported monitor size rather than subtracting an origin, so it stays correct under per-monitor DPI scaling.
  • xcap needs the wgc feature. Without it the backend is GDI BitBlt, which returns an all-black frame for anything the compositor draws on the GPU. The geometry is identical either way, so the failure is silent: a correctly marked screenshot of nothing. The ignored test prints the geometry and asserts a real marker pixel for exactly that reason.

Previews come back as base64 thumbnails beside each path. The asset protocol is not enabled in this app, and turning it on just for a thumbnail would mean a new protocol plus scope management.

Talk instead

The dictation that existed only in the chat input is now a shared hook in src/voice, with a button under the QA notes.

Pulling it out of InputBar surfaced two things it had been hiding:

  • On failure it wrote # (speech-to-text is not supported in this WebView) into the user's own input box.
  • onerror discarded the error code entirely.

With the code surfaced, we learned the Web Speech API has been failing with network since WebView2 Runtime 153 (it worked in 152, WebView2Feedback#5724). Confirmed here on Runtime 155. That is not fixed in this PR and is not ours to fix; the settings copy now says so plainly instead of claiming the engine works, and Whisper is listed as not available yet. A local Whisper engine is the follow-up.

One latent bug fixed along the way

resultsRef was assigned during render, so handlers read it one render stale. Screenshots got away with it by arriving rarely. Dictated phrases arrive about a second apart, and two close together would both read the same old notes, with the second overwriting the first. commitResult now writes ref, state and event together.

Verification

  • cargo check and cargo clippy clean — no new warnings from capture.rs; the ~20 existing ones are pre-existing elsewhere.
  • npx tsc --noEmit and npm run build pass.
  • The ignored shoots_where_the_pointer_is test (needs a desktop session) confirmed the ring lands on the pointer with real, non-black pixels.
  • The dictation button was exercised in both windows: it reaches the service and reports the WebView2 failure, which is the current correct behaviour. End-to-end transcription cannot be verified until that service works or Whisper lands.

🤖 Generated with Claude Code

Writing a QA note meant leaving the app under test, describing a spot in
words, and typing. Two ways in, both landing on the same item's result.

Point and capture. Ctrl+Alt+Q grabs the monitor under the pointer, rings
where you were pointing, and attaches a full frame plus a detail crop. The
trigger is an OS-level shortcut rather than a DOM keydown because by
definition the app under test has focus, not ours. Alt+Shift was avoided on
purpose: it switches keyboard layout on Windows.

Two things about the capture were not obvious. Pointer-to-pixel mapping is
done by the ratio of captured pixels to reported monitor size rather than by
subtracting an origin, so it stays correct under per-monitor DPI scaling.
And xcap needs the `wgc` feature: without it the backend is GDI BitBlt,
which returns an all-black frame for anything the compositor draws on the
GPU. The geometry is identical either way, so that failure is silent, and
the ignored test prints the geometry and asserts a real marker pixel for
exactly that reason.

Previews come back as base64 thumbnails beside each path. The asset protocol
is not enabled in this app, and turning it on for a thumbnail would mean a
new protocol plus scope management.

Talk instead. The dictation that existed only in the chat input is now a
shared hook in src/voice, with a button under the QA notes. Pulling it out
of InputBar also fixed two things it had been hiding: a failure wrote
"(speech-to-text is not supported in this WebView)" into the user's own
input box, and `onerror` discarded the error code entirely. It now says
which failure happened, which is how we learned the Web Speech API has been
failing with `network` since WebView2 Runtime 153 (it worked in 152,
WebView2Feedback#5724). That is not fixed here. The settings copy says so
plainly rather than claiming the engine works, and Whisper stays listed as
not available yet.

One latent bug came out of this. `resultsRef` was assigned during render, so
handlers read it one render stale. Screenshots got away with it by arriving
rarely; dictated phrases arrive about a second apart, and two close together
would both read the same old notes, the second overwriting the first.
`commitResult` now writes ref, state and event together.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@savvasioakeim
savvasioakeim merged commit 57fb63a into main Oct 31, 2026
3 checks passed
@savvasioakeim
savvasioakeim deleted the feat/qa-capture-and-dictation branch October 31, 2026 18:29
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 31, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant