From 2d1350fac843a9c8b8dde14a2d84e4dd1a9833c1 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 9 Oct 2026 15:14:42 -0700 Subject: [PATCH 1/2] Plan the Voice AI demo and its buffer view Co-Authored-By: Claude Opus 5.5 --- quest/README.md | 1 + quest/m1/README.md | 14 +++++++++ quest/m1/voice-api-moqdev.md | 17 +++++++++++ quest/m1/voice-buffers.md | 38 +++++++++++++++++++++++ quest/m1/voice-page.md | 48 ++++++++++++++++++++++++++++++ quest/m1/voice-timeline-track.md | 16 ++++++++++ quest/m1/watch-arrivals-release.md | 17 +++++++++++ 7 files changed, 151 insertions(+) create mode 100644 quest/m1/README.md create mode 100644 quest/m1/voice-api-moqdev.md create mode 100644 quest/m1/voice-buffers.md create mode 100644 quest/m1/voice-page.md create mode 100644 quest/m1/voice-timeline-track.md create mode 100644 quest/m1/watch-arrivals-release.md diff --git a/quest/README.md b/quest/README.md index 0251850..427c8b7 100644 --- a/quest/README.md +++ b/quest/README.md @@ -16,3 +16,4 @@ its scope, until it is scheduled. ## Required - [m0: immediate priorities](/quest/m0/README.md) - the @moq bump for draft-18 relays, the live deploy, and hosted worklets +- [m1: next](/quest/m1/README.md) - the Voice AI demo page and its live buffer view diff --git a/quest/m1/README.md b/quest/m1/README.md new file mode 100644 index 0000000..31751e0 --- /dev/null +++ b/quest/m1/README.md @@ -0,0 +1,14 @@ +# m1: next + +## Goal + +The demos after m0: a Voice AI demo you can talk to, which shows why MoQ beats +WebRTC for voice agents. + +## Required + +- [moq.pro accepts moq.dev](/quest/m1/voice-api-moqdev.md) - condition: the wtf voice API and Turnstile widget serve the moq.dev origin +- [Voice AI demo page](/quest/m1/voice-page.md) - talk to a Pipecat agent over MoQ at `/voice`, with a live transcript and a `/demo` card +- [An @moq/watch release ships per-frame arrivals](/quest/m1/watch-arrivals-release.md) - condition: the reply-side data is on npm +- [The voice agent publishes voice.json](/quest/m1/voice-timeline-track.md) - condition: word timings and prompt-side arrivals on wtf +- [Voice buffers under the words](/quest/m1/voice-buffers.md) - the live transcript draws each word's audio frames, marking the ones WebRTC would have dropped diff --git a/quest/m1/voice-api-moqdev.md b/quest/m1/voice-api-moqdev.md new file mode 100644 index 0000000..58710e4 --- /dev/null +++ b/quest/m1/voice-api-moqdev.md @@ -0,0 +1,17 @@ +# [XS] moq.pro accepts moq.dev + +## Goal + +The staging voice demo API (`api.moq.wtf/voice/sessions`) accepts a session +started from `https://moq.dev`, `https://new.moq.dev`, and +`http://localhost:4321`: the wtf Turnstile widget lists those hostnames and +the API answers their CORS preflight. + +This quest tracks a condition outside the repository. When it holds, delete +this quest and every `Required` entry that links it. + +## Plan + +moq.pro tracks the change as +[its own quest](https://github.com/moq-dev/moq.pro/blob/main/quest/m1/voice-moqdev.md). +Check by starting a session from `just dev` against `api.moq.wtf`. diff --git a/quest/m1/voice-buffers.md b/quest/m1/voice-buffers.md new file mode 100644 index 0000000..17050c7 --- /dev/null +++ b/quest/m1/voice-buffers.md @@ -0,0 +1,38 @@ +# [M] Voice buffers under the words + +## Goal + +Under each word of the live `/voice` transcript, a strip draws that word's +audio frames from real measurements: when each arrived against when it plays. +A frame that arrived later than a WebRTC jitter buffer would have waited +turns red, because WebRTC would have concealed or skipped it. Both +directions: your prompt as the agent received it, and the agent's reply as +your browser received it. Nothing is simulated or injected. + +## Plan + +Decided 2026-10-09: + +- **Real data only.** No "network blip" button and no injected loss. On a + clean network the strips are all on time, and the reply strip still shows + how far ahead the agent sent its audio. +- **A WebRTC deadline line, computed from the real arrivals.** A frame is + "late for WebRTC" when its arrival spread exceeds a typical WebRTC jitter + target. Measure lateness as arrival minus timestamp, relative to the + smallest one seen, so the sender's and receiver's clocks never need to + agree. Choose and cite the target when implementing. +- **Draw it like `src/components/voice/`.** Reuse its segment colors and + word layout on a sliding window of live, append-only data, in place of its + fixed axis and simulated `FLIGHT`. +- **Reply side** reads `@moq/watch`'s per-frame arrival signal (a bounded + window of recent `{timestamp, arrival}` per frame, with late and skipped + marks) and maps frames to words with the agent's TTS word timings. +- **Prompt side** reads the agent's `voice.json` track on the response + broadcast: word timings from STT and the arrival time of each request + audio frame. + +## Required + +- [Voice AI demo page](/quest/m1/voice-page.md) - the page and transcript this draws under +- [An @moq/watch release ships per-frame arrivals](/quest/m1/watch-arrivals-release.md) - the reply-side data +- [The voice agent publishes voice.json](/quest/m1/voice-timeline-track.md) - word timings and the prompt-side data diff --git a/quest/m1/voice-page.md b/quest/m1/voice-page.md new file mode 100644 index 0000000..50e62e8 --- /dev/null +++ b/quest/m1/voice-page.md @@ -0,0 +1,48 @@ +# [M] Voice AI demo page + +## Goal + +`/voice` lets a reader talk to the moq.pro Pipecat agent over MoQ and shows a +live transcript of both sides. `/demo` gets a card for it. The component is +reusable, so a blog post can embed it later; the post itself (branch +`voice-api-2`) is not part of this quest. + +## Plan + +Decided 2026-10-09: + +- **A native Solid component, not moq.pro's iframe.** The buffer view needs + the player's internals, which an iframe from another site hides, and demos + belong in moq.dev. moq.pro keeps its own `/voice` page. +- **It talks to the moq.pro session API directly.** + `POST /voice/sessions {turnstileToken}` returns `{id, relayUrl, + requestBroadcast, responseBroadcast, token, expiresAt}`. One token + publishes the request and subscribes the response. Poll + `GET /voice/sessions/:id` until the agent claims it, and render refusals by + `code` (`busy`, `rate_limit`, `daily_budget`, ...). + moq.pro's `app/src/lib/VoiceDemo.svelte` is the reference client. + The older component on `voice-ai2` / `codex/voice-demo-embed` predates this + contract (`publishToken`/`watchToken`, no poll), so port its Turnstile + loader and layout, not its session code. +- **Live points at staging.** The demo only runs on `api.moq.wtf` until + moq.pro's production quest lands, so a separate `PUBLIC_VOICE_API_URL` is + `https://api.moq.wtf` for live and staging, and `PUBLIC_TURNSTILE_SITE_KEY` + is the wtf widget's public key. `PUBLIC_API_URL` stays on its environment's + API for `try`. +- **No Pipecat client.** `` sends the mic and `` + plays the reply with a latency ceiling near the agent's 25s send-ahead. + The page does the small part of RTVI the agent needs itself: publish a + `transcript.json.z` track carrying `client-ready`, and read the agent's + `transcript.json.z` (not in the catalog; subscribe by name) for + transcripts. On barge-in the agent retires `bot-audio` for + `bot-audio-` through the catalog, which `` follows. +- The transcript renders the way `src/components/voice/` lays out words, so + [Voice buffers under the words](/quest/m1/voice-buffers.md) can draw under + them. + +Verify against staging in `just dev`: a conversation, barge-in, Stop, the +five-minute limit, and a `busy` refusal. + +## Required + +- [moq.pro accepts moq.dev](/quest/m1/voice-api-moqdev.md) - the wtf voice API and Turnstile widget serve the moq.dev origin diff --git a/quest/m1/voice-timeline-track.md b/quest/m1/voice-timeline-track.md new file mode 100644 index 0000000..408693c --- /dev/null +++ b/quest/m1/voice-timeline-track.md @@ -0,0 +1,16 @@ +# [XS] The voice agent publishes voice.json + +## Goal + +The moq.pro demo agent on wtf publishes a `voice.json` track on each session's +response broadcast, carrying word timings for both directions and the arrival +time of each request audio frame. + +This quest tracks a condition outside the repository. When it holds, delete +this quest and every `Required` entry that links it. + +## Plan + +moq.pro tracks the track as +[its own quest](https://github.com/moq-dev/moq.pro/blob/main/quest/m1/voice-timeline.md), +which owns its schema. Check by subscribing to it during a staging session. diff --git a/quest/m1/watch-arrivals-release.md b/quest/m1/watch-arrivals-release.md new file mode 100644 index 0000000..6415f23 --- /dev/null +++ b/quest/m1/watch-arrivals-release.md @@ -0,0 +1,17 @@ +# [XS] An @moq/watch release ships per-frame arrivals + +## Goal + +The newest `@moq/watch` on npm exposes a signal with a bounded window of +recent per-frame arrivals for the audio and video decoders, and moq.dev +builds against it. + +This quest tracks a condition outside the repository. When it holds, bump +`@moq/watch` here, then delete this quest and every `Required` entry that +links it. + +## Plan + +moq tracks the signal as +[its own quest](https://github.com/moq-dev/moq/blob/main/quest/m1/watch-arrivals.md). +Check `npm view @moq/watch version` and its type definitions. From ad7970c47fdf7b852089402c159641fb2a0c61ec Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 14:25:30 -0700 Subject: [PATCH 2/2] Describe the shipped arrivals() handle in the voice buffer quests Co-Authored-By: Claude Opus 5.5 --- quest/m1/voice-buffers.md | 9 ++++++--- quest/m1/watch-arrivals-release.md | 12 ++++++------ 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/quest/m1/voice-buffers.md b/quest/m1/voice-buffers.md index 17050c7..b4c2ec9 100644 --- a/quest/m1/voice-buffers.md +++ b/quest/m1/voice-buffers.md @@ -24,9 +24,12 @@ Decided 2026-10-09: - **Draw it like `src/components/voice/`.** Reuse its segment colors and word layout on a sliding window of live, append-only data, in place of its fixed axis and simulated `FLIGHT`. -- **Reply side** reads `@moq/watch`'s per-frame arrival signal (a bounded - window of recent `{timestamp, arrival}` per frame, with late and skipped - marks) and maps frames to words with the agent's TTS word timings. +- **Reply side** reads `@moq/watch`'s per-frame `arrivals()` handle on the + audio decoder and maps frames to words with the agent's TTS word timings. + It yields each frame's `{timestamp, arrival, late}` and a gap item + (`timestamp` and `duration` in `Time.Milli`) where media will never + arrive. Its queue is unbounded, so the page keeps its own window. There is + no gap at the join: the strip starts on the first frame read. - **Prompt side** reads the agent's `voice.json` track on the response broadcast: word timings from STT and the arrival time of each request audio frame. diff --git a/quest/m1/watch-arrivals-release.md b/quest/m1/watch-arrivals-release.md index 6415f23..c75356f 100644 --- a/quest/m1/watch-arrivals-release.md +++ b/quest/m1/watch-arrivals-release.md @@ -2,9 +2,8 @@ ## Goal -The newest `@moq/watch` on npm exposes a signal with a bounded window of -recent per-frame arrivals for the audio and video decoders, and moq.dev -builds against it. +The newest `@moq/watch` on npm exposes the per-frame `arrivals()` handle on +the audio and video decoders, and moq.dev builds against it. This quest tracks a condition outside the repository. When it holds, bump `@moq/watch` here, then delete this quest and every `Required` entry that @@ -12,6 +11,7 @@ links it. ## Plan -moq tracks the signal as -[its own quest](https://github.com/moq-dev/moq/blob/main/quest/m1/watch-arrivals.md). -Check `npm view @moq/watch version` and its type definitions. +moq shipped the handle in +[moq-dev/moq#5311](https://github.com/moq-dev/moq/pull/5311). +Check `npm view @moq/watch version` and its type definitions for +`Audio.Decoder.arrivals()` and `Watch.Arrival`.