Repository navigation
Plan the Voice AI demo and its buffer view #173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
kixelated
wants to merge
2
commits into
main
Choose a base branch
from
claude/voice-ai-demo-buffers-4043d9
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # m1: next | ||
|
|
||
| ## Goal | ||
|
|
||
| The demos after m0: a Voice AI demo you can talk to, which shows why MoQ beats | ||
| WebRTC for voice agents. | ||
|
|
||
| ## Required | ||
|
|
||
| - [moq.pro accepts moq.dev](/quest/m1/voice-api-moqdev.md) - condition: the wtf voice API and Turnstile widget serve the moq.dev origin | ||
| - [Voice AI demo page](/quest/m1/voice-page.md) - talk to a Pipecat agent over MoQ at `/voice`, with a live transcript and a `/demo` card | ||
| - [An @moq/watch release ships per-frame arrivals](/quest/m1/watch-arrivals-release.md) - condition: the reply-side data is on npm | ||
| - [The voice agent publishes voice.json](/quest/m1/voice-timeline-track.md) - condition: word timings and prompt-side arrivals on wtf | ||
| - [Voice buffers under the words](/quest/m1/voice-buffers.md) - the live transcript draws each word's audio frames, marking the ones WebRTC would have dropped |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # [XS] moq.pro accepts moq.dev | ||
|
|
||
| ## Goal | ||
|
|
||
| The staging voice demo API (`api.moq.wtf/voice/sessions`) accepts a session | ||
| started from `https://moq.dev`, `https://new.moq.dev`, and | ||
| `http://localhost:4321`: the wtf Turnstile widget lists those hostnames and | ||
| the API answers their CORS preflight. | ||
|
|
||
| This quest tracks a condition outside the repository. When it holds, delete | ||
| this quest and every `Required` entry that links it. | ||
|
|
||
| ## Plan | ||
|
|
||
| moq.pro tracks the change as | ||
| [its own quest](https://github.com/moq-dev/moq.pro/blob/main/quest/m1/voice-moqdev.md). | ||
| Check by starting a session from `just dev` against `api.moq.wtf`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # [M] Voice buffers under the words | ||
|
|
||
| ## Goal | ||
|
|
||
| Under each word of the live `/voice` transcript, a strip draws that word's | ||
| audio frames from real measurements: when each arrived against when it plays. | ||
| A frame that arrived later than a WebRTC jitter buffer would have waited | ||
| turns red, because WebRTC would have concealed or skipped it. Both | ||
| directions: your prompt as the agent received it, and the agent's reply as | ||
| your browser received it. Nothing is simulated or injected. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-10-09: | ||
|
|
||
| - **Real data only.** No "network blip" button and no injected loss. On a | ||
| clean network the strips are all on time, and the reply strip still shows | ||
| how far ahead the agent sent its audio. | ||
| - **A WebRTC deadline line, computed from the real arrivals.** A frame is | ||
| "late for WebRTC" when its arrival spread exceeds a typical WebRTC jitter | ||
| target. Measure lateness as arrival minus timestamp, relative to the | ||
| smallest one seen, so the sender's and receiver's clocks never need to | ||
| agree. Choose and cite the target when implementing. | ||
| - **Draw it like `src/components/voice/`.** Reuse its segment colors and | ||
| word layout on a sliding window of live, append-only data, in place of its | ||
| fixed axis and simulated `FLIGHT`. | ||
| - **Reply side** reads `@moq/watch`'s per-frame `arrivals()` handle on the | ||
| audio decoder and maps frames to words with the agent's TTS word timings. | ||
| It yields each frame's `{timestamp, arrival, late}` and a gap item | ||
| (`timestamp` and `duration` in `Time.Milli`) where media will never | ||
| arrive. Its queue is unbounded, so the page keeps its own window. There is | ||
| no gap at the join: the strip starts on the first frame read. | ||
| - **Prompt side** reads the agent's `voice.json` track on the response | ||
| broadcast: word timings from STT and the arrival time of each request | ||
| audio frame. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Voice AI demo page](/quest/m1/voice-page.md) - the page and transcript this draws under | ||
| - [An @moq/watch release ships per-frame arrivals](/quest/m1/watch-arrivals-release.md) - the reply-side data | ||
| - [The voice agent publishes voice.json](/quest/m1/voice-timeline-track.md) - word timings and the prompt-side data | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # [M] Voice AI demo page | ||
|
|
||
| ## Goal | ||
|
|
||
| `/voice` lets a reader talk to the moq.pro Pipecat agent over MoQ and shows a | ||
| live transcript of both sides. `/demo` gets a card for it. The component is | ||
| reusable, so a blog post can embed it later; the post itself (branch | ||
| `voice-api-2`) is not part of this quest. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-10-09: | ||
|
|
||
| - **A native Solid component, not moq.pro's iframe.** The buffer view needs | ||
| the player's internals, which an iframe from another site hides, and demos | ||
| belong in moq.dev. moq.pro keeps its own `/voice` page. | ||
| - **It talks to the moq.pro session API directly.** | ||
| `POST /voice/sessions {turnstileToken}` returns `{id, relayUrl, | ||
| requestBroadcast, responseBroadcast, token, expiresAt}`. One token | ||
| publishes the request and subscribes the response. Poll | ||
| `GET /voice/sessions/:id` until the agent claims it, and render refusals by | ||
| `code` (`busy`, `rate_limit`, `daily_budget`, ...). | ||
| moq.pro's `app/src/lib/VoiceDemo.svelte` is the reference client. | ||
| The older component on `voice-ai2` / `codex/voice-demo-embed` predates this | ||
| contract (`publishToken`/`watchToken`, no poll), so port its Turnstile | ||
| loader and layout, not its session code. | ||
| - **Live points at staging.** The demo only runs on `api.moq.wtf` until | ||
| moq.pro's production quest lands, so a separate `PUBLIC_VOICE_API_URL` is | ||
| `https://api.moq.wtf` for live and staging, and `PUBLIC_TURNSTILE_SITE_KEY` | ||
| is the wtf widget's public key. `PUBLIC_API_URL` stays on its environment's | ||
| API for `try`. | ||
| - **No Pipecat client.** `<moq-publish>` sends the mic and `<moq-watch>` | ||
| plays the reply with a latency ceiling near the agent's 25s send-ahead. | ||
| The page does the small part of RTVI the agent needs itself: publish a | ||
| `transcript.json.z` track carrying `client-ready`, and read the agent's | ||
| `transcript.json.z` (not in the catalog; subscribe by name) for | ||
| transcripts. On barge-in the agent retires `bot-audio` for | ||
| `bot-audio-<n>` through the catalog, which `<moq-watch>` follows. | ||
| - The transcript renders the way `src/components/voice/` lays out words, so | ||
| [Voice buffers under the words](/quest/m1/voice-buffers.md) can draw under | ||
| them. | ||
|
|
||
| Verify against staging in `just dev`: a conversation, barge-in, Stop, the | ||
| five-minute limit, and a `busy` refusal. | ||
|
|
||
| ## Required | ||
|
|
||
| - [moq.pro accepts moq.dev](/quest/m1/voice-api-moqdev.md) - the wtf voice API and Turnstile widget serve the moq.dev origin |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # [XS] The voice agent publishes voice.json | ||
|
|
||
| ## Goal | ||
|
|
||
| The moq.pro demo agent on wtf publishes a `voice.json` track on each session's | ||
| response broadcast, carrying word timings for both directions and the arrival | ||
| time of each request audio frame. | ||
|
|
||
| This quest tracks a condition outside the repository. When it holds, delete | ||
| this quest and every `Required` entry that links it. | ||
|
|
||
| ## Plan | ||
|
|
||
| moq.pro tracks the track as | ||
| [its own quest](https://github.com/moq-dev/moq.pro/blob/main/quest/m1/voice-timeline.md), | ||
| which owns its schema. Check by subscribing to it during a staging session. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # [XS] An @moq/watch release ships per-frame arrivals | ||
|
|
||
| ## Goal | ||
|
|
||
| The newest `@moq/watch` on npm exposes the per-frame `arrivals()` handle on | ||
| the audio and video decoders, and moq.dev builds against it. | ||
|
|
||
| This quest tracks a condition outside the repository. When it holds, bump | ||
| `@moq/watch` here, then delete this quest and every `Required` entry that | ||
| links it. | ||
|
|
||
| ## Plan | ||
|
|
||
| moq shipped the handle in | ||
| [moq-dev/moq#5311](https://github.com/moq-dev/moq/pull/5311). | ||
| Check `npm view @moq/watch version` and its type definitions for | ||
| `Audio.Decoder.arrivals()` and `Watch.Arrival`. |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
Repository: moq-dev/moq.dev
Length of output: 3143
🏁 Script executed:
Repository: moq-dev/moq.dev
Length of output: 3221
Describe red frames as a reference estimate.
The plan computes lateness against a typical WebRTC target from MoQ frame arrivals. It does not measure WebRTC playout. Update the wording to avoid claiming that an actual WebRTC receiver would conceal or skip each red frame. Apply the same qualification in the M1 summary.
Suggested wording
🤖 Prompt for AI Agents