diff --git a/developer_guides/firmware/images/wov_ecns_pipeline_architecture.svg b/developer_guides/firmware/images/wov_ecns_pipeline_architecture.svg index 2950125..bd657f9 100644 --- a/developer_guides/firmware/images/wov_ecns_pipeline_architecture.svg +++ b/developer_guides/firmware/images/wov_ecns_pipeline_architecture.svg @@ -1,4 +1,4 @@ - + @@ -50,6 +50,9 @@ + + + @@ -58,261 +61,325 @@ - + - - - Multi-Slot WOV & ECNS Dual-Stream Pipeline Architecture + + + Multi-Slot WOV & Dual-Rate ECNS Pipeline Architecture - - wcl-uaol-wov-002: 4-Channel DMIC Capture, 20ms DP ECNS, 2.0s KPB Pre-roll, Concurrent WOV Slots, and Notifier Bus + + Dual-Rate DMIC (48k & 16k), 20ms DP ECNS (+30dB Boost), 2.0s KPB Pre-roll, 3 MWW Slots (Hi Intel), and WOV Arbiter - - - - - - - Pipeline 100: DAI CAPTURE + + + + + + + DUAL-RATE DMIC CAPTURE - - Core 0 • LL 1ms (16 frames) + + Core 0 • LL 1ms Domains - - - - dai-copier.1 (dmic01) - • 4-Channel Native 16 kHz - Ch 0,1: Primary Mics - Ch 2,3: Echo Reference + + + + Pipeline 110: 48k DAI Ingest + LL 1ms (48 frames / tick) + + + + dai-copier (dmic01) + • PDM0 · DAI Index 0 + • 48 kHz · 2ch Stereo S16_LE + + + + + mixin.110.1 + • 48 kHz 2ch Pass-through + - - - - mixin.100.1 - • 4ch S16_LE Pass-through + + + + Pipeline 119: 16k DAI Ingest + LL 1ms (16 frames / tick) + + + + dai-copier (dmic16k) + • PDM1 · DAI Index 1 + • 16 kHz · 2ch Stereo S16_LE + + + + + mixin.119.1 + • 16 kHz 2ch Speech Ingest + - - - 4ch 16k - - - - - - - - Pipeline 105: ECNS DP + + + 48k 2ch + + + + 16k 2ch + + + + + + + + Pipeline 115: DUAL-RATE ECNS - - Core 0 • DP 20ms (320 frames) + + Core 0 • DP 20ms (lp_mode 1) - - - - mixout.105.1 - 4ch input buffer (ibs = 2560 B) + + + + mixout.115.2 (Pin 0 Input) + 16 kHz 2ch (ibs = 1280 B) - - - - ecns.105.1 (AEC & NS) - • Multi-Pin Audio Contract - • 4ch in (2 Mic + 2 Echo Ref) - Pin 0: Ch 0 Mono Clean - obs = 640 B (to KPB) - Pin 1: Ch 0,1 Stereo Clean - obs = 1280 B (to Host PCM 10) + + + + mixout.115.1 (Pin 1 Input) + 48 kHz 2ch (ibs = 3840 B) - - - - mixin.105.1 (KPB Branch) - 1ch mono clean speech + + + + ecns.115.1 (Dual-Rate DP) + • Multi-Pin Dual-Rate Contract + • Pin 0 In: 16k 2ch | Pin 1 In: 48k 2ch + + Pin 0 Out: 16 kHz 1ch Clean + • 100 Hz HPF (+30dB Boost) · obs 640 B + Pin 1 Out: 48 kHz 2ch Clean + • 40 Hz HPF (+20dB Boost) · obs 3840 B + + KControls: DMIC ECNS boost (0/1) + DMIC ecns_kpb_test_signal (ramp) - - - - mixin.105.2 (Host Branch) - 2ch stereo clean speech + + + + Pin 0 Out → KPB Path + 16 kHz 1ch Mono Clean (obs = 640 B) + + + + Pin 1 Out → Host Capture (P117) + 48 kHz 2ch Stereo Clean (obs = 3840 B) - - - 1ch clean - - - - 2ch clean - - - - - - - - Pipeline 106: KPB BUFFER + + + 16k clean + + + + 48k stereo + + + + + + + + Pipeline 116: KPB BUFFER - + Core 0 • DP 20ms (320 frames) - - - - mixout.106.1 - 1ch mono input - - - - - kpb.106.1 - • 2.0s Pre-roll Ring Buffer - • Size: 64 KB mono S16_LE - • Drains on NOTIFIER event + + + kpb.116.1 + • 2.0s Pre-roll Ring Buffer + • Size: 64 KB mono S16_LE + + Pin 0: Live Feed to Mixin + Pin 1: Drain Stream + • Connects to Arbiter Pin 0 + • Drains history upon trigger - - - - mixin.106.1 (Fan-Out) - 3-way splitter to WOV slots + + + + mixin.116.1 (Fan-Out Bus) + • 3-Way Splitter to Slots + → mixout.111.1 (Slot 0) + → mixout.112.1 (Slot 1) + → mixout.113.1 (Slot 2) + Decoupled SRAM Ring Buffers + Lockless Cross-Core Feeds - - - + + + - + - - - - - - Pipeline 101 • Slot 0 (Core 0) + + + + + + Pipeline 111 • Slot 0 (Core 0) - DP 10ms (160 frames) - • mixout.101.1 - • mfcc.101.1 (Mel-40 10ms) - • mww.101.1 "strawberry" - (or custom algorithm) + DP 10ms (160 frames) + • mixout.111.1 → mfcc.111.1 + • mww.111.1: "strawberry" + Avg 4 inf >= 50% | wovdebug_111 + 2 Hz Volatile Score: MWW 111 Score + CCOUNT Operator Profiling - - - - - - - Pipeline 102 • Slot 1 (Core 0) + + + + + + + Pipeline 112 • Slot 1 (Core 0) - DP 10ms (160 frames) - • mixout.102.1 - • mfcc.102.1 (Mel-40 10ms) - • mww.102.1 "banana" - (or custom algorithm) + DP 10ms (160 frames) + • mixout.112.1 → mfcc.112.1 + • mww.112.1: "Hi Intel" + Avg 4 inf >= 50% | wovdebug_112 + 2 Hz Volatile Score: MWW 112 Score + CCOUNT Operator Profiling - - - - - - - Pipeline 103 • Slot 2 (Core 1) + + + + + + + Pipeline 113 • Slot 2 (Core 0/1) - Cross-Core Offload • DP 10ms - • mixout.103.1 - • mfcc.103.1 (Mel-40 10ms) - • mww.103.1 "orange" - (or custom algorithm) + Cross-Core Capable • DP 10ms + • mixout.113.1 → mfcc.113.1 + • mww.113.1: "orange" + Avg 4 inf >= 50% | wovdebug_113 + 2 Hz Volatile Score: MWW 113 Score + CCOUNT Operator Profiling - - - - - - - - - - - - - Pipeline 104: ARBITER + + + KPB Pin 1 Drain Audio + + + + + + + + + + + + + + Pipeline 114: ARBITER - Core 0 • LL 1ms + Core 0 • LL 1ms (16 fr) - - - wov_arbiter.104.1 - • 3 in pins, 1 out - • wov_active_slot + + + + wov-arbiter.114.1 + • Pin 0: KPB Drain Audio + • Pins 1-3: Slot Triggers + + DMIC Phrase Detect ID + 0=Listen, 1..3=Slot - - - host-copier.11 - hw:0,11 (PCM 11) + + + + host-copier.12 + hw:0,12 (PCM 12) 1ch 16k WOV Audio + Pre-roll + Live Spoken + D0i3 Compatible - - - - - + + + + + + + + - Pipeline 107: ECNS HOST CAPTURE (Core 0 • LL 1ms) + Pipeline 117: ECNS HOST CAPTURE (Core 0 • LL 1ms) - - - mixout.107.1 - 2ch clean input - from mixin.105.2 + + + Direct Pin Binding + Source: ecns.115.1 Pin 1 + Rate: 48 kHz Stereo + obs = 3840 B (20ms) - - - host-copier.10 - hw:0,10 (PCM 10) - 2ch 16k Clean Comm + + + host-copier.11.capture + hw:0,11 (PCM 11) + 2ch 48k Clean Teleconf + D0i3 Compatible Wake - - + + - - - - SOF NOTIFIER INTER-MODULE EVENT BUS: + + + + SOF NOTIFIER BUS & INTER-MODULE SIGNALING PROTOCOL: + + + 1. NOTIFIER_ID_WOV_DETECT: Detector notifies Arbiter (slot_id) FIRST so Arbiter latches the route before KPB empties. + + + 2. NOTIFIER_ID_KPB_CLIENT_EVT: Detector notifies KPB to drain 2.0s history to Arbiter (dcache writeback/invalidate coherency). - - • NOTIFIER_ID_KPB_CLIENT_EVT: Winning detector commands KPB to drain 2.0s pre-roll buffer to active slot. + + 3. NOTIFIER_ID_WOV_CTRL: Arbiter broadcasts WOV_ARB_CMD_PAUSE to suppress sibling detector slots during host streaming. - - • NOTIFIER_ID_WOV_DETECT: Winning detector notifies Arbiter (slot_id) • NOTIFIER_ID_WOV_CTRL: Arbiter broadcasts PAUSE to sibling slots. + + 4. SOF_IPC4_MODULE_NOTIFICATION: Firmware asynchronously pushes Phrase Detect ID & 2 Hz Volatile Scoring updates to ALSA driver. diff --git a/developer_guides/firmware/images/wov_scheduling_periods.svg b/developer_guides/firmware/images/wov_scheduling_periods.svg index e7b906e..6d78e6e 100644 --- a/developer_guides/firmware/images/wov_scheduling_periods.svg +++ b/developer_guides/firmware/images/wov_scheduling_periods.svg @@ -1,4 +1,4 @@ - + @@ -33,32 +33,30 @@ - + - - + + Low-Latency (LL) vs. Data Processing (DP) Multi-Period Scheduling - - Matching Buffer Sizes (IBS/OBS), Execution Domains, and Cross-Core Task Balancing + + Matching Buffer Constraints (IBS/OBS), Execution Domains, and Cross-Core Load Balancing - + - Execution Timeline Across Scheduling Domains (16 kHz Audio Stream) + Execution Timeline Across Scheduling Domains (Dual-Rate 48k & 16k Audio Streams) - - 0ms 5ms @@ -67,10 +65,10 @@ 20ms (DP ECNS) - + - LL Domain (1ms) + LL Domain (1ms) @@ -96,14 +94,14 @@ - 16 samples/tick + 16/48 fr / tick - DP 10ms (MWW) - 160 samples / batch + DP 10ms (MWW) + 160 frames / batch @@ -122,92 +120,105 @@ - + - DP 20ms (ECNS) - 320 samples / batch + DP 20ms (ECNS) + 320 (16k) / 960 (48k) fr - Active AEC & NS (4.2ms) - Filters 4ch -> 1ch + 2ch + Active AEC & Boost (3.8ms) + 16k In -> 16k Clean / 48k In -> 48k Clean - Idle / Low-Power Sleep (15.8ms) + Idle / Low-Power Sleep (16.2ms) - - - - - + + + + + 1. Buffer Sizing Math (IBS / OBS) - - IBS = R * (P / 1000) * CH * B - • 20ms 4ch S16_LE (ECNS Input): - 16000 * 0.02 * 4 * 2 = 2560 B - • 20ms 1ch S16_LE (Pin 0 Clean): - 16000 * 0.02 * 1 * 2 = 640 B - • 10ms 1ch S16_LE (MWW Input): - 16000 * 0.01 * 1 * 2 = 320 B - - RULE: IBS/OBS must match period! + + Size = Rate * (Period / 1000) * CH * B + • 20ms 16k 2ch S16 (ECNS Pin 0 In): + 16000 * 0.02 * 2 * 2 = 1280 B (ibs) + • 20ms 48k 2ch S16 (ECNS Pin 1 In): + 48000 * 0.02 * 2 * 2 = 3840 B (ibs) + • 20ms 16k 1ch Clean (to KPB): + 16000 * 0.02 * 1 * 2 = 640 B (obs) + • 20ms 48k 2ch Clean (to Host 11): + 48000 * 0.02 * 2 * 2 = 3840 B (obs) + • 10ms 16k 1ch MWW Slot Input: + 16000 * 0.01 * 1 * 2 = 320 B (ibs) + + RULE: IBS/OBS must match period exactly! - - - - - + + + + + 2. Multi-Core Affinity Allocation - - Core 0 (Primary DSP Core): - • P100 DAI Copier (LL 1ms) - • P105 ECNS Processing (DP 20ms) - • P106 KPB History Ring (DP 20ms) - • P101 Slot 0 & P102 Slot 1 (DP 10ms) + + Core 0 (Primary DSP Core): + • P110 (48k) & P119 (16k) DAI Ingest (LL 1ms) + • P115 Dual-Rate ECNS Processing (DP 20ms) + • P116 KPB 2.0s Pre-roll Ring Buffer (DP 20ms) + • P111 Slot 0 ("strawberry") Detector (DP 10ms) + • P112 Slot 1 ("Hi Intel") Detector (DP 10ms) + • P114 WOV Arbiter & P117 Host Capture (LL 1ms) - Core 1 (Secondary DSP Core): - • P103 Slot 2 (DP 10ms - core_id = 1) - - Offloads neural network inference. + Core 1 (Secondary DSP Core): + • P113 Slot 2 ("orange") Detector (DP 10ms) + + Offloads neural network inference via core_id 1. + + + Inter-core audio routed locklessly via SRAM mixin. - - - - - + + + + + 3. Topology Class & lp_mode Flags - - Class.Pipeline."custom-capture": - lp_mode 0 # Low Latency (1ms) - Used for: DAI, Arbiter, Host Copiers + + Class.Pipeline."custom-capture": + lp_mode 0 # Low Latency (1ms domain) + Used for: DAI 110/119, Arbiter 114, Host 117 - Class.Pipeline."custom-dp-capture": - lp_mode 1 # Data Processing (DP) - Used for: ECNS (20ms), KPB, MWW (10ms) + Class.Pipeline."custom-dp-capture": + lp_mode 1 # Data Processing (DP domain) + Used for: ECNS 115 (20ms), KPB 116 (20ms), + MWW Slots 111-113 (10ms domain) - + time_domain "timer" - + Enables RTOS tick-driven periodic scheduling. + + capture_compatible_d0i3: true on PCMs 11 & 12. + diff --git a/developer_guides/firmware/wov_ecns_pipeline_guide.rst b/developer_guides/firmware/wov_ecns_pipeline_guide.rst index 7da0a51..6e2d668 100644 --- a/developer_guides/firmware/wov_ecns_pipeline_guide.rst +++ b/developer_guides/firmware/wov_ecns_pipeline_guide.rst @@ -7,41 +7,56 @@ How-To Guide: Customizing Multi-Slot WOV & ECNS Pipelines :local: :depth: 3 -Sound Open Firmware (SOF) features an advanced, multi-pipeline **Multi-Slot Wake-on-Voice (WOV) and Echo Cancellation & Noise Suppression (ECNS)** subsystem developed on the ``wcl-uaol-wov-002`` branch. This architecture enables simultaneous full-duplex communications and multi-keyword spotting from a single Digital Microphone (DMIC) array. +Sound Open Firmware (SOF) features a high-performance, multi-pipeline **Multi-Slot Wake-on-Voice (WOV) and Echo Cancellation & Noise Suppression (ECNS)** subsystem. This architecture enables simultaneous full-duplex communications, low-power continuous keyword spotting, and seamless pre-roll history audio streaming from a single Digital Microphone (DMIC) array or SoundWire headset interface. -This comprehensive developer guide details how the subsystem is structured, how inter-module communication is coordinated locklessly via the **SOF Notifier**, and provides a step-by-step walkthrough on how to **replace the ECNS and WOV modules with custom or third-party algorithms**, modify scheduling periods, adjust buffer constraints (IBS/OBS), and reconfigure ALSA Topology 2.0. +The unified subsystem architecture is deployed across **Panther Lake (PTL)**, **Tiger Lake (TGL)**, and **Wildcat Lake (WCL)** (which shares the ACE 3.0 PTL architecture via topology symlinks). + +This comprehensive developer guide details how the subsystem is structured, how inter-module communication is coordinated locklessly via the **SOF Notifier**, and provides a step-by-step walkthrough on how to **replace the ECNS and WOV modules with custom or third-party algorithms**, configure dual-rate DMIC inputs, adjust buffer constraints (IBS/OBS), utilize real-time scoring kcontrols, and reconfigure ALSA Topology 2.0. --- Subsystem Architecture & Signal Flow ************************************ -The baseline architecture coordinates eight interconnected pipelines spanning two distinct host audio capture streams originating from a single Digital Microphone (DMIC) physical interface: +The baseline architecture coordinates ten interconnected pipelines spanning two distinct host audio capture streams originating from physical Digital Microphone (DMIC) hardware interfaces: + +1. **ECNS Communication Stream (ALSA PCM 11 / ``hw:0,11``)**: Captures continuous, clean, noise-suppressed stereo 48 kHz speech (or 16 kHz when configured in single-producer mode). This stream feeds host-side teleconferencing applications (e.g. Teams, Zoom, WebRTC) or audio recorders requiring high-fidelity voice transmission free from acoustic feedback and room reverberation. +2. **WOV Keyword & Audio Stream (ALSA PCM 12 / ``hw:0,12``)**: Captures 2.0 seconds of cached historical audio from the Key Phrase Buffer (KPB) seamlessly concatenated with live speech from whichever keyword spotting slot triggered detection. This stream feeds host voice assistants (e.g. Alexa, Google Assistant, custom on-device wake-word engines). + +Both capture endpoints support low-power D0i3 listening and wake (``capture_compatible_d0i3: true``). + +Dual-Rate DMIC Hardware Interface (PDM0 & PDM1) +=============================================== + +Modern DMIC controllers support multiple independent PDM (Pulse Density Modulation) hardware links. The baseline architecture establishes two synchronous hardware DAI capture interfaces: -1. **ECNS Communication Stream (ALSA PCM 10 / ``hw:0,10``)**: Captures clean, noise-suppressed stereo 16 kHz speech. This stream is routed to host-side teleconferencing applications (e.g. Teams, Zoom, WebRTC) or audio recorders requiring high-fidelity voice transmission free from acoustic feedback and room reverberation. -2. **WOV Keyword & Audio Stream (ALSA PCM 11 / ``hw:0,11``)**: Captures pre-roll audio (2.0 seconds of cached historical audio from the Key Phrase Buffer) seamlessly concatenated with live speech from whichever keyword spotting slot triggered detection. This stream feeds host voice assistants (e.g. Alexa, Google Assistant, custom on-device wake-word engines). +* **PDM Interface ``dmic01`` (DAI Index 0 on PDM0)**: Ingests 48 kHz, 2-channel stereo, 16-bit audio (Pipeline 110). This link supplies the high-resolution acoustic stream for teleconferencing, media capture, and host applications. +* **PDM Interface ``dmic16k`` (DAI Index 1 on PDM1)**: Ingests 16 kHz, 2-channel stereo, 16-bit audio (Pipeline 119). This link provides a dedicated lower-rate voice stream optimized for low-power voice trigger processing without requiring software sample rate conversion (SRC). -A physical 4-channel 16 kHz DMIC array supplies the DSP with two primary channels of near-end acoustic speech (Microphones 0 and 1) and two channels of far-end acoustic echo reference (Speaker Channels 2 and 3 looped back from the playback audio subsystem). By co-locating near-end speech and far-end echo in the same synchronous 4-channel DMIC stream, the hardware guarantees zero sample-drift and deterministic phase alignment between speaker playback and microphone capture. +.. note:: + **Single-Producer Fallback**: + The ECNS module includes built-in fallback support for single-producer configurations (such as hardware platforms or topologies with only a single 16 kHz DMIC DAI or when the 48 kHz stream is disabled). In this fallback mode, ECNS consumes from the 16 kHz input and synthesizes the required routing. -The audio graph processes this stream through three distinct functional phases: +Three Functional Processing Phases +================================== -* **Phase 1: Low-Latency DAI Capture (Pipeline 100)**: Ingests raw 4-channel 16 kHz audio from the DMIC hardware at 1 ms intervals (16 frames per tick), maintaining minimum input latency before handing off to processing. -* **Phase 2: Data Processing & Dual-Pin Separation (Pipelines 105 & 106)**: Batches samples into a 20 ms period (320 frames). The ECNS module cancels speaker echo using the reference channels and removes ambient noise, generating two independent output streams: - - **Pin 0 (Clean Mono Speech)**: Fed to Pipeline 106 (Key Phrase Buffer), which continuously maintains a 2.0-second circular ring buffer in DSP memory. - - **Pin 1 (Clean Stereo Speech)**: Fed directly to Pipeline 107 (ECNS Host Copier) for teleconferencing on PCM 10. -* **Phase 3: Multi-Slot Keyword Detection & Arbitration (Pipelines 101–104)**: The KPB fans out the mono clean speech across three concurrent detector slots running in 10 ms Data Processing periods. When a slot matches its target keyword model, it triggers the KPB to drain its 2.0-second history buffer and commands the WOV Arbiter to multiplex that slot's audio to PCM 11. +The audio graph processes audio through three distinct functional phases: + +* **Phase 1: Low-Latency DAI Ingest (Pipelines 110 & 119)**: Ingests 48 kHz 2-channel audio (Pipeline 110, 48 frames per tick) and 16 kHz 2-channel audio (Pipeline 119, 16 frames per tick) at 1 ms intervals, maintaining minimum input latency before batching into Data Processing chunks. +* **Phase 2: Dual-Rate ECNS & History Buffering (Pipelines 115 & 116)**: Operates in a 20 ms Data Processing (DP) period (``lp_mode 1``). Pipeline 115 (Dual-Rate ECNS) ingests both 16 kHz voice (Pin 0 In, 320 frames = 1280 bytes) and 48 kHz capture (Pin 1 In, 960 frames = 3840 bytes), producing 16 kHz mono clean speech (Pin 0 Out, 640 bytes) directly to Pipeline 116 (Key Phrase Buffer) and 48 kHz stereo clean speech (Pin 1 Out, 3840 bytes) directly to Pipeline 117 (ECNS Host Capture on PCM 11). Pipeline 116 maintains a continuous 2.0-second circular history buffer (64 KB mono) in DSP SRAM. +* **Phase 3: Multi-Slot Keyword Detection & Arbitration (Pipelines 111–114)**: Three parallel detector slots (Pipelines 111–113) execute in 10 ms DP periods (160 frames at 16 kHz) monitoring for "strawberry" (Slot 0), "Hi Intel" (Slot 1), and "orange" (Slot 2). When a slot triggers, Pipeline 114 (WOV Arbiter, LL 1ms) latches the active slot, updates the ``DMIC Phrase Detect ID`` control, drains the 2.0-second pre-roll history buffer, and streams audio to ALSA PCM 12. .. figure:: images/wov_ecns_pipeline_architecture.svg - :alt: Multi-Slot WOV and ECNS Pipeline Architecture + :alt: Multi-Slot WOV and Dual-Rate ECNS Pipeline Architecture :width: 100% :align: center - Figure 322: Multi-Slot WOV & ECNS Pipeline Architecture across 4ch DMIC capture, 20ms DP ECNS, 2.0s KPB pre-roll, 3 concurrent detector slots, WOV Arbiter, and the SOF Notifier event bus. + Figure: Multi-Slot WOV & Dual-Rate ECNS Pipeline Architecture across dual DMIC capture (48k & 16k), 20ms DP ECNS, 2.0s KPB pre-roll, 3 concurrent detector slots ("strawberry", "Hi Intel", "orange"), WOV Arbiter, and the SOF Notifier bus. Pipeline Graph Breakdown ======================== -.. list-table:: Multi-Slot WOV and ECNS Pipeline Map +.. list-table:: Multi-Slot WOV and ECNS Pipeline Specification :widths: 12 18 20 50 :header-rows: 1 @@ -49,50 +64,68 @@ Pipeline Graph Breakdown - Name - Scheduling Domain - Components & Signal Flow - * - **Pipeline 100** - - DAI Capture + * - **Pipeline 110** + - 48k DAI Capture + - Core 0 • LL 1 ms (48 frames) + - ``dai-copier.DMIC.dmic01.capture`` (2ch 48 kHz S16_LE) → ``mixin.110.1`` (48k 2ch pass-through). + * - **Pipeline 119** + - 16k DAI Capture - Core 0 • LL 1 ms (16 frames) - - ``dai-copier.1`` (4ch 16 kHz S16_LE) → ``mixin.100.1`` (4ch pass-through). - * - **Pipeline 105** - - ECNS Processing - - Core 0 • DP 20 ms (320 frames) - - ``mixout.105.1`` → ``ecns.105.1`` (AEC & NS). Produces: - - **Pin 0**: Mono clean mic → ``mixin.105.1`` (to KPB). - - **Pin 1**: Stereo clean mic → ``mixin.105.2`` (to Host Copier 10). - * - **Pipeline 106** + - ``dai-copier.DMIC.dmic16k.capture`` (2ch 16 kHz S16_LE) → ``mixin.119.1`` (16k 2ch pass-through). + * - **Pipeline 115** + - Dual-Rate ECNS + - Core 0 • DP 20 ms (``lp_mode 1``) + - ``mixout.115.2`` (16k) & ``mixout.115.1`` (48k) → ``ecns.115.1``. Produces: + - **Pin 0 Out**: 16 kHz 1ch mono clean (OBS 640 B) → direct to ``kpb.116.1``. + - **Pin 1 Out**: 48 kHz 2ch stereo clean (OBS 3840 B) → direct to ``host-copier.11.capture``. + * - **Pipeline 116** - KPB History Buffer - Core 0 • DP 20 ms (320 frames) - - ``mixout.106.1`` → ``kpb.106.1`` (2.0s = 64 KB mono ring buffer) → ``mixin.106.1`` (3-way fanout mixin). - * - **Pipeline 101** + - ``kpb.116.1`` (2.0s = 64 KB mono circular buffer): + - **Pin 0**: Live feed → ``mixin.116.1`` (3-way fan-out to detector slots). + - **Pin 1**: Host sink drain stream → ``wov-arbiter.114.1`` Pin 0. + * - **Pipeline 111** - WOV Slot 0 - Core 0 • DP 10 ms (160 frames) - - ``mixout.101.1`` → ``mfcc.101.1`` → ``mww.101.1`` (keyword model: "strawberry"). - * - **Pipeline 102** + - ``mixout.111.1`` → ``mfcc.111.1`` → ``mww.111.1`` (keyword model: "strawberry"). + * - **Pipeline 112** - WOV Slot 1 - Core 0 • DP 10 ms (160 frames) - - ``mixout.102.1`` → ``mfcc.102.1`` → ``mww.102.1`` (keyword model: "banana"). - * - **Pipeline 103** + - ``mixout.112.1`` → ``mfcc.112.1`` → ``mww.112.1`` (keyword model: "Hi Intel"). + * - **Pipeline 113** - WOV Slot 2 - - **Core 1** • DP 10 ms (160 frames) - - ``mixout.103.1`` → ``mfcc.103.1`` → ``mww.103.1`` (keyword model: "orange"). Demonstrates **cross-core task offload**. - * - **Pipeline 104** - - WOV Host Capture + - **Core 0 or Core 1** • DP 10 ms + - ``mixout.113.1`` → ``mfcc.113.1`` → ``mww.113.1`` (keyword model: "orange"). Demonstrates **cross-core task offload**. + * - **Pipeline 114** + - WOV Arbiter & Capture - Core 0 • LL 1 ms (16 frames) - - ``wov_arbiter.104.1`` (3 input pins, 1 output pin) → ``host-copier.11`` (ALSA PCM 11, ``hw:0,11``). - * - **Pipeline 107** + - ``wov-arbiter.114.1`` (Pin 0 audio + Pins 1–3 triggers) → ``host-copier.12.capture`` (ALSA PCM 12, ``hw:0,12``). + * - **Pipeline 117** - ECNS Host Capture - - Core 0 • LL 1 ms (16 frames) - - ``mixout.107.1`` (stereo clean input from ``mixin.105.2``) → ``host-copier.10`` (ALSA PCM 10, ``hw:0,10``). + - Core 0 • LL 1 ms (48 frames) + - ``host-copier.11.capture`` (ALSA PCM 11, ``hw:0,11``, 48 kHz stereo clean). + +SoundWire Headset Alternative (Pipelines 140-148) +================================================= + +In topologies configured with SoundWire headset audio (such as Realtek ALC721 or ALC722 on Tiger Lake or Panther Lake via ``sdw-jack-wov-multi.conf``), a parallel pipeline topology structure is instantiated: + +* **Pipeline 140**: Sample Rate Converter (SRC) converting 48 kHz headset microphone capture down to 16 kHz. +* **Pipelines 141–143**: Multi-slot keyword detector slots for headset microphones. +* **Pipeline 144**: Headset WOV Arbiter exposing the ``Jack In Phrase Detect ID`` ALSA mixer control. +* **PCM Endpoints**: + - **ALSA PCM 13 (``hw:0,13``)**: Headset ECNS Stereo Capture (48 kHz S16_LE). + - **ALSA PCM 14 (``hw:0,14``)**: Headset Multi-WOV Keyword Capture (16 kHz S16_LE). --- SOF Notifier Inter-Module Event Bus *********************************** -A central architectural innovation in this subsystem is the lockless, zero-IPC event dispatch provided by the **SOF Notifier**. Because all modules execute inside the same DSP address space, inter-module events are dispatched synchronously without serialization or kernel intervention: +A central architectural pillar in this subsystem is the lockless, zero-IPC event dispatch provided by the **SOF Notifier**. Because all modules execute inside the same DSP address space, inter-module events are dispatched synchronously without serialization or kernel intervention. -.. list-table:: SOF Notifier Events in WOV/ECNS Architecture - :widths: 25 20 20 35 +.. list-table:: SOF Notifier Events in WOV/ECNS Subsystem + :widths: 25 18 18 39 :header-rows: 1 * - Notifier Event ID @@ -101,629 +134,502 @@ A central architectural innovation in this subsystem is the lockless, zero-IPC e - Action & Payload * - ``NOTIFIER_ID_WOV_DETECT`` - Triggered Detector (e.g. Slot 0) - - ``wov_arbiter`` - - Passes ``slot_id`` (0, 1, or 2). Arbiter switches the active audio route to this slot and unmuts audio. + - ``wov-arbiter`` + - Passes ``struct wov_detect_notif { uint8_t slot_id; }``. Arbiter latches the winning slot, switches the active audio route, and unmuts audio. * - ``NOTIFIER_ID_KPB_CLIENT_EVT`` - Triggered Detector - - ``kpb`` (Key Phrase Buffer) - - Commands KPB to transition to drain mode (``KPB_EVENT_DRAIN``), emptying 2.0s of cached history audio. + - ``kpb`` + - Passes ``struct kpb_event_data`` with ``KPB_EVENT_BEGIN_DRAINING``. Commands KPB to rapidly pump 2.0s of cached history audio to the Arbiter. * - ``NOTIFIER_ID_WOV_CTRL`` - - ``wov_arbiter`` + - ``wov-arbiter`` - Sibling Detectors - - Broadcasts ``WOV_CMD_PAUSE`` to prevent competing slots from firing, or ``WOV_CMD_RESUME`` on host stream reset. - -Notifier Registration & Dispatch Pattern -======================================== - -The SOF Notifier provides a lockless, publish-subscribe event mechanism that coordinates DSP processing components executing within the firmware address space without incurring host IPC latency, kernel transitions, or DMA overhead. - -Notifier Registration & Lifecycle ---------------------------------- + - Broadcasts ``WOV_ARB_CMD_PAUSE`` to pause sibling inference, or ``WOV_ARB_CMD_RESUME`` on stream reset. + * - ``SOF_IPC4_MODULE_NOTIFICATION`` + - Firmware (Arbiter / MWW) + - Linux ALSA Driver + - Asynchronously pushes ``DMIC Phrase Detect ID`` enum changes and 2 Hz volatile scoring kcontrol updates to host. -Modules subscribe to events using ``notifier_register()``, typically during the component's ``prepare()`` or ``init()`` lifecycle phases: - -.. code-block:: c - - #include - #include - #include +Critical Architectural Rules for Event Dispatch +=============================================== - /* Prototype: - * void notifier_register(struct comp_dev *dev, void *caller_data, - * enum notify_id type, - * void (*cb)(void *arg, enum notify_id type, void *data), - * uint32_t flags); - */ - notifier_register(dev, cd, NOTIFIER_ID_WOV_CTRL, my_wov_ctrl_callback, 0); +1. Mandatory Notification Sequencing Order +------------------------------------------ -Key parameters for registration: +.. important:: + **Activate Arbiter BEFORE Draining KPB**: + The WOV Arbiter discards everything arriving on its KPB input pin while no slot is active. If the keyword detector signals KPB to drain before notifying the Arbiter, the entire 2.0-second history buffer is drained into the Arbiter and discarded before the Arbiter switches to the detecting slot. As a result, the captured stream would start with live audio following the keyword, losing the wake-word! -* ``dev``: Pointer to the component device (``struct comp_dev *``). -* ``caller_data``: Private context pointer passed back as the first argument (``void *arg``) to the callback function (e.g. ``struct my_wov_comp_data *cd``). -* ``type``: The unique enumeration value identifying the event class (e.g. ``NOTIFIER_ID_WOV_CTRL``). -* ``cb``: Callback function pointer invoked synchronously whenever matching events are dispatched. -* ``flags``: Operational modifiers (typically ``0`` for standard synchronous delivery). + Therefore, detector modules **must** send the detection notification to the Arbiter first and command KPB draining second: -.. warning:: - **Mandatory Unregistration**: - Any module that registers a notifier callback **must** explicitly call ``notifier_unregister(dev, cd, NOTIFIER_ID_WOV_CTRL)`` during its ``reset()`` and ``free()`` lifecycle routines. Failing to unregister leaves dangling function pointers in the global notifier dispatch table, causing fatal DSP exception faults (null-pointer or unmapped memory dereferences) when subsequent events are fired after pipeline teardown. + .. code-block:: c -Callback Execution Context & Rules ----------------------------------- + /* 1. Activate Arbiter FIRST so audio routing is latched */ + struct wov_detect_notif arb_payload = { .slot_id = cd->slot_id }; + notifier_event(dev, NOTIFIER_ID_WOV_DETECT, + NOTIFIER_TARGET_CORE_ALL_MASK, + &arb_payload, sizeof(arb_payload)); -Notifier callbacks are executed **synchronously within the thread/task context of the component that dispatches the event** via ``notifier_event()``: + /* 2. Command KPB to drain 2.0s history audio downstream */ + mww_notify_kpb(mod); -.. code-block:: c +2. Cross-Core Cache Coherency +----------------------------- - static void my_wov_ctrl_callback(void *arg, enum notify_id type, void *data) - { - struct my_wov_comp_data *cd = (struct my_wov_comp_data *)arg; - struct wov_ctrl_event_data *ctrl = (struct wov_ctrl_event_data *)data; +When a detector module runs on a secondary core (such as Core 1) while KPB and Arbiter execute on Core 0: - if (ctrl->cmd == WOV_CMD_PAUSE) { - cd->paused = true; - } else if (ctrl->cmd == WOV_CMD_RESUME) { - cd->paused = false; - cd->detected = false; - my_model_reset(cd->model_context); - } - } +* The keyword trigger passes a ``struct kpb_event_data`` whose ``client_data`` pointer references a ``struct kpb_client`` allocated in the detector's core memory. +* Standard ``notifier_event()`` dispatch only writes back the outer ``kpb_event_data`` structure, not nested pointer targets. +* **Firmware Rule**: The notifying module must explicitly call ``dcache_writeback_region()`` on its ``client_data`` structure prior to firing ``notifier_event()``. In turn, the KPB event handler calls ``dcache_invalidate_region()`` on the received ``client_data`` pointer before inspecting buffer parameters. -Because the callback runs directly on the caller's execution thread, strict real-time rules apply: +3. Blocking Remote Core Notifications +------------------------------------- -* **Non-Blocking Execution**: Callbacks must never sleep, yield, or pend on semaphores or mutexes. -* **Zero Dynamic Allocation**: Never allocate memory (``rballoc``, ``malloc``) inside a callback. -* **Minimal State Manipulation**: Limit logic to setting volatile state flags, resetting pointer offsets, or triggering internal state machines. +In multi-core DSP configurations, ``notifier_event()`` delivers cross-core notifications via Inter-Processor Communication (IDC) messages using ``IDC_BLOCKING`` mode. Because each target core has a single notification data slot, blocking until remote completion guarantees that rapid back-to-back events (such as ``NOTIFIER_ID_WOV_DETECT`` followed immediately by ``NOTIFIER_ID_KPB_CLIENT_EVT``) do not overwrite queued event payloads or resubmit in-flight IDC work items. -Synchronous Event Dispatch & Payloads -------------------------------------- +Notifier Registration & Lifecycle +================================= -When a keyword detector achieves confidence scoring above its operational threshold, it issues two back-to-back synchronous notifications: +Modules subscribe to events during ``prepare()`` or ``init()``: .. code-block:: c - static void my_wov_trigger_detection(struct comp_dev *dev, int slot_id) - { - struct kpb_event_data kpb_evt = { - .event_id = KPB_EVENT_DRAIN, - .client_id = slot_id, - }; - struct wov_detect_event_data arb_evt = { - .slot_id = slot_id, - }; - - /* 1. Command KPB to drain 2.0 seconds of pre-roll history audio */ - notifier_event(dev, NOTIFIER_ID_KPB_CLIENT_EVT, - CORE_SPECIFIC_BROADCAST, &kpb_evt, sizeof(kpb_evt)); - - /* 2. Inform WOV Arbiter of winning slot to route audio to PCM 11 */ - notifier_event(dev, NOTIFIER_ID_WOV_DETECT, - CORE_SPECIFIC_BROADCAST, &arb_evt, sizeof(arb_evt)); - } + #include + #include + #include -Dispatch Mechanics: + /* Subscribe to PAUSE / RESUME control commands from WOV Arbiter */ + notifier_register(dev, cd, NOTIFIER_ID_WOV_CTRL, my_wov_ctrl_callback, 0); -* ``core_mask``: Specifying ``CORE_SPECIFIC_BROADCAST`` instructs the notifier infrastructure to dispatch the event locally on the current core and cross-dispatch to secondary cores (e.g. Core 1) via DSP Inter-Processor Interrupts (IPI). -* ``NOTIFIER_ID_KPB_CLIENT_EVT``: Ingested by ``kpb.106.1``. The ``KPB_EVENT_DRAIN`` payload causes the Key Phrase Buffer to cease circular overwriting and rapidly pump its 64 KB history ring buffer downstream into the active slot's pipeline. -* ``NOTIFIER_ID_WOV_DETECT``: Ingested by ``wov_arbiter.104.1``. The ``slot_id`` payload instructs the arbiter to lock its audio multiplexer to that slot's input pin, unmute the stream to ``host-copier.11`` (ALSA PCM 11), and broadcast a ``WOV_CMD_PAUSE`` event across ``NOTIFIER_ID_WOV_CTRL`` to suppress sibling slots from firing. +.. warning:: + **Mandatory Unregistration**: + Any module that registers a notifier callback **must** explicitly call ``notifier_unregister(dev, cd, NOTIFIER_ID_WOV_CTRL)`` during its ``reset()`` and ``free()`` routines. Failing to unregister leaves dangling function pointers in the global dispatch table, triggering kernel or DSP exception panics when subsequent events occur after pipeline teardown. --- Step-by-Step Guide: Replacing the ECNS Module ********************************************* -The ECNS module performs Acoustic Echo Cancellation (AEC) and Noise Suppression (NS). Developers can replace the reference ECNS module with custom neural noise suppressors (e.g. RNNoise, DeepFilterNet), open-source algorithms (SpeexDSP, WebRTC AEC3), or proprietary vendor DSP libraries. +The ECNS module performs Acoustic Echo Cancellation (AEC), Noise Suppression (NS), and dual-stream signal routing. Developers can replace the reference ECNS module with custom neural noise suppressors (e.g. RNNoise, DeepFilterNet), open-source algorithms (WebRTC AEC3, SpeexDSP), or proprietary vendor DSP libraries. + +Step 1: Understand the Dual-Rate Multi-Pin Contract +=================================================== + +The ECNS component operates as a multi-sink, multi-source Data Processing (DP) block under a 20 ms period (``lp_mode 1``): + +* **Input Pin 0 (16 kHz 2-Channel Stereo Ingest)**: + - Source: ``dai-copier.DMIC.dmic16k.capture`` (Pipeline 119). + - Rate: 16,000 Hz · 2 channels (or 4 channels on specific hardware layouts) · ``S16_LE``. + - Input Buffer Size: :math:`16000 \times 0.020 \times 2 \times 2 = \mathbf{1280\text{ bytes}}` (``ibs 1280``). +* **Input Pin 1 (48 kHz 2-Channel Stereo Ingest)**: + - Source: ``dai-copier.DMIC.dmic01.capture`` (Pipeline 110). + - Rate: 48,000 Hz · 2 channels (or 4 channels) · ``S16_LE``. + - Input Buffer Size: :math:`48000 \times 0.020 \times 2 \times 2 = \mathbf{3840\text{ bytes}}` (``ibs 3840``). +* **Output Pin 0 (Clean Mono Voice to KPB / WOV)**: + - Destination: ``kpb.116.1`` (Pipeline 116). + - Rate: 16,000 Hz · 1 channel (mono clean speech) · ``S16_LE``. + - Output Buffer Size: :math:`16000 \times 0.020 \times 1 \times 2 = \mathbf{640\text{ bytes}}` (``obs 640``). +* **Output Pin 1 (Clean Stereo Voice to Host PCM 11)**: + - Destination: ``host-copier.11.capture`` (Pipeline 117). + - Rate: 48,000 Hz · 2 channels (stereo clean speech) · ``S16_LE``. + - Output Buffer Size: :math:`48000 \times 0.020 \times 2 \times 2 = \mathbf{3840\text{ bytes}}` (``obs 3840``). + +Step 2: Implement Built-In High-Pass Filters & Boost Switching +============================================================== + +To optimize detection accuracy on acoustic microphone streams, the ECNS reference implementation incorporates built-in Direct Form I (DF1) second-order biquad high-pass filters: + +1. **Pin 0 Detection Output Filter (100 Hz HPF)**: + - Configured via ``CONFIG_ECNS_KPB_HPF``. + - Boosted mode applies **+30 dB fixed gain** to elevate quiet speech peaks (typically around -36 dBFS) well above ambient room background. + - Normal mode applies the identical 100 Hz response with **0 dB gain**. +2. **Pin 1 Host Output Filter (40 Hz HPF)**: + - Configured via ``CONFIG_ECNS_HOST_HPF``. + - Boosted mode applies **+20 dB fixed gain**. + - Normal mode applies the identical 40 Hz response with **0 dB gain**. + +Capture Boost Switch Control (``DMIC ECNS boost``) +-------------------------------------------------- -Step 1: Understand the Multi-Pin Contract -========================================= +The firmware exposes an ALSA mixer boolean switch control (``DMIC ECNS boost`` for DMIC instances; ``Jack In ECNS boost`` for headset instances). -In full-duplex communication systems, microphones pick up both the near-end user's voice and the far-end speaker playback echoing off the room walls and device chassis. To eliminate this acoustic coupling without distorting speech, the ECNS component must ingest both the microphone signals and an uncorrupted echo reference, and provide separate outputs tailored for voice trigger and communications. +* **Live Dynamic Switching**: Changing this control does **not** reset the filter delay lines. The firmware simply swaps the active IIR coefficient pointer between the 0 dB and +30 dB / +20 dB coefficient sets during streaming inside ``copy()``, ensuring clickless, glitch-free transitions. +* **Default State**: Firmware initializes with boosted filters enabled. When the ALSA card is instantiated, the host driver sets the control to its default value (0 dB). -An ECNS component in this topology must adhere to a strict multi-pin input/output contract: +Synthetic Test Signal Generator (``CONFIG_ECNS_TEST_SIGNAL``) +------------------------------------------------------------- -* **Input Pin 0 (4-Channel 16 kHz S16_LE Capture)**: - - **Channels 0 and 1 (Microphones)**: Primary physical microphone signals containing near-end speech, ambient noise, and acoustic speaker echo. - - **Channels 2 and 3 (Echo Reference)**: Digital loopback of the speaker audio currently being rendered by the DAC/amplifier. The AEC adaptive filter cross-correlates this reference with the microphone input to model the room impulse response and subtract the loudspeaker echo. -* **Output Pin 0 (Mono Clean Speech - 1 Channel 16 kHz S16_LE)**: - - Feeds ``mixin.105.1`` → ``kpb.106.1`` for keyword spotting. - - Keyword spotting engines (e.g. MFCC feature extractors, DNN classifiers) require a single, normalized, echo-free mono channel. Spatial cues or stereo phase differences can degrade acoustic model recognition accuracy. -* **Output Pin 1 (Stereo Clean Speech - 2 Channels 16 kHz S16_LE)**: - - Feeds ``mixin.105.2`` → ``host-copier.10`` (ALSA PCM 10) for human teleconferencing. - - Preserves natural stereo spatial imaging for VoIP, meeting applications, and host-side recordings while stripping out background noise and speaker feedback. +For automated CI and hardware validation, the ECNS component includes an optional synthetic ramp generator enabled via ``CONFIG_ECNS_TEST_SIGNAL=y``. When the mixer switch control ``DMIC ecns_kpb_test_signal`` (control index 1) is enabled, the ECNS module replaces Pin 0 output data with a deterministic sawtooth test ramp. This allows end-to-end verification of KPB history draining and WOV audio capture without needing a physical loudspeaker or spoken voice. -Step 2: Implement the Component Interface +Step 3: Component Implementation Skeleton ========================================= -Implement the multi-pin audio adapter in ``src/audio/my_ecns/my_ecns.c`` using the SOF processing module framework: - -Module Lifecycle Architecture ------------------------------ - -A compliant SOF processing module must implement several key lifecycle handlers: - -1. **``init()``**: Allocates the component's private context structure (``struct my_ecns_comp_data``) using ``mod_alloc()`` and initializes default algorithmic parameters. -2. **``prepare()``**: Invoked before audio streaming commences. The module queries negotiated stream configurations, validates buffer parameters via ``comp_verify_params()``, and allocates large runtime memory buffers (e.g. AEC filter state, FFT scratch memory) using ``rballoc_align()`` with ``SOF_MEM_ZONE_SYS_RUNTIME`` and ``SOF_MEM_CAPS_RAM``. -3. **``process()``**: The core audio processing routine called repeatedly by the pipeline scheduler. -4. **``reset()`` and ``free()``**: Tears down the module, frees all dynamically allocated memory via ``rfree()``, and cleans up registered notifiers. -5. **Runtime IPC Tuning (``set_value()`` / ``get_value()``)**: Processes large configuration blobs sent from host user space (e.g. ``sof-ctl`` or ALSA mixer controls) to dynamically tune filter adaptation speeds, double-talk sensitivity, and noise attenuation decibels. - -Component Implementation Template ---------------------------------- +Implement the multi-pin audio adapter in ``src/audio/ecns/custom_ecns.c``: .. code-block:: c // SPDX-License-Identifier: BSD-3-Clause #include - #include - #include #include + #include #include - SOF_DEFINE_REG_UUID(my_ecns); - LOG_MODULE_REGISTER(my_ecns, CONFIG_SOF_LOG_LEVEL); + SOF_DEFINE_REG_UUID(custom_ecns); + LOG_MODULE_REGISTER(custom_ecns, CONFIG_SOF_LOG_LEVEL); - struct my_ecns_comp_data { + #define ECNS_CTRL_INDEX_CAPTURE_BOOST 0 + #define ECNS_CTRL_INDEX_TEST_SIGNAL 1 + + struct custom_ecns_comp_data { + struct ipc4_base_module_cfg base_cfg; + bool capture_boost; + bool capture_boost_req; + uint32_t test_signal_enabled; void *aec_state; - int period_frames; /* 320 frames for 20ms @ 16 kHz */ - int16_t scratch[1280] __aligned(16); }; - static int my_ecns_init(struct processing_module *mod) + static int custom_ecns_init(struct processing_module *mod) { - struct comp_dev *dev = mod->dev; - struct my_ecns_comp_data *cd; - - cd = mod_alloc(sizeof(*cd)); + struct custom_ecns_comp_data *cd = mod_alloc(sizeof(*cd)); if (!cd) return -ENOMEM; mod->priv_data = cd; - cd->period_frames = 320; /* 20ms period */ + cd->capture_boost = false; + cd->capture_boost_req = false; + cd->test_signal_enabled = 0; return 0; } - static int my_ecns_prepare(struct processing_module *mod) + static int custom_ecns_process(struct processing_module *mod, + struct sof_source **sources, int num_of_sources, + struct sof_sink **sinks, int num_of_sinks) { - struct my_ecns_comp_data *cd = module_get_private_data(mod); + struct custom_ecns_comp_data *cd = module_get_private_data(mod); + struct sof_source *src16k = sources[0]; /* Pin 0: 16 kHz 2ch */ + struct sof_source *src48k = sources[1]; /* Pin 1: 48 kHz 2ch */ + struct sof_sink *sink_kpb = sinks[0]; /* Pin 0: 16 kHz 1ch Clean */ + struct sof_sink *sink_host = sinks[1]; /* Pin 1: 48 kHz 2ch Clean */ - /* Allocate algorithm filter state cache-aligned to 16 bytes */ - cd->aec_state = rballoc_align(0, SOF_MEM_CAPS_RAM, - SOF_MEM_ZONE_SYS_RUNTIME, 16, - my_aec_get_state_size()); - if (!cd->aec_state) - return -ENOMEM; + /* 1. Handle live capture boost kcontrol switch request */ + if (cd->capture_boost != cd->capture_boost_req) + cd->capture_boost = cd->capture_boost_req; - my_aec_init(cd->aec_state, 16000, 4, 1); - return 0; - } + /* 2. Verify independent input data availability */ + if (source_get_data_available(src16k) < 1280) + return 0; /* Wait for full 20ms period @ 16 kHz */ - static int my_ecns_process(struct processing_module *mod, - struct sof_source **sources, int num_of_sources, - struct sof_sink **sinks, int num_of_sinks) - { - struct my_ecns_comp_data *cd = module_get_private_data(mod); - struct sof_source *src = sources[0]; - struct sof_sink *sink_clean_mono = sinks[0]; /* Pin 0 -> KPB */ - struct sof_sink *sink_clean_stereo = sinks[1]; /* Pin 1 -> Host */ - - int in_frames = source_get_data_frames_available(src); - int out_frames_0 = sink_get_free_frames(sink_clean_mono); - int out_frames_1 = sink_get_free_frames(sink_clean_stereo); - - /* Determine the maximum common frames that can be safely processed */ - int frames = MIN(in_frames, MIN(out_frames_0, out_frames_1)); - if (frames < cd->period_frames) - return 0; /* Wait until a full 20ms period (320 frames) is available */ - - /* Retrieve read pointer from 4-channel input and write pointers for sinks */ - int16_t *in_ptr = source_get_read_ptr(src); - int16_t *out_mono_ptr = sink_get_write_ptr(sink_clean_mono); - int16_t *out_stereo_ptr = sink_get_write_ptr(sink_clean_stereo); - - /* Execute custom acoustic echo cancellation & noise suppression */ - my_custom_aec_process(cd->aec_state, in_ptr, out_mono_ptr, - out_stereo_ptr, cd->period_frames); - - /* Commit consumed input frames and produced output frames */ - source_seek_read_ptr(src, cd->period_frames * source_get_frame_bytes(src)); - sink_seek_write_ptr(sink_clean_mono, cd->period_frames * sink_get_frame_bytes(sink_clean_mono)); - sink_seek_write_ptr(sink_clean_stereo, cd->period_frames * sink_get_frame_bytes(sink_clean_stereo)); + if (src48k && source_get_data_available(src48k) < 3840) + return 0; /* Wait for full 20ms period @ 48 kHz */ - return 0; - } + /* 3. Check independent sink free space */ + if (sink_kpb && sink_get_free_size(sink_kpb) < 640) + return 0; - static int my_ecns_reset(struct processing_module *mod) - { - struct my_ecns_comp_data *cd = module_get_private_data(mod); + if (sink_host && sink_get_free_size(sink_host) < 3840) + return 0; + + /* 4. Acquire buffer pointers and execute custom algorithm */ + const int16_t *in16k = source_get_read_ptr(src16k); + int16_t *out_kpb = sink_get_write_ptr(sink_kpb); - if (cd->aec_state) { - rfree(cd->aec_state); - cd->aec_state = NULL; + /* Extract Left channel or run AEC/NS */ + for (int i = 0; i < 320; i++) + out_kpb[i] = in16k[i * 2]; + + if (src48k && sink_host) { + const int16_t *in48k = source_get_read_ptr(src48k); + int16_t *out_host = sink_get_write_ptr(sink_host); + memcpy_s(out_host, 3840, in48k, 3840); + sink_seek_write_ptr(sink_host, 3840); + source_seek_read_ptr(src48k, 3840); } + + /* 5. Commit consumed and produced frames */ + source_seek_read_ptr(src16k, 1280); + sink_seek_write_ptr(sink_kpb, 640); + return 0; } -Step 3: Update Buffer Sizing in Topology 2.0 (IBS & OBS Deep Dive) -================================================================== +--- -In ALSA Topology 2.0, two essential configuration tokens govern audio buffer scheduling and memory allocation: +Step-by-Step Guide: Replacing the WOV / Keyword Detector Module +*************************************************************** -* **``ibs`` (Input Buffer Size)**: Specifies the minimum number of **bytes** required in the component's input buffer before the pipeline scheduler triggers the module's ``process()`` callback. -* **``obs`` (Output Buffer Size)**: Specifies the minimum number of **free bytes** required in the downstream sink buffer before the module can execute, and represents the byte quantity produced by the module in a single execution period. +In multi-slot keyword spotting architectures, multiple detector modules run in parallel pipelines, each monitoring the clean mono speech stream for a specific trigger phrase (Slot 0 for "strawberry", Slot 1 for "Hi Intel", Slot 2 for "orange"). Developers can replace the reference ``mww`` (microWakeWord) component with custom neural acoustic models, TensorFlow Lite for Microcontrollers (TFLM) graphs, or proprietary vendor engines (e.g. Picovoice Porcupine, Sensory TrulyHandsfree). -Why IBS and OBS are Critical to Pipeline Execution --------------------------------------------------- +Acoustic Inference Architecture +=============================== -1. **Topology Memory Allocation**: When the SOF kernel driver parses the topology manifest, it allocates inter-component circular ring buffers sized as integer multiples of ``ibs`` and ``obs`` (typically :math:`2 \times \text{ibs}` or :math:`3 \times \text{ibs}` to allow double or triple buffering). An incorrect token will allocate either insufficient memory (causing overrun) or excessive memory (wasting tightly constrained DSP SRAM). -2. **Scheduler Execution Thresholds**: The SOF pipeline scheduler queries ``source_get_data_available()`` and ``sink_get_free_size()``. It will **only** dispatch the module when: +The detector component executes under a 10 ms Data Processing period (160 frames at 16 kHz): - .. math:: +1. **Feature Extraction (MFCC)**: Batches 160 samples (10 ms) into 40-band Mel-frequency cepstral coefficients (184 bytes per chunk). +2. **Streaming Neural Network (TFLM)**: microWakeWord executes streaming convolutions using stateful temporal ring buffers, outputting a sigmoid confidence probability :math:`P \in [0.0, 1.0]`. - \text{data\_available} \ge \text{ibs} \quad \text{and} \quad \text{free\_space} \ge \text{obs} +Averaged Probability Trigger Mechanism +====================================== - If ``ibs`` is mistakenly set to 128 bytes (1 ms) while the C module internally waits for 2560 bytes (20 ms), the pipeline scheduler will wake the component 20 times per period, 19 of which will immediately exit without processing data. This introduces severe CPU scheduling overhead and prevents the DSP from entering low-power sleep states. -3. **Parameter Validation Handshake**: During stream startup, the firmware invokes ``comp_verify_params()`` to cross-check that the component's configured algorithmic frame chunk matches the topology's declared ``ibs`` and ``obs``. If a discrepancy exists, the driver fails with an IPC parameter error (``-EINVAL``). +.. note:: + **Overcoming Single-Frame Fluctuations**: + Neural network inference probabilities fluctuate significantly across consecutive audio hops during keyword utterance (e.g., 95% followed immediately by 22%). Requiring multiple consecutive inferences above 85% frequently causes missed wake-words. -Mathematical Calculation for Multi-Pin ECNS -------------------------------------------- + The subsystem implements a **4-inference moving window average**: + + .. math:: -The buffer size in bytes is determined by the standard chunk formula: + \bar{P} = \frac{1}{4} \sum_{i=0}^{3} P[n - i] -.. math:: + Detection triggers when :math:`\bar{P} \ge \text{MWW\_DETECT\_THRESHOLD}` (default **0.50** or 50%). Testing confirms that spoken keywords achieve 4-frame averages between 56% and 67%, while conversational speech remains reliably below 25%. - \text{Size (bytes)} = \text{Sample Rate (Hz)} \times \left(\frac{\text{Period (ms)}}{1000}\right) \times \text{Channels} \times \text{Bytes Per Sample} +Test Trigger Switch with 5-Second Delay (``wovdebug``) +====================================================== -For our 20 ms ECNS configuration at 16 kHz S16_LE (2 bytes per sample, 320 frames per chunk): +Each detector slot exposes a volatile ALSA mixer switch control named ``wovdebug_`` (e.g. ``wovdebug_111``, ``wovdebug_112``, ``wovdebug_113``): -* **Input Pin 0 (4 Channels)**: - :math:`16000 \times 0.020 \times 4 \times 2 = \mathbf{2560\text{ bytes}} \implies \mathbf{ibs\ 2560}` -* **Output Pin 0 (Mono Clean to KPB - 1 Channel)**: - :math:`16000 \times 0.020 \times 1 \times 2 = \mathbf{640\text{ bytes}} \implies \mathbf{obs\ 640}` -* **Output Pin 1 (Stereo Clean to Host - 2 Channels)**: - :math:`16000 \times 0.020 \times 2 \times 2 = \mathbf{1280\text{ bytes}} \implies \mathbf{obs\ 1280}` +* **Automated CI Validation**: Setting this control to ``1`` via ``amixer`` arms a synthetic test trigger. +* **5-Second Arming Delay**: Rather than triggering instantly, the component starts a 5000 ms countdown timer (configurable via ``CONFIG_COMP_MWW_FAKE_WAKE_MS``). The timer counts down during active audio processing in ``process()``. +* **Expiration & Reset**: Upon timer expiration, the component activates the Arbiter, commands KPB history buffer draining, and automatically pushes an ALSA notification resetting the control back to ``0``. +* **D0i3 State Resilience**: If the system transitions into low-power D0i3 while the timer is armed, the deadline is automatically re-calculated from D0i3 entry, guaranteeing that the target sleeps for the expected duration before firing wake. -Topology 2.0 Widget Declaration -------------------------------- +Per-Operator Inference Cycle Profiling +====================================== -In `tools/topology/topology2/dmic-wov-multi-4ch-manifest.conf `_, declare the ECNS widget with separate output audio format blocks matching each output pin index: +When ``CONFIG_COMP_MWW_PROFILE=y`` is enabled, the detector implements TensorFlow Lite Micro's ``MicroProfilerInterface``. The profiler reads Xtensa processor ``CCOUNT`` registers before and after every neural network operator in invoke order: -.. code-block:: text +* Evaluates execution cycles spent in ``CONV_2D``, ``DEPTHWISE_CONV_2D``, ``FULLY_CONNECTED``, and activation kernels. +* Excludes the initial cold subgraph invocation after reset. +* Logs a per-operator average cycle summary to DSP trace logs upon stream closure. - Object.Widget.ecns.1 { - uuid $MY_ECNS_UUID - num_input_pins 1 - num_output_pins 2 - num_input_audio_formats 1 - num_output_audio_formats 2 - - Object.Base.input_audio_format [ - { - in_rate 16000 - in_channels 4 - in_bit_depth 16 - in_valid_bit_depth 16 - ibs 2560 # 20ms @ 16 kHz 4ch S16_LE - } - ] - Object.Base.output_audio_format [ - # Pin 0: Mono clean to KPB (output_pin_index 0) - { - output_pin_index 0 - out_rate 16000 - out_channels 1 - out_bit_depth 16 - out_valid_bit_depth 16 - obs 640 # 20ms @ 16 kHz 1ch S16_LE - } - # Pin 1: Stereo clean to Host (output_pin_index 1) - { - output_pin_index 1 - out_rate 16000 - out_channels 2 - out_bit_depth 16 - out_valid_bit_depth 16 - obs 1280 # 20ms @ 16 kHz 2ch S16_LE - } - ] - } +Real-Time Volatile Scoring Controls (2 Hz) +========================================== ---- +To provide real-time UI feedback (such as signal strength meters or wake-word confidence visualization), each slot exposes a discrete enum scoring control (e.g. ``MWW 111 Score``): -Step-by-Step Guide: Replacing the WOV / Keyword Detector Module -*************************************************************** +.. important:: + **Kernel Dynamic Widget Constraint**: + In the Linux SOF driver (``sound/soc/sof/topology.c``), dynamic pipeline widgets are **prohibited** from including ``SNDRV_CTL_ELEM_ACCESS_VOLATILE`` in their topology manifests: -In multi-slot keyword spotting architectures, multiple detector modules run in parallel pipelines, each monitoring the clean mono speech stream for a specific trigger phrase (e.g. Slot 0 for "strawberry", Slot 1 for "banana", Slot 2 for "orange"). Developers can replace the reference ``mww`` (microWakeWord) or ``detect_test`` component with custom neural acoustic models, TensorFlow Lite for Microcontrollers (TFLM) models, or third-party engines (e.g. Picovoice Porcupine, Sensory TrulyHandsfree). + .. code-block:: c -Step 1: Implement the Detector Interface -======================================== + if (pipe_widget->dynamic_pipeline_widget) { + if (scontrol->access & SNDRV_CTL_ELEM_ACCESS_VOLATILE) + return -EINVAL; /* Rejects topology loading! */ + } -A keyword detector component in SOF must handle two concurrent responsibilities: -1. **Real-Time Acoustic Inference**: Continuously stream mono 16 kHz audio frames (typically in 10 ms chunks = 160 frames), extract features (e.g. Mel filterbanks), and execute the neural network classifier. -2. **Asynchronous Arbitration & State Synchronization**: Respond to Arbiter control events (pause/resume) and trigger downstream pre-roll draining when a keyword is recognized. + **Architecture Solution**: Omit ``volatile`` from the topology ``access []`` definition. Instead, the DSP firmware calculates peak confidence over 50 hops (500 ms = 2 Hz) and pushes asynchronous updates using ``SOF_IPC4_MODULE_NOTIFICATION`` with ``SOF_IPC4_ENUM_CONTROL_PARAM_ID``. The kernel notification handler updates its internal cached value directly and dispatches ``snd_ctl_notify()``. -Streaming Detector Architecture -------------------------------- +Detector Implementation Skeleton +================================ -In ``src/audio/my_wov/my_wov.c``, implement the detector module adapter: +Implement the detector module in ``src/audio/my_wov/my_wov.c``: .. code-block:: c // SPDX-License-Identifier: BSD-3-Clause #include - #include + #include #include #include + #include - SOF_DEFINE_REG_UUID(my_wov); - LOG_MODULE_REGISTER(my_wov, CONFIG_SOF_LOG_LEVEL); + #define MWW_DETECT_AVG_LEN 4 + #define MWW_DETECT_THRESHOLD 0.50f struct my_wov_comp_data { - int slot_id; /* 0, 1, or 2 assigned via topology */ - bool paused; /* Suppressed when a sibling slot wins */ - bool detected; /* Latched true upon keyword trigger */ - void *model_context; /* Neural network / TFLM runtime state */ + uint8_t slot_id; + bool paused; + float prob_hist[MWW_DETECT_AVG_LEN]; + uint32_t prob_hist_idx; + struct kpb_event_data event_data; + struct kpb_client client_data; }; - /* Notifier callback: Arbiter commands this slot to PAUSE or RESUME */ - static void my_wov_ctrl_cb(void *arg, enum notify_id type, void *data) - { - struct comp_dev *dev = arg; - struct my_wov_comp_data *cd = module_get_private_data(dev->mod); - struct wov_ctrl_event_data *ctrl = data; - - if (ctrl->cmd == WOV_CMD_PAUSE) { - /* Sibling slot triggered: pause local inference to save DSP MCPS */ - cd->paused = true; - } else if (ctrl->cmd == WOV_CMD_RESUME) { - /* Host stream reset: re-arm detector for the next trigger */ - cd->paused = false; - cd->detected = false; - my_model_reset(cd->model_context); - } - } - - static int my_wov_init(struct processing_module *mod) - { - struct comp_dev *dev = mod->dev; - struct my_wov_comp_data *cd; - - cd = mod_alloc(sizeof(*cd)); - if (!cd) - return -ENOMEM; - - mod->priv_data = cd; - cd->slot_id = dev->ipc_config.index; /* Slot ID from topology widget index */ - cd->paused = false; - cd->detected = false; - return 0; - } - - static int my_wov_prepare(struct processing_module *mod) + static void my_wov_trigger_detection(struct processing_module *mod) { struct comp_dev *dev = mod->dev; struct my_wov_comp_data *cd = module_get_private_data(mod); - /* Initialize acoustic model engine and allocate scratch buffers */ - cd->model_context = my_model_init(); - if (!cd->model_context) - return -ENOMEM; - - /* Register callback to receive PAUSE / RESUME commands from Arbiter */ - notifier_register(dev, cd, NOTIFIER_ID_WOV_CTRL, my_wov_ctrl_cb, 0); - return 0; - } - - static int my_wov_process(struct processing_module *mod, - struct sof_source **sources, int num_of_sources, - struct sof_sink **sinks, int num_of_sinks) - { - struct my_wov_comp_data *cd = module_get_private_data(mod); - struct sof_source *source = sources[0]; - struct sof_sink *sink = sinks[0]; - int frames = source_get_data_frames_available(source); - - if (frames == 0) - return 0; + /* 1. Activate Arbiter FIRST */ + struct wov_detect_notif arb_notif = { .slot_id = cd->slot_id }; + notifier_event(dev, NOTIFIER_ID_WOV_DETECT, + NOTIFIER_TARGET_CORE_ALL_MASK, + &arb_notif, sizeof(arb_notif)); - /* If active and not paused by arbiter, run acoustic model inference */ - if (!cd->paused && !cd->detected) { - if (my_model_detect_keyword(cd->model_context, source, frames)) { - cd->detected = true; + /* 2. Command KPB to drain history buffer */ + cd->client_data.id = cd->slot_id; + cd->client_data.drain_req = 2000; /* 2.0s pre-roll */ + cd->event_data.event_id = KPB_EVENT_BEGIN_DRAINING; + cd->event_data.client_data = &cd->client_data; - /* 1. Command KPB to drain 2.0s pre-roll buffer to active slot */ - /* 2. Inform Arbiter that this slot won detection */ - my_wov_trigger_detection(mod->dev, cd->slot_id); - } - } + /* Write back cache for cross-core coherency */ + dcache_writeback_region((__sparse_force void __sparse_cache *)&cd->client_data, + sizeof(cd->client_data)); - /* Forward audio downstream to maintain pipeline continuity */ - source_to_sink_copy(source, sink, true, frames * source_get_frame_bytes(source)); - return 0; + notifier_event(dev, NOTIFIER_ID_KPB_CLIENT_EVT, + NOTIFIER_TARGET_CORE_ALL_MASK, + &cd->event_data, sizeof(cd->event_data)); } - static int my_wov_reset(struct processing_module *mod) + static void on_wov_ctrl(void *arg, enum notify_id id, void *data) { - struct comp_dev *dev = mod->dev; - struct my_wov_comp_data *cd = module_get_private_data(mod); - - /* Mandatory: Unregister notifier to avoid dangling callbacks */ - notifier_unregister(dev, cd, NOTIFIER_ID_WOV_CTRL); + struct comp_dev *dev = arg; + struct my_wov_comp_data *cd = module_get_private_data(dev->mod); + const struct wov_ctrl_notif *n = data; - if (cd->model_context) { - my_model_free(cd->model_context); - cd->model_context = NULL; + if (n->cmd == WOV_ARB_CMD_PAUSE && n->slot_id != cd->slot_id) { + cd->paused = true; + } else if (n->cmd == WOV_ARB_CMD_RESUME) { + cd->paused = false; + memset(cd->prob_hist, 0, sizeof(cd->prob_hist)); } - return 0; } -Step 2: Configure Model Parameters in Topology -============================================== - -To integrate the custom keyword detector into ALSA Topology 2.0, declare the widget class in `tools/topology/topology2/include/components/my_wov.conf`: - -Understanding the Topology 2.0 Widget Definition ------------------------------------------------- - -* **Attributes**: - - ``index``: Pipeline index where the widget is instantiated. - - ``instance``: Unique instance number across all detector widgets. - - ``cpc``: Cycles Per Chunk (CPC) estimate for the scheduler's dynamic load balancer. -* **UUID Binding**: - - The ``uuid`` string must match the 16-byte UUID defined via ``SOF_DEFINE_REG_UUID(my_wov)`` in the C driver. -* **Pin Counts**: - - Single input pin (``num_input_pins 1``) receiving 1ch clean mono speech from KPB. - - Single output pin (``num_output_pins 1``) routing to ``wov_arbiter``. -* **Buffer Constraints**: - - Sized for a 10 ms Data Processing period: :math:`16000 \times 0.010 \times 1 \times 2 = \mathbf{320\text{ bytes}}` (``ibs 320``, ``obs 320``). +--- -Widget Class Definition ------------------------ +WOV Arbiter & Multi-Slot Coordination +************************************* -.. code-block:: text +The ``wov-arbiter`` component synchronizes multi-slot pipelines, arbitrates between competing detection triggers, and routes the winning audio stream into ``host-copier.12.capture`` (ALSA PCM 12). - Class.Widget."my_wov" { - DefineAttribute."index" { type "integer" } - DefineAttribute."instance" { type "integer" } - DefineAttribute."cpc" { token_ref "comp.word" } +Arbiter State Machine & Control Enum +==================================== - +The arbiter exposes the volatile read-only ALSA mixer control **``DMIC Phrase Detect ID``** (or **``Jack In Phrase Detect ID``** for headset instances): - attributes { - !constructor [ "index", "instance" ] - !mandatory [ "uuid", "num_input_audio_formats", "num_output_audio_formats" ] - unique "instance" - } +.. list-table:: Phrase Detect ID Control States + :widths: 15 20 65 + :header-rows: 1 - uuid "1f:d5:a8:eb:27:78:b5:47:82:ee:de:6e:77:43:af:67" - type "effect" - no_pm "true" - num_input_pins 1 - num_output_pins 1 - } + * - Enum Value + - Item String + - Operational State + * - ``0`` + - ``Listening`` + - All detector slots actively scanning microphone audio. + * - ``1`` + - ``Slot 1`` + - Slot 0 ("strawberry") triggered. Sibling slots paused. Pre-roll + live audio streaming to host. + * - ``2`` + - ``Slot 2`` + - Slot 1 ("Hi Intel") triggered. Sibling slots paused. Pre-roll + live audio streaming to host. + * - ``3`` + - ``Slot 3`` + - Slot 2 ("orange") triggered. Sibling slots paused. Pre-roll + live audio streaming to host. + +Mutual Exclusion (Single Active WoV PCM) +======================================== -In the multi-slot manifest (`dmic-wov-multi-4ch-manifest.conf `_), instantiate the widget within each detector pipeline (Pipelines 101, 102, 103), binding the model-specific binary data blobs (such as trained neural network weight files or detection threshold structures). +Because DMIC and headset WoV graphs share ``NOTIFIER_ID_WOV_DETECT`` and DSP memory accommodates one active neural network working set, the arbiter enforces strict mutual exclusion. Opening a second WoV PCM stream while one is already active is immediately rejected with an error, preventing memory collisions. --- -Scheduling, Periods & Core Affinity in Topology 2.0 +Scheduling, Periods & Buffer Sizing in Topology 2.0 *************************************************** -Configuring a high-performance audio graph requires matching **Execution Domains**, **Scheduling Periods**, and **Multi-Core Affinities**: - .. figure:: images/wov_scheduling_periods.svg :alt: WOV Scheduling Timeline and Period Sizing :width: 100% :align: center - Figure 323: Scheduling domains, period boundaries (1ms LL vs 10ms/20ms DP), and multi-core affinity assignments. - -Low-Latency (LL) vs. Data Processing (DP) Domains -================================================= - -1. **Low-Latency Domain (``lp_mode 0``, 1 ms period)**: - - Synchronized directly to the hardware 1 ms timer tick. - - Processes small sample chunks (16 frames at 16 kHz). - - Applied to hardware DAIs (``dai-copier``), ``wov_arbiter``, and host PCMs (``host-copier``) to maintain minimum end-to-end latency. -2. **Data Processing Domain (``lp_mode 1``, 10 ms or 20 ms period)**: - - Batches samples into larger algorithmic frames (160 frames for 10 ms; 320 frames for 20 ms). - - Applied to compute-intensive algorithms (ECNS, MFCC, microWakeWord, TFLM). - - **Power Optimization**: Allows the DSP core to execute heavy vector math in a brief burst and sleep in low-power idle states between chunks. + Figure: Execution timeline across Low-Latency (LL 1ms) and Data Processing (DP 10ms / 20ms) domains, mathematical IBS/OBS buffer sizing, and multi-core task balancing. -Mathematical Period & Buffer Sizing Rule -======================================== +Buffer Sizing Formula (IBS & OBS) +================================= -When configuring an audio component in ALSA Topology 2.0, input buffer size (``ibs``) and output buffer size (``obs``) must satisfy the sample chunk equation: +In ALSA Topology 2.0, input buffer size (``ibs``) and output buffer size (``obs``) govern memory allocation and scheduling wakeups: .. math:: \text{Buffer Size (bytes)} = \text{Sample Rate (Hz)} \times \left(\frac{\text{Period (ms)}}{1000}\right) \times \text{Channels} \times \text{Bytes Per Sample} -.. list-table:: Standard Buffer Sizing Lookup Matrix (16 kHz, S16_LE = 2 Bytes) - :widths: 25 20 20 35 +.. list-table:: Dual-Rate Buffer Sizing Lookup Matrix (S16_LE = 2 Bytes Per Sample) + :widths: 22 15 15 18 30 :header-rows: 1 - * - Processing Block + * - Component Block - Period + - Rate - Channels - - Calculated Buffer Size (IBS / OBS) - * - **LL 1ms Capture** - - 1 ms - - 4 channels - - :math:`16000 \times 0.001 \times 4 \times 2 = \mathbf{128\text{ bytes}}` - * - **DP 10ms Keyword Slot** - - 10 ms - - 1 channel (mono) - - :math:`16000 \times 0.010 \times 1 \times 2 = \mathbf{320\text{ bytes}}` - * - **DP 20ms ECNS Input** - - 20 ms - - 4 channels - - :math:`16000 \times 0.020 \times 4 \times 2 = \mathbf{2560\text{ bytes}}` - * - **DP 20ms Clean Mono** - - 20 ms - - 1 channel (mono) - - :math:`16000 \times 0.020 \times 1 \times 2 = \mathbf{640\text{ bytes}}` - * - **DP 20ms Clean Stereo** - - 20 ms + - Calculated Size (IBS / OBS) + * - **48k DAI Capture** (P110) + - 1 ms (LL) + - 48,000 Hz - 2 channels - - :math:`16000 \times 0.020 \times 2 \times 2 = \mathbf{1280\text{ bytes}}` - -.. warning:: - If the topology ``ibs`` or ``obs`` token differs from the buffer size expected by the firmware module, pipeline initialization will fail with an IPC buffer alignment error (``-EINVAL``) during ``comp_verify_params()``. + - :math:`48000 \times 0.001 \times 2 \times 2 = \mathbf{192\text{ bytes}}` + * - **16k DAI Capture** (P119) + - 1 ms (LL) + - 16,000 Hz + - 2 channels + - :math:`16000 \times 0.001 \times 2 \times 2 = \mathbf{64\text{ bytes}}` + * - **ECNS Pin 0 In** (16k) + - 20 ms (DP) + - 16,000 Hz + - 2 channels + - :math:`16000 \times 0.020 \times 2 \times 2 = \mathbf{1280\text{ bytes}}` (``ibs``) + * - **ECNS Pin 1 In** (48k) + - 20 ms (DP) + - 48,000 Hz + - 2 channels + - :math:`48000 \times 0.020 \times 2 \times 2 = \mathbf{3840\text{ bytes}}` (``ibs``) + * - **ECNS Pin 0 Out** (Clean Mono) + - 20 ms (DP) + - 16,000 Hz + - 1 channel + - :math:`16000 \times 0.020 \times 1 \times 2 = \mathbf{640\text{ bytes}}` (``obs``) + * - **ECNS Pin 1 Out** (Clean Stereo) + - 20 ms (DP) + - 48,000 Hz + - 2 channels + - :math:`48000 \times 0.020 \times 2 \times 2 = \mathbf{3840\text{ bytes}}` (``obs``) + * - **KPB Pre-Roll Buffer** (P116) + - 2000 ms + - 16,000 Hz + - 1 channel + - :math:`16000 \times 2.000 \times 1 \times 2 = \mathbf{64000\text{ bytes}}` (~64 KB) + * - **MWW Detector Slot** (P111–113) + - 10 ms (DP) + - 16,000 Hz + - 1 channel + - :math:`16000 \times 0.010 \times 1 \times 2 = \mathbf{320\text{ bytes}}` (``ibs/obs``) Cross-Core Task Affinity (``core_id``) ====================================== -Multi-keyword spotting workloads execute continuous STFT framing, Mel filterbank feature generation, and neural network matrix multiplications. On multi-core DSP architectures (such as Intel cAVS 2.5 on Tiger Lake and ACE on Meteor Lake / Arrow Lake / Panther Lake), running three or more concurrent keyword detectors on Core 0 can saturate the primary core, starving real-time audio copiers and causing audible buffer underruns. - -Inter-Core Audio Routing Architecture -------------------------------------- - -To achieve deterministic real-time performance, SOF supports cross-core task affinity in ALSA Topology 2.0: - -* **Core 0 (Primary Core)**: Runs real-time hardware copiers and preprocessing: - - Pipeline 100 (DAI Capture, LL 1ms) - - Pipeline 105 (ECNS Processing, DP 20ms) - - Pipeline 106 (KPB History Buffer, DP 20ms) - - Pipeline 101 (Slot 0 Detector, DP 10ms) - - Pipeline 102 (Slot 1 Detector, DP 10ms) - - Pipeline 104 (WOV Arbiter & PCM 11 Copier, LL 1ms) - - Pipeline 107 (ECNS Host Copier & PCM 10, LL 1ms) -* **Core 1 (Secondary Core)**: Offloaded compute-intensive detector: - - Pipeline 103 (Slot 2 Detector, DP 10ms) - -SOF manages inter-core audio communication locklessly using decoupled ring buffers located in shared DSP SRAM. When Pipeline 106 on Core 0 pushes audio into its output mixin (``mixin.106.1``), Pipeline 103 on Core 1 reads from ``mixout.103.1`` across DSP cache boundaries without requiring kernel mutex locks or blocking synchronization. - -Configuring Core Affinity in Topology 2.0 ------------------------------------------ - -To assign a pipeline to a secondary core, set the ``core_id`` token in the pipeline widget definition: - -.. code-block:: text - - # Pipeline 103: WOV Slot 2 offloaded to DSP Core 1 - Object.Pipeline."custom-dp-capture" [ - { - index 103 +On multi-core DSP architectures (such as Intel cAVS 2.5 on Tiger Lake and ACE on Panther Lake / Wildcat Lake), running three concurrent keyword detectors on Core 0 can cause core saturation. SOF supports cross-core task affinity in Topology 2.0: - Object.Widget.pipeline.1 { - priority 0 - lp_mode 1 - core_id 1 # Cross-core execution on Core 1! - } - } - ] +* **Core 0 (Primary Core)**: Pipelines 110, 119, 115 (ECNS), 116 (KPB), 111 (Slot 0), 112 (Slot 1), 114 (Arbiter), 117 (Host Capture). +* **Core 1 (Secondary Core)**: Pipeline 113 (Slot 2 Detector) assigned via ``core_id 1``. Inter-core audio frames are transferred locklessly across DSP cache boundaries via SRAM buffers. --- Building, Deploying & Verifying the Pipeline ******************************************** -Compiling Topology with the NHLT Preprocessor Plugin -==================================================== +Target Topology Manifests +========================= -The Non-HD-Audio Link Table (NHLT) is an ACPI BIOS data structure that defines microphone physical geometry, hardware PDM decimation filters, clock configurations, and vendor-specific audio endpoints. +.. list-table:: Target Topology Manifest Matrix + :widths: 25 35 40 + :header-rows: 1 -Under normal circumstances, the SOF Linux driver parses the NHLT table supplied by the host PC's BIOS. However, on pre-production development hardware, BIOS ACPI tables frequently lack 4-channel native 16 kHz DMIC descriptors (often providing only generic 2-channel 48 kHz tables). To resolve this, the ALSA topology compiler (``alsatplg``) utilizes a dedicated **NHLT Preprocessor Plugin** to generate and embed a validated 16 kHz 4-channel NHLT blob directly into the binary topology file: + * - Target Platform + - Manifest File + - Binary Topology Output + * - **Panther Lake (PTL / ACE 3.0)** + - ``dmic-wov-multi-ptl-4ch-manifest.conf`` + - ``sof-ptl-dmic-wov-multi-4ch.tplg`` + * - **Tiger Lake (TGL / CAVS 2.5)** + - ``dmic-wov-multi-4ch-manifest.conf`` + - ``sof-tgl-dmic-wov-multi-4ch.tplg`` + * - **Wildcat Lake (WCL / ACE 3.0)** + - Symlink to Panther Lake topology + - ``sof-wcl-dmic-wov-multi-4ch.tplg`` + * - **SoundWire Headset WoV** + - ``sdw-jack-wov-multi.conf`` + - Embedded in SoundWire codec topologies + +Compiling Topology with NHLT Preprocessor Plugin +================================================ .. code-block:: bash @@ -736,7 +642,7 @@ Under normal circumstances, the SOF Linux driver parses the NHLT table supplied ln -sf $TPLG2/include $ALSA_TMP/include ln -sf $TPLG2/platform $ALSA_TMP/platform - # Compile Native 16 kHz 4-Channel Multi-WOV Topology with embedded NHLT + # Compile Multi-WOV Topology with embedded NHLT ALSA_CONFIG_DIR=$ALSA_TMP ALSA_TOPOLOGY_PLUGIN_DIR=/usr/lib/alsa-topology alsatplg \ -I $TPLG2 -p \ -c tools/topology/topology2/dmic-wov-multi-4ch-manifest.conf \ @@ -745,16 +651,14 @@ Under normal circumstances, the SOF Linux driver parses the NHLT table supplied BIOS NHLT Table Override (``sof_use_tplg_nhlt=1``) ================================================== -To instruct the Linux SOF kernel driver to bypass the incomplete host BIOS ACPI table and consume the NHLT table packaged inside the topology file, configure the driver module option ``sof_use_tplg_nhlt=1``. - -Add to `/etc/modprobe.d/sof.conf` on the target device: +Add to ``/etc/modprobe.d/sof.conf`` on the target DUT: .. code-block:: ini options snd_sof tplg_path=intel/sof-ipc4-tplg tplg_filename=sof-tgl-dmic-wov-multi-4ch.tplg options snd_sof_intel_hda_common sof_use_tplg_nhlt=1 -Reload the audio driver stack to apply the override: +Reload the audio driver stack: .. code-block:: bash @@ -764,71 +668,94 @@ Reload the audio driver stack to apply the override: modprobe snd_sof_pci_intel_tgl ' -Inspect the kernel log via ``dmesg | grep -i nhlt`` to confirm successful activation: - -.. code-block:: text - - sof-audio-pci-intel-tgl: using topology NHLT table instead of ACPI - -Verifying Both Audio Streams on Target Hardware -=============================================== +Verifying Streams & Controls on Hardware +======================================== Step 1: Inspect ALSA Capture Endpoints -------------------------------------- -Verify that the kernel has enumerated both host copier endpoints: - .. code-block:: bash ssh root@ 'arecord -l' -Expected output should list both PCM 10 and PCM 11: +Expected output listing PCM 11 and PCM 12: .. code-block:: text - card 0: sofhdadsp [sof-hda-dsp], device 10: ECNS Capture (*) [] + card 0: sofhdadsp [sof-hda-dsp], device 11: DMIC ECNS Capture (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 - card 0: sofhdadsp [sof-hda-dsp], device 11: WOV Capture (*) [] + card 0: sofhdadsp [sof-hda-dsp], device 12: DMIC Multi-WOV (*) [] Subdevices: 1/1 Subdevice #0: subdevice #0 -Step 2: Verify Simultaneous ECNS Teleconferencing Audio (PCM 10) ----------------------------------------------------------------- +Step 2: Capture Continuous ECNS Clean Speech (PCM 11) +----------------------------------------------------- -Record a 5-second sample from PCM 10 to confirm that the ECNS module is actively processing 2-channel clean speech: +Record a 5-second sample from PCM 11 at 48 kHz stereo: .. code-block:: bash - ssh root@ 'arecord -D hw:0,10 -c 2 -r 16000 -f S16_LE -d 5 /tmp/ecns_clean.wav' - -Play back `/tmp/ecns_clean.wav` to verify that speaker playback echo is cancelled and ambient room noise is suppressed. + ssh root@ 'arecord -D hw:0,11 -c 2 -r 48000 -f S16_LE -d 5 /tmp/ecns_clean.wav' Step 3: Monitor Active Keyword Slot State via ALSA Mixer -------------------------------------------------------- -Query the ALSA mixer control exposed by the WOV Arbiter to observe slot detection status: +Query the active phrase detection state: .. code-block:: bash - ssh root@ "amixer -c0 sget 'wov_active_slot'" + ssh root@ "amixer -c0 cget name='DMIC Phrase Detect ID'" -Output states: -* ``0``: Idle / Listening mode across all detector slots. -* ``1``: Slot 0 detected ("strawberry"). -* ``2``: Slot 1 detected ("banana"). -* ``3``: Slot 2 detected ("orange"). +Output indicates: +* ``0``: Listening across all detector slots. +* ``1``: Slot 1 detected ("strawberry"). +* ``2``: Slot 2 detected ("Hi Intel"). +* ``3``: Slot 3 detected ("orange"). -Step 4: Capture Triggered Keyword Audio with 2.0s Pre-Roll (PCM 11) +Step 4: Capture Triggered Keyword Audio with 2.0s Pre-Roll (PCM 12) ------------------------------------------------------------------- -When speaking a trigger phrase (e.g. "strawberry"), capture the drained audio stream from PCM 11: +When speaking a trigger phrase (e.g. "Hi Intel"), record the drained audio stream from PCM 12: .. code-block:: bash - ssh root@ 'arecord -D hw:0,11 -c 1 -r 16000 -f S16_LE -d 4 /tmp/wov_trigger.wav' + ssh root@ 'arecord -D hw:0,12 -c 1 -r 16000 -f S16_LE -d 4 /tmp/wov_trigger.wav' + +Automated Testing & Diagnostic Tools +==================================== + +The SOF tree provides dedicated testing and capture utilities under ``tools/wov_capture/``: + +1. ``wov_capture_app`` +---------------------- + +A standalone C capture tool communicating directly via Linux ALSA PCM kernel UAPI ioctls with zero external library dependencies: + +* **Keyword-Gated Readiness**: Explicitly waits for PCM readiness when capturing from gated WOV devices. +* **CLI Options**: + - ``-D CARD``: Sound card index (default 0). + - ``-d DEV``: PCM device number (e.g. ``-d 12`` for WOV; ``-d 11`` for ECNS). + - ``-r RATE``: Sample rate in Hz (e.g. ``-r 16000`` or ``-r 48000``). + - ``-c CHANS``: Number of audio channels (e.g. ``-c 1`` for mono; ``-c 2`` for stereo). + - ``-f FORMAT``: Sample format (``S16_LE`` or ``S32_LE``). + - ``-t DURATION``: Capture duration in seconds. + - ``-n CYCLES``: Number of automated capture cycles. + - ``-o DIR``: Output directory for standard RIFF WAV files. + +2. Automated Multi-Slot Test Suite (``wov_multi_slot_test.py``) +--------------------------------------------------------------- + +Automates trigger injection and state verification across all three detector slots: + +.. code-block:: bash + + python3 tools/wov_capture/wov_multi_slot_test.py --card 0 --device 12 + +3. Automated 10x S0 / D0i3 Test Runner (``run_wov_10x.py``) +----------------------------------------------------------- -Verify the recorded waveform in an audio editor (e.g. Audacity or MHWaveEdit). The file should contain approximately 2.0 seconds of history buffer audio preceding the trigger point, followed seamlessly by live spoken speech. +Validates power transitions and driver stability across 10 consecutive S0 active captures and 10 consecutive low-power D0i3 wake cycles. --- @@ -842,21 +769,27 @@ Summary: Developer "Watch Out" Checklist * - Domain - Common Failure Mode - Correct Engineering Rule + * - **Notification Order** + - Signaling KPB drain before notifying the Arbiter. + - **Always activate Arbiter first** (``NOTIFIER_ID_WOV_DETECT``), then command KPB to drain (``NOTIFIER_ID_KPB_CLIENT_EVT``). * - **Multi-Pin Output** - Attempting to output both mono and stereo streams from a single output pin. - - Declare 2 output pins on ECNS: Pin 0 = 1ch mono (to KPB), Pin 1 = 2ch stereo (to Host Copier 10). + - Declare 2 output pins on ECNS: Pin 0 = 16k 1ch mono (to KPB), Pin 1 = 48k 2ch stereo (to Host PCM 11). * - **Buffer Sizing (IBS/OBS)** - - Mismatched ``ibs`` or ``obs`` in topology versus component frame chunk size. - - Calculate exact byte size: :math:`\text{Rate} \times \text{Period} \times \text{Channels} \times \text{Bytes}`. - * - **Notifier Callbacks** - - Forgetting to register for ``NOTIFIER_ID_WOV_CTRL``. - - All detector slots must handle ``WOV_CMD_PAUSE`` and ``WOV_CMD_RESUME`` to prevent race conditions. - * - **Pre-Roll Sizing** - - Setting KPB history buffer depth too small in topology. - - Set ``KPB_BUFF_TIME_MS "2000"`` (2.0s = 64 KB mono) to capture complete trigger words. - * - **Cross-Core Affinity** - - Placing all compute-intensive neural network models on Core 0. - - Assign secondary detector slots to ``core_id 1`` in Topology 2.0 to distribute DSP utilization. - * - **NHLT ACPI Descriptors** - - DMIC failing to open natively at 16 kHz on 4 channels due to outdated BIOS ACPI table. - - Enable ``options snd_sof_intel_hda_common sof_use_tplg_nhlt=1`` to override with topology NHLT. + - Declaring mismatched ``ibs`` or ``obs`` in topology versus component frame chunk size. + - Calculate exact byte sizes: Pin 0 In = 1280 B, Pin 1 In = 3840 B, Pin 0 Out = 640 B, Pin 1 Out = 3840 B. + * - **Cross-Core Coherency** + - Passing nested event pointers to Core 0 without dcache writeback. + - Call ``dcache_writeback_region()`` on ``client_data`` before dispatching ``notifier_event()`` to remote cores. + * - **Dynamic Widget Controls** + - Specifying ``volatile`` in topology ``access []`` for dynamic widgets. + - Omit ``volatile`` from topology to avoid kernel ``-EINVAL``. Dispatch asynchronous ``SOF_IPC4_MODULE_NOTIFICATION`` from firmware. + * - **Model Naming** + - Referencing obsolete "banana" model for Slot 1. + - Use updated **"Hi Intel"** model (``mww_model_data_hi_intel``) on Slot 1 (Pipeline 112). + * - **Mixer Control Naming** + - Looking for legacy ``wov_active_slot`` control name. + - Query **``DMIC Phrase Detect ID``** for DMIC topologies and **``Jack In Phrase Detect ID``** for headset topologies. + * - **Mutual Exclusion** + - Attempting to open DMIC and headset WOV streams simultaneously. + - The arbiter permits only one active WoV PCM stream at a time.