[lgirdwood]

Summary

Adds extended telemetry sections to Memory Window 2 (TEL2 v2.0) and Memory Window 3 (PMW2 v2.0) to support real-time host monitoring tools such as sof-top (see thesofproject/sof-utils#2) without invasive IPC polling during active audio streaming.

Memory Window 2 (TEL2, 0x0100..0x0AFF)

• Directory Header (0x0100, 64B): TEL2 magic, v2.0, seqlock counter, microsecond timestamp, and 5 section descriptors.
• Hot-Path Cache Lines (0x0140..0x023F):
  • Line 0 (0x0140, 64B): per-core LL overruns, DP deadline misses, total/per-pipeline XRUNs, IPC count/max latency, and active LL task count.
  • Lines 1–2 (0x0180..0x01FF, 16 x 8B): per-gateway active bytes/frames, XRUN counts, and error flags.
• Memory Allocators Section (0x0240, 256B):
  • L2 HP-SRAM runtime/system heaps, LPSRAM heap, fast-get pool, physical vpage pool, blob pool, shared memory, and Zephyr system heap (used, free, peak HWM, and largest free block).
• Virtual Regions Section (0x0340, 512B):
  • Up to 20 vregion descriptors reporting virtual size, committed bytes, allocated bytes, free blocks, and peak HWM.
• Pipelines & Per-Core Scheduler Section (0x0540, 384B):
  • Up to 16 active pipeline records (ID, core, priority, state, component count, period, scheduler ID, last XRUN bytes) and 4 per-core scheduler records (active KCPS budget, LL/DP task counts).
• Gateways & DAIs Cold Section (0x06C0, 256B):
  • Up to 16 gateway cold descriptors (type, index, direction, DMA channel, component ID, state, group ID, period bytes, DMA buffer size).
• Zephyr RTOS Threads Section (0x07C0, 832B):
  • Up to 24 Zephyr thread slots (thread ID, core affinity, priority, state, stack size/used/HWM, runtime cycle delta/total, and name).

Memory Window 3 (PMW2)

• Extends the performance measurements window with a PMW2 trailer, per-module cold resource table (sof_perf_mod_cold_item: heap usage, heap HWM, configured CPC), and per-module hot counters (sof_perf_mod_hot_item: CPC violations and XRUN counts).

Systick / Idle Core Cycle Fix

• Updates telemetry_update() to accumulate elapsed active LL tick cycles (current_stamp - begin_stamp) rather than inter-tick wall-clock ccount deltas, and resets idle core tick metrics when ll_task_count drops to zero.

[Copilot]

🟡 Changes recommended

Telemetry still permits MW3 table overflow and unsynchronized collection of objects that may be freed concurrently, creating critical memory-corruption and use-after-free risks.

17 open issues

• Bound telemetry entries to the actual memory partition capacity · New
• Protect IPC component traversal against concurrent deletion · New
• Protect telemetry slot collection from concurrent destruction · New
• Protect vregion lifetimes and mutable statistics during collection · New
• Refresh PMW3 heap usage after module frees · New
• Publish pipeline state after transitions and refresh after the last callback · New
• Synchronize cross-core staging buffers before publication · New
• Serialize cold-section writers to prevent partial snapshots · New
• Map pipeline IDs to stable tracked slots · New
• Connect the deadline-miss helper to DP scheduler accounting · New
• Clear the gateway active bit before the early return · New
• Count DAI XRUNs when the error is detected · New
• Publish host skips from the no-bytes path · New
• Reclaim thread history entries for departed threads · New
• Decouple telemetry publication from core 0 audio ticks · New
• Preserve the full 32-bit component token in telemetry · New
• Report actual free SRAM capacity, not powered-on capacity · New

What changed in this PR

Extends SOF’s shared telemetry windows to support host monitoring tools such as sof-top without additional IPC polling during audio streaming.

Changes:

• Adds TEL2 sections for memory, virtual regions, pipelines, gateways, and Zephyr threads.
• Adds PMW2 module resource records and hot counters, and relocates I/O performance data.
• Changes systick accounting to measure active LL execution cycles and adds idle-metric resets.

File	Description
zephyr/​lib/​vregion.c	Registers virtual regions and collects their statistics.
zephyr/​lib/​vpage.c	Tracks page allocation usage and failures.
zephyr/​lib/​regions_mm.c	Exports virtual-heap block statistics.
zephyr/​lib/​fast-get.c	Collects cache-pool usage and allocation counters.
zephyr/​lib/​alloc.c	Collects heap usage, failures, and SRAM statistics.
zephyr/​include/​sof/​lib/​regions_mm.h	Declares the virtual-heap telemetry interface.
src/​ipc/​ipc4/​helper.c	Adds pipeline lifecycle notifications.
src/​include/​sof/​debug/​telemetry/​telemetry.h	Defines TEL2 layouts and collection interfaces.
src/​include/​sof/​debug/​telemetry/​performance_monitor.h	Defines PMW2 partitions and module records.
src/​debug/​telemetry/​telemetry.c	Implements collection, publication, and revised tick accounting.
src/​debug/​telemetry/​performance_monitor.c	Implements PMW2 resource tables and I/O storage.
src/​debug/​telemetry/​Kconfig	Adds telemetry feature and capacity settings.
src/​audio/​pipeline/​pipeline-xrun.c	Records pipeline and component XRUN events.
src/​audio/​pipeline/​pipeline-stream.c	Notifies telemetry when triggers are submitted.
src/​audio/​module_adapter/​module/​generic.c	Publishes module resource usage after allocations.
src/​audio/​host-zephyr.c	Adds host gateway registration and transfer updates.
src/​audio/​dai-zephyr.c	Adds DAI gateway registration and transfer updates.
src/​audio/​component.c	Updates module resources and CPC violation counters.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Critical issues include privileged spinlocks that can fault userspace IPC, telemetry scans that can stall audio processing, and failed initialization that can lead to scheduling an uninitialized work item.

10 open issues

• Avoid privileged spinlocks in the userspace LL path · New
• Collect thread statistics outside the telemetry lock · New
• Do not schedule background work after init failure · New
• Synchronize LL state updates with background resets · New
• Refresh gateway snapshots after state transitions · New
• Identify absent threads before evicting history · New
• Use the timing counter’s rate for overrun budgets · New
• Use scheduler task accounting for idle detection · New
• Increment the allocation-failure count for L3 · New
• Update SRAM high-water mark on allocation · New

17 resolved since last review

• Protect vregion lifetimes and mutable statistics during collection
• Protect telemetry slot collection from concurrent destruction
• Protect IPC component traversal against concurrent deletion
• Bound telemetry entries to the actual memory partition capacity
• Report actual free SRAM capacity, not powered-on capacity
• Preserve the full 32-bit component token in telemetry
• Decouple telemetry publication from core 0 audio ticks
• Reclaim thread history entries for departed threads
• Publish host skips from the no-bytes path
• Count DAI XRUNs when the error is detected
• Clear the gateway active bit before the early return
• Connect the deadline-miss helper to DP scheduler accounting
• Map pipeline IDs to stable tracked slots
• Serialize cold-section writers to prevent partial snapshots
• Synchronize cross-core staging buffers before publication
• Publish pipeline state after transitions and refresh after the last callback
• Refresh PMW3 heap usage after module frees

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

An unguarded adapter call causes a link failure when module adapters are disabled, while unsynchronized ChainDMA rollback can let telemetry dereference a freed component.

9 open issues

• Guard module adapter telemetry when the adapter is disabled · New
• Unlink ChainDMA entries under lock before freeing components · New
• Clear the performance handle before freeing its slot · New
• Update the directory sequence when publishing cold data · New
• Sample DMA buffer occupancy instead of pipeline buffer · New
• Count skipped events only when a no-bytes event occurs · New
• Account for task deadline misses when a task completes · New
• Preserve the allocator high-water mark across telemetry refreshes · New
• Clamp exported lifetime usage to its partition size · New

10 resolved since last review

• Do not schedule background work after init failure
• Collect thread statistics outside the telemetry lock
• Avoid privileged spinlocks in the userspace LL path
• Update SRAM high-water mark on allocation
• Increment the allocation-failure count for L3
• Use scheduler task accounting for idle detection
• Use the timing counter’s rate for overrun budgets
• Identify absent threads before evicting history
• Refresh gateway snapshots after state transitions
• Synchronize LL state updates with background resets

_{Review effort: Balanced}

[lgirdwood]

Lets split this int a patch per susbsystem

[lgirdwood]

Split the PR into a 9-patch series organized by subsystem (updated to remove all telemetry spinlocks/mutexes for userspace compatibility and replace magic heap index constants):

1. 93ba65395 telemetry: headers: add extended TEL2 and PMW2 window definitions
2. fc11e699b zephyr: lib: add heap, VMH, and fast-get memory telemetry
3. 65bcac867 zephyr: vregion: add vpage and virtual region telemetry
4. d8d8ca7a6 telemetry: implement extended TEL2 and PMW2 telemetry windows
5. e5ebe5d96 ipc: unlink freed components/pipelines first and notify telemetry
6. 493d841a8 schedule: report active LL task counts and DP deadline misses to telemetry
7. 50608e37b pipeline: notify telemetry on pipeline state changes and XRUNs
8. f72e6b724 audio: module_adapter: update per-module telemetry resource metrics
9. 5cdfda192 audio: dai/host: wire DAI and Host DMA gateway telemetry

[kv2019i]

Looks very useful! Only bigger one is this adds spinlocks to code paths that are running in user-space in SOF mainline. This isn't caught as CONFIG_SOF_TELEMETRY is disable in user builds. Is this intentional that this is only support in kernel builds?

[kv2019i]

This '1' is magic value linked to shared-heap in multiple places in this source file. Better to use a file local define of some sorts to increase readability.

[lgirdwood]

Fixed in fc11e699b — added file-local TEL_HEAP_SYS_IDX, TEL_HEAP_SHARED_IDX, TEL_HEAP_LL_USER_IDX, TEL_HEAP_L3_IMR_IDX, and TEL_HEAP_OTHER_IDX defines in zephyr/lib/alloc.c and replaced all magic heap array indices.

[kv2019i]

This code can be run in userspace, so using spinlocks here is a problem.

[lgirdwood]

Lets remove the spinlocks since this is read only data that is not used internally, they are probably not needed.

[lgirdwood]

Fixed across the series (fc11e699b, 65bcac867, d8d8ca7a6, e5ebe5d96) — removed telemetry_ipc_lock() / telemetry_ipc_unlock() and all added spinlocks and mutexes (tel_lock, tel_cold_mutex, tel_vregion_lock, vpage_tel_lock, ipc->lock, h->lock, fast_get_data.lock, and vr->interim.heap.lock). Since telemetry windows export read-only diagnostic snapshots to the host and are not consumed internally by firmware control logic, unlinking icd->list / ipc_pipe->list and clearing icd->cd / ipc_pipe->pipeline prior to freeing (together with the TEL2 sequence counter) is sufficient and safe in both kernel and userspace (CONFIG_SOF_USERSPACE_LL) contexts.

[intel-sofci]

PR 11407: test results

Run date: 2026-11-10 14:51 UTC

Tested commit: e442b4a1e09a618e57e1db63b23981f365d973ed