Skip to content

rig: record the rig-nodes host monitor (/rig/host) in every run's bag - #282

Merged
5 commits merged into
mainfrom
claude/rig-perf-node-7942b8
Oct 29, 2026
Merged

5 commits merged into
mainfrom
claude/rig-perf-node-7942b8

Conversation

@mbreiser

@mbreiser mbreiser commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The arena rig now runs the generic host monitor from rig-nodes and records the PC's "pressure" into every run's MCAP bag, so each run carries how loaded the PC was. The monitor publishes CPU per core (window mean + busiest 100 ms), P-/E-core class, sampler lateness, memory, page faults, disks, network, NVIDIA GPU (incl. NVENC) and the top processes, once a second on /rig/host.

The node itself, rig_nodes.host_monitor, and its messages, rig_interfaces/HostStats (+ DiskStats, NetStats, GpuStats, ProcStats), shipped in rig-nodes 0.3.0. They were prototyped on ww18; the prototype is on branch rig/host-monitor-port @ f5cf5a6, and the message spec is on docs/host-stats-request. This PR is only the arena half:

  • rig/nodes/bridge_node.py (+13): records /rig/host into the bag on arrival. It uses the typed HostStats when rig_interfaces has it, else the String JSON fallback.
  • rig/nodes/rig.py (+22): starts python -m rig_nodes.host_monitor --log-dir rig/runlogs --py-root rig/ by default. --py-root lets it name this workspace's nodes py:<script>.
    • With an older rig-nodes package it prints a note and starts without the monitor.
    • New flags: --no-host-monitor, --host-monitor-args.
  • rig/tests/test_ros2_poc.py (+81): starts the monitor the same way and adds six checks. It skips them with an info line when the installed rig_nodes has no monitor. The checks:
    • samples recorded at ≥ 0.8 per second;
    • fields equal the installed HostStats (when typed);
    • one entry per CPU, with peak ≥ mean;
    • graph nodes pinned by script;
    • the monitor's own CPU ≤ 2 % of one core;
    • /rig/host ≤ 5 % of the run's compressed payload.
  • rig/README.md: one row.

fictrac-bridge/bridge.py is untouched; nothing in rig/pixi.toml / pixi.lock changes.

Depends on the webDisplayTools re-pin to the rig-nodes 0.3.0 channel (and #274) for the monitor to actually start. Until then this PR is inert apart from the bridge subscription.

Tier

Next candidate (rig code; no web-tool change, no version bumps).

Release notes

none (rig-PC ROS graph only; no web tool changes)

Test plan

  • pixi run test (root): 72/72 files
  • cd rig && pixi run test on ww18 (Windows 11, Core Ultra 9 285K, 24 CPUs, RTX PRO 4500), model fly, no hardware:
    • with the installed rig_nodes 0.2.1: exit 0, host-monitor checks skipped with the info line
    • with the rig-nodes 0.3.0 source (release/0.3.0 @ 5e20dee) first on PYTHONPATH: exit 0, all 6 host-monitor checks pass. 24 samples / 25 s, 24 cores, py:bridge_node / py:fictrac_udp_adapter / py:host_monitor pinned, monitor CPU median 0.9 % of one core, /rig/host 4.8 % of the compressed payload (≈ 1 MB/h, String JSON)
  • Typed, on ww18 from candidate 037a5cc (contains rig: pin rig-nodes 0.3.0 + re-lock with pixi 0.83 (#280, LAB-338) #281) with this branch merged in, after pixi self-update to 0.83 and pixi install --locked -e typed. pixi run -e typed test: /rig/host recorded as rig_interfaces/msg/HostStats (24 samples / 25 s, fields = the installed HostStats, 24 cores, graph nodes pinned, monitor CPU median 0.9 % of one core). Every check passes except the storage gate: 6.7 % of the compressed payload (1.77 MB/h).
    • Typed is smaller raw (1.8 KB vs 4.6 KB per sample), but float32 noise compresses worse than the String form's 3-decimal text.
    • Re-serializing the same samples with the floats rounded to binary steps brings it to 4.1 % (1.06 MB/h): 0..1 fractions and cores to 1/1024, rates and MB to 1/64. Coarser steps (1/256, 1/16) give 3.1 %.
    • The fix belongs in the node, so it's a rig-nodes 0.3.1 item. Until it lands the gate fails in the typed environment only; the default environment (String JSON, what pixi run rig uses) passes at 4.8 %.
    • The harness decodes typed /rig/host with rclpy's deserializer (db51439). The converter's pure CDR decoder reads no nested message arrays, and the converter never needs this topic.
  • A real run: pixi run rig with FicTrac + the Studio, confirm /rig/host in the run's bag

🤖 Generated with Claude Code

mbreiser and others added 4 commits October 28, 2026 17:20
…every run's bag

A generic node (nothing FicTrac- or arena-specific) that publishes, once a second, CPU per
core (window mean + busiest 100 ms sub-window, P/E-core class, sampler lateness, context
switches / interrupts / DPCs), memory + page file + hard faults, log-volume free space,
per-disk and per-adapter rates, NVIDIA GPU (incl. NVENC/NVDEC) and the top processes plus
pinned groups (FicTrac, BIAS, MATLAB, browser, Foxglove, router, each graph node). The
method is atlas-bench tools/cpu_sampler.py: cycle counters and one NtQuerySystemInformation
call for all processes on Windows; GetIfEntry2 for network; NVML through ctypes; psutil
for the rest and on Linux/macOS.

- rig/nodes/host_sampler.py (no ROS) + host_monitor.py: rig-nodes material, prototyped here.
  Typed rig_interfaces/HostStats when installed (requested for rig-nodes 0.3.0; spec in
  rig/docs/host-stats-request.md), else String JSON with the same field names.
- bridge_node.py records /rig/host into the bag; rig.py starts the monitor by default
  (--no-host-monitor, --host-monitor-args).
- Cost on ww18: 0.9 % of one core (median); 4.9 % of the harness run's compressed payload,
  about 1.0 MB/h. Gated in the harness (<= 2 % of a core, <= 5 % of the payload).
- tests/test-host-sampler.py (root suite, no ROS/psutil): fields and types against the .msg
  spec, rates, CPU mean vs peak, process selection, strict JSON.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rig-nodes

The generic host monitor (host_sampler.py, host_monitor.py, its unit test, design note and
the HostStats message spec) belongs in reiserlab/rig-nodes as rig_nodes.host_monitor; it is
being ported there from branch rig/host-monitor-port (f5cf5a6). What stays here is the arena
rig's composition and gateway:

- bridge_node.py records /rig/host into each run's bag (typed HostStats when rig_interfaces
  has it, else String).
- rig.py starts `python -m rig_nodes.host_monitor --log-dir rig/runlogs --py-root rig/` when
  the installed rig_nodes ships it (a note otherwise); --no-host-monitor, --host-monitor-args.
- test_ros2_poc.py runs it the same way and checks samples, fields, per-core sanity, pinned
  graph nodes, its CPU (<= 2 % of a core) and storage (<= 5 % of the compressed payload);
  skipped with an info line when the installed rig_nodes has no host monitor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…7942b8

# Conflicts:
#	rig/nodes/rig.py
#	rig/tests/test_ros2_poc.py
…after the BIAS node, docstring)

Coordinated with the session building the candidate with #269 (stimulus node): its hunks edit
the --sim argparse neighbourhood, the --foxglove docstring line and the lines after
nodes.append(bridge). Ours now sit after --sim-args, after the BIAS node append and after the
--sim docstring example, so the two PRs merge without conflict. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… recorded schema

The converter's pure CDR decoder reads no nested message arrays (HostStats carries DiskStats[],
NetStats[], GpuStats[], ProcStats[]), and the converter never needs /rig/host, so the typed
check deserializes with ROS itself. The info line now says which type was recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mbreiser added a commit that referenced this pull request Oct 28, 2026
mbreiser added a commit that referenced this pull request Oct 28, 2026
Candidate for Production. PRs: #281, #269, #282, #284.

Release-Manifest: releases/2026-10-28.json
@mbreiser mbreiser closed this pull request by merging all changes into main in 59a377b Oct 29, 2026
@mbreiser
mbreiser deleted the claude/rig-perf-node-7942b8 branch October 29, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant