Skip to content

Task #72: macOS x64 sweep SIGILL is vspipe's teardown race; classify it in CI crash reports - #232

Merged
StuartCameronCode merged 5 commits into
mainfrom
stuart/72-macos-x64-ci-format-sweep-previews-die-w
Nov 19, 2026
Merged

StuartCameronCode merged 5 commits into
mainfrom
stuart/72-macos-x64-ci-format-sweep-previews-die-w

Conversation

@StuartCameronCode

Copy link
Copy Markdown
Owner

Summary

The macOS x64 push gate was failing random format-sweep rows with vspipe exited with signal 4 (SIGILL). That looked like a plugin using an instruction the runner lacks. It isn't: every crash report shows the same stack, entirely vspipe's own code and the system's:

frameDoneCallback → condition_variable::notify_one → pthread_cond_signal → _os_unfair_lock_unowned_abort (ud2)

This is vspipe R78's teardown race (task #68). The last frame is counted as done before its callback finishes, so main frees the condition variable underneath it. On macOS, signalling a freed condvar traps as SIGILL; on Linux the same race is a SIGSEGV in outputFrame. The fix (upstream 4ff3f8a0) has been in deps 1.18.0-rc1 since task #68. Every failing run used 1.17.0-rc3. No retry or quarantine is added: native crashes in the sweep still catch real bugs.

Changes

  • .github/scripts/print-crash-reports.py
    • Prints every thread of a macOS .ips report.
    • Classifies a crash as vspipe-teardown or by the faulting image (--classify FILE…, for .ips or gdb text).
    • Says whether the bundle's vspipe carries the fix.
  • ci-test.yml / nightly.yml: macOS jobs upload the raw .ips reports as crash-reports-* artifacts.
  • app/test/crash_report_classifier_test.dart (push gate, every platform), with fixtures from the real CI crashes: the macOS SIGILL and the Linux SIGSEGV are both classified as the race; a MoltenVK crash is named by its image.
  • replay-vspipe.sh: macOS portability (BSD env without -0; no timeout).
  • Docs: CLAUDE.md Debugging Tip REQUEST: Frame-by-frame navigation using the arrow keys #11 (the macOS form of the race), a new report docs/reports/task-72-macos-x64-sigill/, an ENGINEERING_NOTES entry, and a dated "measured in task Name hallucination? #72" note in task MacOS unable to open VIDEO_TS - 0.9.12 #68's report.
  • Temporary A/B workflow: it ran once (run 44500126463) and now lives only in the report's scripts/ folder, not in .github/workflows/.

Testing

A/B of the fix on x86 (run 44500126463). Same deps 1.17.0-rc3 bundle on both sides; only vspipe-bin differs:

Platform R78 vspipe Patched vspipe
Linux x64 66 / 10,720 crashes 0 / 10,720
macOS x64 2 / 10,720 (both classified as the race) 0 / 10,720
Windows x64 3 / 10,720 0 / 10,720
Total 71 / 32,160 0 / 32,160
  • Replaying the captured format-sweep runs was inconclusive. The same OpenCL-dependent runs failed on both sides, and no replay hit the race. The report says so.
  • Push gate since deps 1.18.0-rc1: 8 macOS x64 jobs, 4,704 sweep rows, 0 SIGILL (about 2 expected at the old rate). The two red jobs among them were task Fix conditional visibility, panel grouping, and the settings they silently dropped #75 timeouts.
  • Rate on deps 1.17.0-rc3: 4 of 20 runs failed, with 5 SIGILL rows. An earlier note said 5 runs and 6 reports; the report shows that corrected figure.
  • ci-test on this branch: run 44532277874 passed on all four jobs (Windows, Linux, macOS arm64, macOS x64).
  • crash_report_classifier_test.dart passes locally.

🤖 Generated with Claude Code

@StuartCameronCode
StuartCameronCode force-pushed the stuart/72-macos-x64-ci-format-sweep-previews-die-w branch from 1bf4a22 to ae97e02 Compare November 19, 2026 20:42
StuartCameronCode and others added 5 commits November 20, 2026 08:14
…raw .ips

print-crash-reports.py prints every thread of a macOS crash report (the
race's other half is main, freeing the script), classifies each crash by the
first frame outside the system libraries, and names task #68's teardown race
(vspipe's frameDoneCallback/outputFrame) together with whether the installed
vspipe carries the fix. --classify FILE... prints one verdict per .ips or gdb
text. The macOS jobs of ci-test and nightly upload the raw .ips on failure.

crash_report_classifier_test.dart (push gate) checks it on the crashes CI
printed: the macOS x64 SIGILL (os_unfair_lock abort under notify_one), the
Linux SIGSEGV (outputFrame, main in vpy4_freeScript), and a MoltenVK crash
that is not the race.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mat-sweep replays)

repro-vspipe-teardown.yml runs task #68's compare_vspipe.sh on macOS x64,
Linux x64 and Windows x64, and replays the push gate's captured format sweep
with R78's vspipe and the patched one (ab_sweep_replay.sh), both in the
1.17.0-rc3 bundle. replay-vspipe.sh no longer needs GNU timeout or env -0,
which macOS 15 may lack. Delete the workflow before merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…classifier and its data

Debugging Tip #11 names the SIGILL signature (pthread_cond_signal on a freed
condition variable, from vspipe's frameDoneCallback). classify_ci_failures.py
re-collects the push gate's red jobs from their logs; ci-failures.tsv is its
output for 2026-11-17..18.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…own race, fixed by deps 1.18.0

x86 A/B (run 44500126463), only vspipe-bin swapped in a 1.17.0-rc3 bundle:
R78 71/32,160 one-frame runs crashed, the patched vspipe 0/32,160 (Linux
66 vs 0, macOS 2 vs 0, Windows 3 vs 0). Format-sweep replays reproduced
nothing on either side. The temporary workflow moves into the report's
scripts/. Counts corrected against data/ci-failures.tsv: 4 runs, 5 rows on
rc3. Task #68's report gets a dated note with the measurement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@StuartCameronCode
StuartCameronCode force-pushed the stuart/72-macos-x64-ci-format-sweep-previews-die-w branch from ae97e02 to 83a455b Compare November 19, 2026 21:14
@StuartCameronCode
StuartCameronCode merged commit 3d17e23 into main Nov 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant