Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
223 changes: 194 additions & 29 deletions .github/scripts/print-crash-reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@

- macOS writes a crash report (.ips) for every crashed process, naming the
library and function that faulted; this prints the newest few for the given
process names.
process names: the faulting thread, then every other thread's top frames
(task #72: the teardown race's other half is the main thread, in
`freeScript` or `exit`). The workflows also upload the raw reports.
- Linux keeps nothing unless the job asked for core dumps
(`.github/scripts/enable-core-dumps.sh`: `kernel.core_pattern` piped to a
handler that writes into `$VAPOURBOX_CORE_DIR`, default /tmp/cores). This
Expand All @@ -15,9 +17,21 @@
frame can still be resolved later (task #66).
- Windows: nothing to read; says so.

Each crash is then classified: the first frame outside the system libraries
names the image that faulted, and one signature is recognised by name,
vspipe R78's teardown race (task #68): that frame is vspipe's own
`frameDoneCallback` or `outputFrame`. On macOS it arrives as SIGILL
(`pthread_cond_signal` on the freed condition variable traps in
`_os_unfair_lock_unowned_abort`), on Linux as SIGSEGV (task #72). The line
also says whether the installed bundle's vspipe carries the fix (its
`--version` marker, read from the binary).

Never fails the step: it is diagnostic.

Usage: print-crash-reports.py [process-name ...] (default: vspipe-bin python3)
Usage:
print-crash-reports.py [process-name ...] (default: vspipe-bin python3 python3.12)
print-crash-reports.py --classify FILE ... one line per file: FILE<TAB>verdict
(.ips, or gdb backtrace text)
"""
import glob
import json
Expand All @@ -27,42 +41,171 @@
import subprocess
import sys

NAMES = sys.argv[1:] or ["vspipe-bin", "python3", "python3.12"]
DEFAULT_NAMES = ["vspipe-bin", "python3", "python3.12"]
NEWEST = 3

TEARDOWN = "vspipe-teardown"
# The marker task #68's patch adds to `vspipe --version`; it is a literal in
# the binary, so it can be found without running vspipe (which needs the
# bundle's environment).
FIX_MARKER = b"VapourBox patch: vspipe frame-done count"
VSPIPE_IMAGES = {"vspipe", "vspipe-bin", "vspipe-bin.real", "vspipe.exe"}
VSPIPE_CALLBACKS = ("frameDoneCallback", "outputFrame")
TEARDOWN_MAIN = ("freeScript", "vpy4_freeScript", "exit", "__cxa_finalize")
_SYSTEM = re.compile(
r"^(libsystem_\w+\.dylib|libc\+\+(abi)?\.1\.dylib|libdyld\.dylib|dyld"
r"|libc\.so\.\d+|libpthread\.so\.\d+|libstdc\+\+\.so\.\d+|libgcc_s\.so\.\d+"
r"|ld-linux[-\w]*\.so\.\d+|\?\?)$"
)


def is_system(image):
return bool(_SYSTEM.match(image or "??"))


def classify(frames, other_threads=()):
"""frames: the faulting thread, top first, as (image basename, symbol).
other_threads: the other threads' frames, the same way. Returns
(verdict, detail): verdict is TEARDOWN for task #68's race, else
"image:<name>" for the first frame outside the system libraries, else
"unknown"."""
first = next(((img, sym) for img, sym in frames if not is_system(img)), None)
if first is None:
return "unknown", "every frame is in a system library"
img, sym = first
if img in VSPIPE_IMAGES and any(cb in sym for cb in VSPIPE_CALLBACKS):
main = next((s for t in other_threads for _, s in t[:12]
if any(m in s for m in TEARDOWN_MAIN)), None)
detail = f"{img} {sym.split('(')[0]}"
if main:
detail += f"; another thread is in {main.split('(')[0]}"
return TEARDOWN, detail
return f"image:{img}", sym


def vspipe_fix_state():
"""'yes' / 'no' for the installed bundle's vspipe, or None if not found."""
deps = os.environ.get("VAPOURBOX_DEPS_DIR", "")
if not deps:
return None
for pattern in ("vapoursynth/vspipe-bin.real", "vapoursynth/vspipe-bin",
"vapoursynth/Lib/site-packages/vapoursynth/vspipe.exe"):
path = os.path.join(deps, pattern)
if os.path.isfile(path):
with open(path, "rb") as f:
return "yes" if FIX_MARKER in f.read() else "no"
return None


def verdict_line(verdict, detail):
if verdict == TEARDOWN:
fix = vspipe_fix_state()
has = {"yes": "the installed vspipe carries the fix, so this is NOT that race being unfixed: look again",
"no": "the installed vspipe does not carry the fix (deps before 1.18.0)",
None: "could not check the installed vspipe for the fix"}[fix]
return (f" => vspipe R78's teardown race (task #68): {detail}. "
f"Not a plugin or an instruction-set fault; {has}.")
if verdict.startswith("image:"):
return f" => faulted in {verdict[6:]}: {detail}"
return f" => {detail}"


# --- macOS -----------------------------------------------------------------

def ips_threads(path):
"""(exception summary, faulting thread frames, other threads' frames)."""
with open(path, encoding="utf-8", errors="replace") as f:
f.readline() # one-line JSON header
body = json.loads(f.read())
exc = body.get("exception", {})
summary = f"{exc.get('type')} {exc.get('signal')} {exc.get('subtype', '')}".rstrip()
images = body.get("usedImages", [])

def frames_of(thread):
out = []
for frame in thread.get("frames", []):
idx = frame.get("imageIndex", -1)
image = images[idx] if 0 <= idx < len(images) else {}
lib = os.path.basename(image.get("path", image.get("name", "??")) or "??")
sym = frame.get("symbol", f"+{frame.get('imageOffset', 0):#x}")
out.append((lib, sym))
return out

threads = body.get("threads", [])
faulting = body.get("faultingThread", 0)
fault = frames_of(threads[faulting]) if faulting < len(threads) else []
others = [(i, frames_of(t)) for i, t in enumerate(threads) if i != faulting]
return summary, fault, others


def print_ips(path):
print(f"=== {os.path.basename(path)}")
try:
summary, fault, others = ips_threads(path)
except (OSError, ValueError) as e:
print(f" (unparseable: {e})")
return
print(f" {summary}")
print(" --- faulting thread")
for i, (lib, sym) in enumerate(fault[:30]):
print(f" {i:2} {lib:32} {sym}")
for n, frames in others:
print(f" --- thread {n}")
for i, (lib, sym) in enumerate(frames[:8]):
print(f" {i:2} {lib:32} {sym}")
print(verdict_line(*classify(fault, [f for _, f in others])))

def macos():

def macos(names):
dirs = [os.path.expanduser("~/Library/Logs/DiagnosticReports"), "/Library/Logs/DiagnosticReports"]
reports = sorted(
(p for d in dirs for p in glob.glob(os.path.join(d, "*.ips"))
if any(os.path.basename(p).startswith(n) for n in NAMES)),
if any(os.path.basename(p).startswith(n) for n in names)),
key=os.path.getmtime,
reverse=True,
)[:NEWEST]

if not reports:
print(f"No crash reports for {', '.join(NAMES)}.")
print(f"No crash reports for {', '.join(names)}.")
for path in reports:
print(f"=== {os.path.basename(path)}")
with open(path, encoding="utf-8", errors="replace") as f:
f.readline() # one-line JSON header
try:
body = json.loads(f.read())
except json.JSONDecodeError as e:
print(f" (unparseable: {e})")
continue
exc = body.get("exception", {})
print(f" {exc.get('type')} {exc.get('signal')} {exc.get('subtype', '')}".rstrip())
images = body.get("usedImages", [])
threads = body.get("threads", [])
faulting = body.get("faultingThread", 0)
if faulting >= len(threads):
continue
for i, frame in enumerate(threads[faulting].get("frames", [])[:30]):
image = images[frame.get("imageIndex", -1)] if 0 <= frame.get("imageIndex", -1) < len(images) else {}
lib = os.path.basename(image.get("path", image.get("name", "?")))
sym = frame.get("symbol", f"+{frame.get('imageOffset', 0):#x}")
print(f" {i:2} {lib:32} {sym}")
print_ips(path)


# --- Linux -----------------------------------------------------------------

_GDB_FRAME = re.compile(r"^\s*#\d+\s+(?:0x[0-9a-fA-F]+ in )?(.*?)(?: from (\S+))?\s*$")


def gdb_threads(text, exe="vspipe-bin"):
"""(faulting thread frames, other threads' frames) from gdb's output.
The faulting thread is the block after "--- faulting thread" if this
script printed it, else the first backtrace in the text."""
def parse(block):
out = []
for line in block:
m = _GDB_FRAME.match(line)
if m:
sym, lib = m.group(1), m.group(2)
out.append((os.path.basename(lib) if lib else exe, sym))
return out

lines = text.splitlines()
blocks, cur, start = [], [], 0 # (index of the block's first line, its lines)
for i, line in enumerate(lines):
if _GDB_FRAME.match(line) and line.strip().startswith("#"):
if not cur:
start = i
cur.append(line)
elif cur:
blocks.append((start, cur))
cur = []
if cur:
blocks.append((start, cur))
marker = next((i for i, l in enumerate(lines) if "--- faulting thread" in l), -1)
chosen = next((b for s, b in blocks if s > marker), None)
fault = parse(chosen) if chosen else []
others = [parse(b) for _, b in blocks if b is not chosen]
return fault, others


def core_executable(core):
Expand Down Expand Up @@ -124,12 +267,34 @@ def linux():
print(f" executable: {exe}")
for line in lines:
print(f" {line}")
fault, others = gdb_threads(text, os.path.basename(exe))
print(verdict_line(*classify(fault, others)))


# --- command line ----------------------------------------------------------

def main():
def classify_file(path):
if path.endswith(".ips"):
_, fault, others = ips_threads(path)
return classify(fault, [f for _, f in others])
with open(path, encoding="utf-8", errors="replace") as f:
fault, others = gdb_threads(f.read())
return classify(fault, others)


def main(argv):
if argv and argv[0] == "--classify":
for path in argv[1:]:
try:
verdict, detail = classify_file(path)
except (OSError, ValueError) as e:
verdict, detail = "unreadable", str(e)
print(f"{path}\t{verdict}\t{detail}")
return
names = argv or DEFAULT_NAMES
try:
if sys.platform == "darwin":
macos()
macos(names)
elif sys.platform.startswith("linux"):
linux()
else:
Expand All @@ -139,4 +304,4 @@ def main():


if __name__ == "__main__":
main()
main(sys.argv[1:])
10 changes: 7 additions & 3 deletions .github/scripts/replay-vspipe.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ if [ -n "\$script" ] && grep -Eq '$MATCH' "\$script" 2>/dev/null; then
d=\$(mktemp -d "$CAP/run.XXXXXX")
cp "\$script" "\$d/script.vpy"
printf '%s\0' "\$@" > "\$d/args"
env -0 > "\$d/env"
# BSD env (macOS 15) may lack -0; perl is on every runner.
env -0 > "\$d/env" 2>/dev/null || perl -e 'print "\$_=\$ENV{\$_}\0" for keys %ENV' > "\$d/env"
cat > "\$d/stdin.raw"
"\$REAL" "\$@" < "\$d/stdin.raw"
rc=\$?
Expand Down Expand Up @@ -73,15 +74,17 @@ run)
case "$a" in *.vpy) args+=("$script") ;; *) args+=("$a") ;; esac
done < "$r/args"
(
mode="$RV_MODE" vspipe="$RV_VSPIPE"
mode="$RV_MODE" vspipe="$RV_VSPIPE" RV_TIMEOUT="$RV_TIMEOUT"
while IFS= read -r -d '' kv; do export "$kv" 2>/dev/null; done < "$r/env"
pre=()
if [ "$mode" = valgrind ]; then
# The report goes to stderr, kept in the err file whatever the exit.
export PYTHONMALLOC=malloc
pre=(valgrind --error-exitcode=99 --track-origins=yes --num-callers=40 --error-limit=no)
fi
timeout 3600 "${pre[@]}" "$vspipe" "${args[@]}" < "$r/stdin.raw" > /dev/null 2> "$err"
# GNU timeout where there is one (macOS 15 has none built in).
tmo=(); [ -n "$RV_TIMEOUT" ] && tmo=("$RV_TIMEOUT" 3600)
"${tmo[@]}" "${pre[@]}" "$vspipe" "${args[@]}" < "$r/stdin.raw" > /dev/null 2> "$err"
)
rc=$?
printf '%s\t%s\t%s\n' "$i" "$(basename "$r")" "$rc" >> "$RV_OUT/results.tsv"
Expand All @@ -90,6 +93,7 @@ run)
}
export -f one
export RV_MODE="$MODE" RV_VSPIPE="$VSPIPE" RV_OUT="$OUT"
RV_TIMEOUT="$(command -v timeout || command -v gtimeout || true)"; export RV_TIMEOUT
: > "$OUT/results.tsv"
start=$(date +%s)
for i in $(seq 1 "$REPEAT"); do for r in "${runs[@]}"; do printf '%s\0%s\0' "$i" "$r"; done; done \
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/ci-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,26 @@ jobs:
if: failure()
run: python3 .github/scripts/print-crash-reports.py

- name: Keep the raw crash reports (on failure)
# The .ips files themselves, for symbolication or a second look: the
# step above prints only their top frames (task #72).
if: failure()
run: |
mkdir -p crash-reports
for f in ~/Library/Logs/DiagnosticReports/{vspipe-bin,python3,vapourbox-worker}*.ips; do
[ -f "$f" ] && cp "$f" crash-reports/
done
ls crash-reports

- name: Upload the raw crash reports (on failure)
if: failure()
uses: actions/upload-artifact@v5
with:
name: crash-reports-macos-${{ matrix.arch }}
path: crash-reports/*.ips
if-no-files-found: ignore
retention-days: 14

windows:
runs-on: windows-latest
env:
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,26 @@ jobs:
if: failure()
run: python3 .github/scripts/print-crash-reports.py

- name: Keep the raw crash reports (on failure)
# The .ips files themselves, for symbolication or a second look: the
# step above prints only their top frames (task #72).
if: failure()
run: |
mkdir -p crash-reports
for f in ~/Library/Logs/DiagnosticReports/{vspipe-bin,python3,vapourbox-worker}*.ips; do
[ -f "$f" ] && cp "$f" crash-reports/
done
ls crash-reports

- name: Upload the raw crash reports (on failure)
if: failure()
uses: actions/upload-artifact@v5
with:
name: crash-reports-${{ matrix.asset }}-${{ matrix.suite }}
path: crash-reports/*.ips
if-no-files-found: ignore
retention-days: 14

windows:
if: ${{ github.event_name == 'schedule' || inputs.platform == 'all' || inputs.platform == 'windows' }}
# Both CPU-tier bundles (issue #92); see the macOS job for what the v2 run proves.
Expand Down
8 changes: 8 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3615,6 +3615,14 @@ Rules:
in vspipe's own `frameDoneCallback` / `outputFrame` and main in
`freeScript` is vspipe R78's teardown race** (task #68, fixed in R80), not
the script: any one-frame preview can hit it, roughly 1 in 1,000.
**On macOS the same race is a SIGILL** (task #72): `pthread_cond_signal`
from vspipe's `frameDoneCallback` on the condition variable main has
freed traps in `_os_unfair_lock_unowned_abort`. It is not an instruction
the CPU lacks. The crash-report step prints every thread, names this
signature and says whether the bundle's vspipe has the fix
(`print-crash-reports.py --classify FILE…` does it for any `.ips` or
gdb text); the macOS jobs upload the raw `.ips` as
`crash-reports-*` artifacts.
**A memcheck "uninitialised" report at more than 8 bits and an odd
plane width is expected and harmless on its own** (task #70):
VapourSynth's AVX2 kernels read stride padding into lanes they discard,
Expand Down
19 changes: 19 additions & 0 deletions Tests/TestResources/crash_reports/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Crash-report fixtures (task #72)

Inputs for `app/test/crash_report_classifier_test.dart`, which runs
`.github/scripts/print-crash-reports.py --classify` over them.

- `macos-x64-vspipe-teardown-sigill.ips`: the faulting thread of
`vspipe-bin-2026-11-18-023804.ips` from the push gate's macOS x64 job,
run 44088351262 attempt 2 (deps 1.17.0-rc3), rebuilt as an `.ips` from the
frames CI printed. CI printed only the faulting thread, so it has only that
thread; the image paths are placeholders. vspipe R78's teardown race (task
#68) as macOS reports it: SIGILL in `_os_unfair_lock_unowned_abort`.
- `macos-arm64-moltenvk-sigsegv.ips`: run 44064242986's macOS arm64
`vspipe-bin-2026-11-17-103557.ips` (RIFE creating its Vulkan device
through MoltenVK), rebuilt the same way, with some frames left out. A
crash that is not the race.
- `linux-vspipe-teardown-sigsegv.gdb.txt`: the gdb output CI printed for
`core.vspipe-bin.52015.1795010338`, run 44318045317's Linux job, from
"faulting thread" to the end of "every thread (top frames)". The same race
on Linux: SIGSEGV in `outputFrame`, main in `vpy4_freeScript`.
Loading
Loading