Repository navigation
Trace this Mac live - #52
Merged
Merged
Conversation
A live trace starts after dyld named some of the strings its events refer to, and an unknown string id raised a KeyError that ended the trace. Show them empty, as the image path already does.
The calls a thread didn't end when it terminated never will, and each kept every later event of its thread.
… flat Every event of a thread is kept for each of its ongoing calls, and a call that never ends (one whose end the kernel dropped, or one that never returns such as a run loop run) grows without bound on a long live trace. The live mode forgets a call once it kept MAX_ONGOING_EVENTS events, and when the kernel drops events, the calls ongoing since before the previous gap. It uses separate methods, so other traces run the same code as before.
… mode dyld names a string for each image it maps, and a trace with dyld events kept every string and image of every process it ever saw. In live mode, they are forgotten when their process exits (or execs), and so are the strings a short while after they are released, and the threads when they terminate. The images are kept by pid, so a process forgets its own at once.
Add the `live` command, and HostTrace for Python: they drive the kernel trace facility through the kdebug sysctl, which only requires root (no entitlement), and present it as a version 2 trace, named by the kernel's threadmap. Class and subclass filters are applied by the kernel's typefilter, so the kernel drops the unwanted events, and tracing is always disabled and removed on exit (Ctrl-C, SIGTERM, errors).
Add the syscall groups of pymobiledevice3's parse-live (`SYSCALL_GROUPS`, with their Mach traps from `group_mach_traps`), `event_eventids` to select events by trace code name and `syscall_names` to list the names `syscall_eventids` accepts, so both tools share them.
When syscalls are selected by event id, DBG_FSYSTEM is only traced for the paths of the selected calls, yet every lookup of every syscall was fed to the traces parser and then dropped as a trace of its own. Drop a DBG_FSYSTEM kevent right after decoding when its thread has no ongoing call: it can't be part of one, and an end event of such a thread has no start to match anyway. The output is the same; selecting a rare syscall under a busy `find` parses 2.1M -> 3.1M events/s.
…device3's `parse-live` traces this Mac with pymobiledevice3's parse-live options: `--bsc`, `-s`/`--syscall`, `-e`/`--event`, the syscall groups (`--fs`, `--net`, ...), `--dyld`, `--json` and `--color-mode`. The kernel filters by subclass at best, so it traces the subclasses of the selected syscalls and events, and the rest are dropped right after decoding. `syscalls` and `events` list the names they accept. To keep up, the output is written in batches and flushed whenever the trace waits for events, and the reads whose newest event is more than `--max-lag` (2s) old are skipped (`HostTrace(max_lag=...)`), or fail with `TraceLagError` with `--no-skip-behind`. On exit it prints whether it kept up: the events read, the kernel's drops, and the skipped events.
Falling more than --max-lag behind now fails parse-live, telling how to skip ahead instead (--skip-behind), allow more lag or narrow the trace, so a trace never silently misses events.
…s-scm Releasing no longer needs a version bump commit: creating the GitHub release (tag vX.Y.Z) builds that version. setuptools-scm writes pykdebugparser/_version.py at build time, as pymobiledevice3 does, and the publish workflow checks out the full history so the tag is found.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds live kdebug tracing of the Mac you run it on, decoded by pykdebugparser's handlers (which decode more than
fs_usage/ktracedo):How
host_trace.HostTracedrives the kernel trace facility through thekern.kdebugsysctl (KERN_KDSETBUF/KDSETUP/KDSET_TYPEFILTER/KDPIDEX/KDENABLE/KDREADTR/KDTHRMAP/KDREMOVE), the same waysc_usagedoes. It needs root but no entitlement, and no stackshot: thread->process names come fromKERN_KDTHRMAP. The MIB numbers and thekd_buf/kd_threadmap/kd_regtype/kbufinfo_tlayouts were checked against XNU'sbsd/sys/kdebug.h,kdebug_private.handbsd/kern/kdebug.c.PyKdebugParserparses them like any dump. Timestamps use the localmach_timebase_info,mach_absolute_timeand timezone.-cf/-sfgo to the kernel typefilter, so the kernel drops the unwanted events (VFS_LOOKUP is added for BSD syscalls, so their paths are named). The tracer's own pid is excluded.-c, and errors (including errors during setup).EBUSY(ktrace, fs_usage or Instruments is using it) andEPERMprint a short message.Commits
KeyErrorand ended the trace. Now it shows empty, as the image path already did.TRACE_DATA_THREAD_TERMINATE.TracesParser(..., live=True)/PyKdebugParser.live, off by default). A live host trace grew by hundreds of MB per minute: when the kernel drops events, a call whose end was dropped keeps every later event of its thread. The live mode forgets a call afterMAX_ONGOING_EVENTS(4096) events. It also forgets calls ongoing since before the previous gap whenever the kernel reports dropped events (the kernel'sTRACE_LOST_EVENTSmarks the point after which the stream has no gaps).{pid: {addr: path}}), so this cleanup is O(1).livecommand, tests with a fake sysctl, and the README.SYSCALL_GROUPS,group_mach_traps),event_eventidsandsyscall_names, so both tools can share them.--bsc,--fs,-s open, live and not); a rare syscall (-s connect) under a busyfindparses 2.1M -> 3.1M events/s.parse-live, with pymobiledevice3's parse-live options:--bsc,-s,-e, the groups (--fs,--net,--proc,--io,--mem,--signal,--cred,--sync,--mach),--dyld,--json,--color-mode, plussyscalls [--group]andeventsto list the names. The kernel filters by subclass at best, so the rest is dropped right after decoding. Output is written in batches and flushed when the trace waits for events.HostTracegainsmax_lag,skip_behind,on_skip,on_waitandTraceLagError. On exit it prints whether it kept up:Read N events, the kernel dropped events N times, N events were skipped to catch up(pymobiledevice3 prints the same line).--helpdescription.--max-lag(2s) behind fails by default, saying how to skip ahead instead (--skip-behind), allow more lag or narrow the trace.vX.Y.Z, with no bump commit. The publish workflow checks out the full history so the tag is found. A clone taggedv1.7.0buildspykdebugparser-1.7.0(wheel and sdist).No change for iOS (pymobiledevice3
parse-live)The live-mode logic is in separate methods and handlers that are only installed when
live=True. pymobiledevice3 doesn't set it, so it runs the same per-event code as before. Outside live mode, the only shared-path changes are the two handlers in commits 1-2 (they run only on rare events) and the per-piddyld_images(only on dyld image events).iPhone 11, iOS 27.2, USB,
developer dvt core-profile-session parse-live --bsc. Each run was 60 s, with system apps launched in a loop. Sessions ran one at a time.PYTHONPATH).uv run --with-editable.Traces/s follows how many apps got launched in the window, so traces per launch is shown too.
How far the last trace was behind the wall clock at stop was the same in every run (a constant offset between this phone's clock and the Mac's in the harness). Skip warnings show up in both base and branch runs (4 and 3). An offline replay of a recorded 750 MB iPhone trace (
--bscfilters, alternating runs) gives the same CPU time within noise: base 5.74-6.65 s, branch 5.64-6.36 s.Host keep-up (M-series Mac, 30 s each,
find / -xdevrunning in the background)-sf 0x40c(BSD syscalls)-cf 0x1f(dyld,lsspawned 20x/s), 90 sBefore commits 3-4 the unfiltered trace grew 56 -> 397 MB in 120 s, and the dyld one 44 -> 123 MB in 30 s.
The
findload makes ~200k syscalls/s, more than the decoder keeps up with, so the kernel drops events there and gaps are marked withTRACE_LOST_EVENTS. Narrower filters keep up (0 gaps).ktrace infoshows the facility as found after every run ("Kernel buffer is not initialized", "Last configured by 'tailspind'").parse-livekeep-up (M-series Mac, 20 s each, output to /dev/null)--bsc--bsc --json--fs --net --proc-s openfindload (1.2M events/s),-s connectfindload,--netfindload,-s openUnder the
findload,findalone opens ~112k directories/s, which is more traces than the parser prints;live -sf 0x40cdrops the same way there. Offline,--bscwith rich colors parses ~386k events/s of a normal-use recording.Tests: 235 pass on 3.9, 3.14 and 3.15, and each commit passes on its own. CI's flake8 is clean on the changed files.