Repository navigation
fix(watchers): stop a chokidar watcher whose fs.watch fails instead of retrying forever - #818
JDProfresh wants to merge 2 commits into
Conversation
…f retrying forever chokidar 3 with `persistent: false` has no guard for an `fs.watch` that throws (EMFILE once the inotify instance limit is used up, ENOSPC once the watch limit is). It emits the error, then a TypeError, then retries on the parent directory, and ends up re-adding `/` forever. TeamWatcher only logged its `'error'` events, so one failed watch became an endless stream of stack traces. WorkflowRunWatcher already closed its watcher on error, but `close()` removes every listener, so the TypeError from the same pass became an unhandled rejection, one per 10 s poll, and the server exits after five of those in a minute. `closeWatcherOnError()` closes the watcher on its first error, keeps a no-op listener on it, and turns its `add()` into a no-op, since a callback chokidar still has in flight can otherwise re-add a path and reopen it. TeamWatcher logs one line per watcher and relies on its 30 s poll; WorkflowRunWatcher still drops the watcher and re-arms it on the next poll.
|
Thank you for this, and for the write-up. Tracing 174 GiB of log back to the missing The chokidar lines you cite read exactly as you describe. The new test fails 6 of 8 on master's watcher code. Taking out the One change I want, minor:
A nit: Next step: I will apply both on top when I merge, once CI is green, so nothing is needed from you. If you would rather make the re-arm change in this PR yourself, push it to this branch before then. |
chokidar 3 reports a one-off ENOENT race (a new file gone between its stat and its watch) through the same TypeError as a lasting EMFILE, so a watcher that stayed closed ended live team and task updates until a restart. Each poll tick now re-creates a stopped watcher, as WorkflowRunWatcher already does, and an outage warns once rather than once per attempt. Also imports getErrorMessage from the types barrel.
|
Thanks for reproducing the ENOENT case, I had only looked at the lasting failures. Both points are in
Taking out the re-arm, either |
|
Thanks for the follow-up in The re-arm works as we discussed: one warning per outage, the watcher comes back on the next tick, and the ENOENT-once case finds a new team without a poll. On master's watcher code, 15 of the 18 tests fail (the three healthy cases pass), and removing the Two small things:
Next step: I will add both on top when I merge, so you don't need to do anything. If you'd rather push them to this branch yourself, do it before then. |
When
fs.watchthrows, chokidar 3 withpersistent: falseretries on the parent directory until it reaches/, then retries/forever, andTeamWatcherlogs every pass. On my machine that came to about 112,000 log lines a second for four hours.Problem
On 2026-11-12 my Codeman (1.51.0, systemd user unit, WSL2 Ubuntu 22.04, Node 26.12.1) restarted while the per-user inotify instance limit (
fs.inotify.max_user_instances, 1024 here) was used up. Leftoverdbus-daemonprocesses on my machine held the instances, a separate problem on my side. From its first second the new process logged the same stack traces in a tight loop:That was about 112,000 lines and 12 MB a second, 174 GiB in four and a quarter hours. journald rate-limited it, but rsyslog wrote every line to
/var/log/syslog, which filled the drive holding the WSL disk image and took WSL down. The failing path climbed from~/.claude/teamsthrough~/.claude,~and/hometo/and stayed there.EMFILE from
fs.watch(or ENOSPC oncefs.inotify.max_user_watchesruns out) is an ordinary condition on Linux. It should cost a log line and the live updates, not the disk or the server.Cause
Two layers.
chokidar 3.6.0: the
!options.persistentbranch ofsetFsWatchListener(lib/nodefs-handler.js:155-160) callswatcher.close.bind(watcher)on theundefinedthatcreateFsWatchInstancereturns afterfs.watchthrew; the persistent branch has anif (!watcher) return;guard there. The TypeError lands in the catch of_addToNodeFs, which emits it and returns the path (:647), andadd()reads a returned path as "not there yet, watch the parent instead" (index.js:458). The parent of/is/, so it never ends. The only way out is a closed watcher. chokidar 4.0.0 added the missing guard.Codeman: only the three
persistent: falsewatchers can get there.TeamWatcher(team-watcher.ts:63and:85) only logs its'error'events, with the whole error object, so every pass prints two full stack traces per watcher.WorkflowRunWatcher(workflow-run-watcher.ts:860) already closes its watcher on error, which does end the loop. Butclose()callsremoveAllListeners()(index.js:507), and the TypeError from the same pass is emitted after that. With no'error'listener, EventEmitter throws it, and it reaches the process as an unhandled rejection. Each poll re-arms the watcher, so while the limit stays used up that is one unhandled rejection every 10 s, andindex.tsexits after five of them (MAX_CONSECUTIVE_ERRORS). This one only runs with the Ultracode panel or floating windows on.ImageWatcheruses chokidar's defaultpersistent: true, which has the guard: it logs one line per directory it could not watch and does not retry.What it does
src/utils/close-watcher-on-error.ts:closeWatcherOnError(watcher, onClosed)closes the watcher on its first'error', leaves a no-op'error'listener on it for the TypeError still to come, turns itsadd()into a no-op, and callsonClosedonce. Theadd()part is for callbacks chokidar already has in flight: when a watched file is replaced while another is deleted,_remove()callsadd()on the closed watcher,add()setsclosed = false, and the climb to/starts again, silently this time (index.js:410,:857).TeamWatcher: each watcher logs one line,[TeamWatcher] teams watcher stopped, relying on polling: <message>(ortasks). The 30 s poll that already runs carries on, so teams and tasks still update, just up to 30 s later than with a live watcher. It does not try to watch again until the next start.WorkflowRunWatcher: behaves as before (drop the watcher, the next poll re-arms it), now without the unhandled rejection.CLAUDE.md: one paragraph after "Process-tree walks are bounded", so the nextpersistent: falsewatcher goes through the helper too.I left chokidar at 3.6.0 to keep this to Codeman's own handlers. Moving to chokidar 4 would also end the loop, if you want that as its own change.
Tests
test/watcher-fs-watch-failure.test.tsdrives the realTeamWatcherandWorkflowRunWatcheragainst temp directories, withfs.watchstubbed to throw EMFILE or ENOSPC. Nothing else is mocked.fs.watchattempts after them, no unhandled rejection, and a team created afterwards is found by the next poll.fs.watchstarts failing with ENOSPC and a watched file is replaced, which stops the watcher. Then a second watched file is deleted and thefs.watchlistener chokidar registered for it is called by hand, the event the OS had queued, so its_remove()always runs after the stop. One warning, and the failing attempts stay at a handful and stop growing.fs.watchattempts on every poll while it keeps failing, and oncefs.watchworks again the next poll installs a live watcher that finds a new run without another poll.fs.watch: no console output and no unhandled rejection, and TeamWatcher's live watcher finds a new team before any poll.On master 6 of the 8 tests fail (the two negative cases pass): about 8,400 warnings in 0.6 s instead of 2, an unhandled rejection in each WorkflowRunWatcher failure case, and 3 warnings instead of 1 in the race case. I took each guard out in turn and every removal turns at least one test red: no close (the loop is back), no no-op listener (unhandled rejections), the close moved to
setImmediatewithout the listener (still an unhandled rejection), noadd()no-op (the race case, 2,000 failing watch attempts until its cap, 10 of 10 runs), and nodirWatchers.delete(no re-arm).Gate: typecheck, lint, format:check, check:frontend-syntax, check:browser-excludes and
npm testpass, except two files that fail the same way on master on this machine:test/agent-skill-local-model.test.ts(30 tests, every run) and "stops the download when the caller cancels" intest/local-model-ollama-client.test.ts(2 of 5 runs on master). I left outtest/docker-start-setup.test.ts, which cannot run here: on this WSL host a connect to a closed loopback port hangs instead of being refused.Verified
A throwaway server (
tsx src/index.ts web, its own HOME andCODEMAN_INSTANCE, loopback) with a preload that makesfs.watchthrow EMFILE, its output only counted through a pipe:fs.watchfails, first 10 sshowUltracodeAgentson,fs.watchfails only under~/.claude/projects, 75 s[FATAL] 5 consecutive unhandled errorsand exit 1 at 41 s