Skip to content

fix(watchers): stop a chokidar watcher whose fs.watch fails instead of retrying forever - #818

Open
JDProfresh wants to merge 2 commits into
Ark0N:masterfrom
JDProfresh:fix/watcher-emfile-loop
Open

JDProfresh wants to merge 2 commits into
Ark0N:masterfrom
JDProfresh:fix/watcher-emfile-loop

Conversation

@JDProfresh

Copy link
Copy Markdown
Contributor

When fs.watch throws, chokidar 3 with persistent: false retries on the parent directory until it reaches /, then retries / forever, and TeamWatcher logs every pass. On my machine that came to about 112,000 log lines a second for four hours.

Problem

On 2026-11-12 my Codeman (1.51.0, systemd user unit, WSL2 Ubuntu 22.04, Node 26.12.1) restarted while the per-user inotify instance limit (fs.inotify.max_user_instances, 1024 here) was used up. Leftover dbus-daemon processes on my machine held the instances, a separate problem on my side. From its first second the new process logged the same stack traces in a tight loop:

[TeamWatcher] chokidar teams watcher error: Error: EMFILE: too many open files, watch '/'
    at FSWatcher.<computed> (node:internal/fs/watchers:323:19)
    ...
[TeamWatcher] chokidar teams watcher error: TypeError: Cannot read properties of undefined (reading 'close')
    at setFsWatchListener (.../node_modules/chokidar/lib/nodefs-handler.js:159:20)
    ...
[TeamWatcher] chokidar tasks watcher error: Error: EMFILE: too many open files, watch '/'
    ...

That was about 112,000 lines and 12 MB a second, 174 GiB in four and a quarter hours. journald rate-limited it, but rsyslog wrote every line to /var/log/syslog, which filled the drive holding the WSL disk image and took WSL down. The failing path climbed from ~/.claude/teams through ~/.claude, ~ and /home to / and stayed there.

EMFILE from fs.watch (or ENOSPC once fs.inotify.max_user_watches runs out) is an ordinary condition on Linux. It should cost a log line and the live updates, not the disk or the server.

Cause

Two layers.

chokidar 3.6.0: the !options.persistent branch of setFsWatchListener (lib/nodefs-handler.js:155-160) calls watcher.close.bind(watcher) on the undefined that createFsWatchInstance returns after fs.watch threw; the persistent branch has an if (!watcher) return; guard there. The TypeError lands in the catch of _addToNodeFs, which emits it and returns the path (:647), and add() reads a returned path as "not there yet, watch the parent instead" (index.js:458). The parent of / is /, so it never ends. The only way out is a closed watcher. chokidar 4.0.0 added the missing guard.

Codeman: only the three persistent: false watchers can get there.

  • TeamWatcher (team-watcher.ts:63 and :85) only logs its 'error' events, with the whole error object, so every pass prints two full stack traces per watcher.
  • WorkflowRunWatcher (workflow-run-watcher.ts:860) already closes its watcher on error, which does end the loop. But close() calls removeAllListeners() (index.js:507), and the TypeError from the same pass is emitted after that. With no 'error' listener, EventEmitter throws it, and it reaches the process as an unhandled rejection. Each poll re-arms the watcher, so while the limit stays used up that is one unhandled rejection every 10 s, and index.ts exits after five of them (MAX_CONSECUTIVE_ERRORS). This one only runs with the Ultracode panel or floating windows on.

ImageWatcher uses chokidar's default persistent: true, which has the guard: it logs one line per directory it could not watch and does not retry.

What it does

  • New src/utils/close-watcher-on-error.ts: closeWatcherOnError(watcher, onClosed) closes the watcher on its first 'error', leaves a no-op 'error' listener on it for the TypeError still to come, turns its add() into a no-op, and calls onClosed once. The add() part is for callbacks chokidar already has in flight: when a watched file is replaced while another is deleted, _remove() calls add() on the closed watcher, add() sets closed = false, and the climb to / starts again, silently this time (index.js:410, :857).
  • TeamWatcher: each watcher logs one line, [TeamWatcher] teams watcher stopped, relying on polling: <message> (or tasks). The 30 s poll that already runs carries on, so teams and tasks still update, just up to 30 s later than with a live watcher. It does not try to watch again until the next start.
  • WorkflowRunWatcher: behaves as before (drop the watcher, the next poll re-arms it), now without the unhandled rejection.
  • CLAUDE.md: one paragraph after "Process-tree walks are bounded", so the next persistent: false watcher goes through the helper too.

I left chokidar at 3.6.0 to keep this to Codeman's own handlers. Moving to chokidar 4 would also end the loop, if you want that as its own change.

Tests

test/watcher-fs-watch-failure.test.ts drives the real TeamWatcher and WorkflowRunWatcher against temp directories, with fs.watch stubbed to throw EMFILE or ENOSPC. Nothing else is mocked.

  • TeamWatcher: exactly two warnings with the exact text, no fs.watch attempts after them, no unhandled rejection, and a team created afterwards is found by the next poll.
  • TeamWatcher, the in-flight race: a healthy watcher, then fs.watch starts failing with ENOSPC and a watched file is replaced, which stops the watcher. Then a second watched file is deleted and the fs.watch listener chokidar registered for it is called by hand, the event the OS had queued, so its _remove() always runs after the stop. One warning, and the failing attempts stay at a handful and stop growing.
  • WorkflowRunWatcher: the run is still discovered, no unhandled rejection, no console output, the same number of fs.watch attempts on every poll while it keeps failing, and once fs.watch works again the next poll installs a live watcher that finds a new run without another poll.
  • Negative cases with the real fs.watch: no console output and no unhandled rejection, and TeamWatcher's live watcher finds a new team before any poll.

On master 6 of the 8 tests fail (the two negative cases pass): about 8,400 warnings in 0.6 s instead of 2, an unhandled rejection in each WorkflowRunWatcher failure case, and 3 warnings instead of 1 in the race case. I took each guard out in turn and every removal turns at least one test red: no close (the loop is back), no no-op listener (unhandled rejections), the close moved to setImmediate without the listener (still an unhandled rejection), no add() no-op (the race case, 2,000 failing watch attempts until its cap, 10 of 10 runs), and no dirWatchers.delete (no re-arm).

Gate: typecheck, lint, format:check, check:frontend-syntax, check:browser-excludes and npm test pass, except two files that fail the same way on master on this machine: test/agent-skill-local-model.test.ts (30 tests, every run) and "stops the download when the caller cancels" in test/local-model-ollama-client.test.ts (2 of 5 runs on master). I left out test/docker-start-setup.test.ts, which cannot run here: on this WSL host a connect to a closed loopback port hangs instead of being refused.

Verified

A throwaway server (tsx src/index.ts web, its own HOME and CODEMAN_INSTANCE, loopback) with a preload that makes fs.watch throw EMFILE, its output only counted through a pipe:

Run master this branch
Every fs.watch fails, first 10 s 768,780 lines (59 MB), still looping when stopped 14 lines, two of them the TeamWatcher warnings
showUltracodeAgents on, fs.watch fails only under ~/.claude/projects, 75 s an unhandled rejection every 10 s, then [FATAL] 5 consecutive unhandled errors and exit 1 at 41 s 12 lines, no rejection, still running at 75 s

…f retrying forever

chokidar 3 with `persistent: false` has no guard for an `fs.watch` that throws (EMFILE once the inotify instance limit is used up, ENOSPC once the watch limit is). It emits the error, then a TypeError, then retries on the parent directory, and ends up re-adding `/` forever. TeamWatcher only logged its `'error'` events, so one failed watch became an endless stream of stack traces. WorkflowRunWatcher already closed its watcher on error, but `close()` removes every listener, so the TypeError from the same pass became an unhandled rejection, one per 10 s poll, and the server exits after five of those in a minute.

`closeWatcherOnError()` closes the watcher on its first error, keeps a no-op listener on it, and turns its `add()` into a no-op, since a callback chokidar still has in flight can otherwise re-add a path and reopen it. TeamWatcher logs one line per watcher and relies on its 30 s poll; WorkflowRunWatcher still drops the watcher and re-arms it on the next poll.
@Ark0N

Ark0N commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Thank you for this, and for the write-up. Tracing 174 GiB of log back to the missing if (!watcher) return; in chokidar 3's non-persistent branch made the review easy. This PR adds closeWatcherOnError(), routes the TeamWatcher and WorkflowRunWatcher 'error' handlers through it, and pins the behaviour with tests against the real watchers.

The chokidar lines you cite read exactly as you describe. The new test fails 6 of 8 on master's watcher code. Taking out the add() no-op or the no-op listener turns tests red, so both guards are load-bearing. Typecheck, lint, format and the frontend checks pass.

One change I want, minor:

  • src/team-watcher.ts:71 and :89: the helper closes on any chokidar error. chokidar 3 emits the same Cannot read properties of undefined (reading 'close') TypeError when fs.watch throws ENOENT because a new file or directory under ~/.claude/teams or ~/.claude/tasks vanished between its stat and its watch. For a directory, that window includes the readdir in _handleDir.

    • On master that cost one log line and the watcher kept working.
    • With this change the watcher closes and TeamWatcher never creates it again, so live team and task updates fall back to the 30 s poll until the next restart.

    I reproduced it by making fs.watch throw ENOENT once for a single new file. Master still picked up a team created afterwards without a poll; this branch did not. The fix is to re-arm from the poll, as WorkflowRunWatcher already does: set the field to null in onClosed, recreate a missing watcher from poll(), and log the stop once per outage rather than once per attempt. The assertion at test/watcher-fs-watch-failure.test.ts:144 then becomes a bounded number of attempts per poll, like the WorkflowRunWatcher test at :360-371.

A nit: src/team-watcher.ts:19 can import getErrorMessage from the ./types.js barrel the file already uses on line 18 (CLAUDE.md, import conventions).

Next step: I will apply both on top when I merge, once CI is green, so nothing is needed from you. If you would rather make the re-arm change in this PR yourself, push it to this branch before then.

chokidar 3 reports a one-off ENOENT race (a new file gone between its
stat and its watch) through the same TypeError as a lasting EMFILE, so a
watcher that stayed closed ended live team and task updates until a
restart. Each poll tick now re-creates a stopped watcher, as
WorkflowRunWatcher already does, and an outage warns once rather than
once per attempt. Also imports getErrorMessage from the types barrel.
@JDProfresh

Copy link
Copy Markdown
Contributor Author

Thanks for reproducing the ENOENT case, I had only looked at the lasting failures. Both points are in c7b6e1cb:

  • src/team-watcher.ts: the closed watcher's field goes back to null in onClosed, and each poll tick re-creates a watcher that is not running (setupFsWatchers() now runs at the top of the interval callback, split into one setup per watcher). A watcher that fails again in the same outage stays quiet: one warning per watcher until a re-created one is still running at the next tick, after which a new failure warns again. A synchronous throw from chokidar.watch() goes through the same once-per-outage warning instead of logging every tick.
  • getErrorMessage now comes from the ./types.js barrel.
  • test/watcher-fs-watch-failure.test.ts: the old "no new attempts after a poll" assertion is now an exact number of attempts per tick, flat across ticks, with the warning count held at two. New cases: your ENOENT-once repro (one warning, re-armed after one tick, a team created afterwards found without another tick), recovery for both watchers once fs.watch works again, re-arming only the watcher that stopped and warning again only after a healthy tick (teams and tasks, EMFILE and ENOSPC), and stop() closing both watchers and the interval. All nine new or changed cases fail on the previous head.
  • The CLAUDE.md paragraph and the helper's JSDoc now say the owner's poll creates the watcher again, and why (the ENOENT race).

Taking out the re-arm, either null reset, the once-per-outage check or either outage reset each turns tests red. Typecheck, lint, format and the frontend checks pass; the full suite passes apart from the two local-model files that fail on unmodified master on this machine.

@Ark0N

Ark0N commented Nov 19, 2026

Copy link
Copy Markdown
Owner

Thanks for the follow-up in c7b6e1cb. This PR closes a persistent: false chokidar watcher on its first error through closeWatcherOnError(), so a failing fs.watch costs one warning instead of an endless climb to /, and TeamWatcher now re-creates a stopped watcher from its 30 s poll.

The re-arm works as we discussed: one warning per outage, the watcher comes back on the next tick, and the ENOENT-once case finds a new team without a poll. On master's watcher code, 15 of the 18 tests fail (the three healthy cases pass), and removing the add() no-op brings back the 2,000 failing attempts in the race case. Typecheck, lint, format, the frontend checks and the full npm test (757 files) all pass.

Two small things:

  • test/watcher-fs-watch-failure.test.ts:100, :429 and :537: the file only passes on Linux. On macOS, chokidar 3 never calls fs.watch for these watchers. It uses fsevents, or polling when fsevents is missing (node_modules/chokidar/index.js:322-332), so the stub never fires and the waits time out. With chokidar forced onto its macOS polling path, 17 of the 18 tests fail. npm test has to stay green on any machine, so the three describe blocks need describe.skipIf(process.platform === 'darwin'), plus a comment saying the loop can't happen there because fs.watch isn't used.
  • src/team-watcher.ts:119: stop() leaves watchersDown filled, so after a stop() and start() on the same instance, it stays silent about a watcher that is still failing. The server only starts and stops TeamWatcher once, so this doesn't show up today. Adding this.watchersDown.clear() next to the other clears fixes it.

Next step: I will add both on top when I merge, so you don't need to do anything. If you'd rather push them to this branch yourself, do it before then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants