Skip to content

fix(file-search): recover dispatch claims abandoned between commit and enqueue - #8209

Merged
icecrasher321 merged 3 commits into
stagingfrom
fix/file-search-dispatch-handoff
Sep 24, 2026
Merged

icecrasher321 merged 3 commits into
stagingfrom
fix/file-search-dispatch-handoff

Conversation

@icecrasher321

Copy link
Copy Markdown
Collaborator

Summary

  • Dispatch claims commit before Trigger.dev accepts their runs. A dispatcher that stopped in between, for example killed at its 60-second limit while PostgreSQL was still committing (a commit wait is not bounded by statement_timeout), left claims that no run would finish, and they held their workspace's two slots until the six-hour stale reaper.
  • Each claim now carries handoff_expires_at (additive nullable column, PostgreSQL clock), set to twice the dispatcher's maximum duration when claimed and cleared once a run is known to exist: Trigger.dev accepted the batch, or the worker began the build. The next dispatch releases a claim whose deadline lapsed and claims the file again under a new token; the existing token fencing turns any late or duplicate run into a no-op.
  • The handoff write runs in a bounded transaction and skips rows another transaction holds, so it cannot stall runs beginning their build or deadlock with bulk file changes; a failed write is only logged and never releases claims. Claims released for a lapsed deadline are logged and counted in the dispatch result, and a run that no longer owns its revision logs it.
  • NULL means no pending handoff, so legacy claims and claims from the still-deployed dispatcher keep the six-hour window across the migrate → app → Trigger promotion order. In-process dispatch (no Trigger.dev) still clears the deadline when it schedules work in memory, so a restart there keeps the six-hour window; the README says so.
  • Kept in the dispatcher rather than the transactional outbox: the dispatcher already reconciles these claims every minute, and an outbox event would add a second record with its own retry and dead-letter lifecycle to keep in step with the claim.

Type of Change

  • Bug fix

Testing

  • 192 file-search unit tests; 72 PostgreSQL 17 integration tests under TZ=America/Los_Angeles (one optional benchmark skipped), covering release after the deadline, enqueued claims kept through the stale window, a token-fenced handoff write, a locked claim skipped instead of waited on, and a failed handoff write keeping its claims.
  • 12 mutants of the fix (reaper predicate, token filter, lock waiting, deadline units, counts, worker completion) each fail a test.
  • End-to-end on PostgreSQL 17: stalled commits with an unreachable synchronous standby and killed the dispatcher client mid-commit. Without the fix the workspace stayed frozen past two minutes; with it, the first dispatch after the deadline re-dispatched the files.
  • Full lint, app type check, all 47 audits, block-registry check, docs-manifest check, check:migrations origin/staging, and check:api-validation.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Sep 24, 2026 9:25pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The file-search handoff recovery appears safe to merge, with token fencing and backward-compatible nullable deadlines protecting both delayed runs and legacy claims.

Summary

This PR adds bounded recovery for file-search dispatch claims abandoned after database commit but before Trigger.dev accepts the run.

  • Adds a nullable PostgreSQL-time handoff deadline to each newly dispatched claim.
  • Clears the deadline after enqueue acceptance or when the worker begins its fenced build.
  • Reclaims expired handoffs under a new dispatch token while preserving the existing six-hour recovery behavior for handed-off and legacy claims.
  • Adds integration and unit coverage for expiry, token fencing, locked rows, failed handoff writes, and worker ownership logging.
Diagram
sequenceDiagram
  participant D as Dispatcher
  participant DB as PostgreSQL
  participant T as Trigger.dev
  participant W as Index worker

  D->>DB: Claim revision and set handoff deadline
  DB-->>D: Commit claim and dispatch token
  D->>T: Enqueue run
  alt Trigger.dev accepts
    T-->>D: Batch accepted
    D->>DB: Clear handoff deadline using token fence
  else Dispatcher stops before enqueue
    Note over D,DB: Deadline eventually expires
    D->>DB: Release expired claim
    D->>DB: Reclaim using a new token
  end
  T->>W: Start accepted run
  W->>DB: Begin build using dispatch token
  alt Token still owns revision
    DB-->>W: Lock revision and clear deadline
    W->>W: Build index
  else Claim was reclaimed
    DB-->>W: No matching revision
    W->>W: Exit without processing
  end
Loading

Reviews (2) · Last reviewed commit: "chore(db): format dispatch handoff migra..."

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 14 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic

Preserve staging migrations 0380 and 0381, regenerate the dispatch handoff migration as 0382, and align the file-search integration fixture with the mothership module path.
@icecrasher321

Copy link
Copy Markdown
Collaborator Author

@greptile

@icecrasher321

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 14 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic

@icecrasher321
icecrasher321 merged commit addb457 into staging Sep 24, 2026
34 checks passed
@icecrasher321
icecrasher321 deleted the fix/file-search-dispatch-handoff branch September 24, 2026 21:42

This branch was successfully deployed

1 active deployment
Preview — 2d91ec99 Deployed Sep 24, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant