Skip to content

fix(realtime): keep spill outside the write mutex - #384

Merged
lxy-9602 merged 2 commits into
apache:mainfrom
HaHaJeff:codex/realtime-spill-outside-write-mutex
Sep 23, 2026
Merged

lxy-9602 merged 2 commits into
apache:mainfrom
HaHaJeff:codex/realtime-spill-outside-write-mutex

Conversation

@HaHaJeff

@HaHaJeff HaHaJeff commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Purpose

Follow up on #327 to avoid blocking realtime writes on sealed-segment spill I/O in append-only and primary-key writers.

  • Serialize Seal() and PrepareCommit() with prepare_mutex_.
  • Run the built-in ArrowRealtimeStore::SealForCommit() outside realtime_store_mutex_. The store already protects segment rotation internally and spills outside its own mutex, allowing writes to the next building segment during spill.
  • Reacquire the write mutex to publish sealed segments and derive has_building_data_ from the current offset and sealed range, preserving data written during spill.

Tests

Local validation completed for this fix:

  • Realtime core tests: 94/94 passed.
  • Realtime integration tests: 77/77 passed.
  • Controlled temporary concurrency validation: 4/4 cases passed; this validation was not added as a permanent test.
  • git diff --check: passed.
  • pre-commit was unavailable, so its checks were not run.

API and Format

No public API, storage format, or protocol changes.

Documentation

No documentation changes; this fixes the existing spill behavior.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

@HaHaJeff
HaHaJeff force-pushed the codex/realtime-spill-outside-write-mutex branch from 2e29525 to c082cb8 Compare September 23, 2026 09:15
Comment thread src/paimon/core/realtime/realtime_append_only_writer.cpp

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 6fa3c81 into apache:main Sep 23, 2026
28 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants