test(parquet): allocate raw Arrow reader buffers from the default pool - #386
Merged
Merged
Conversation
Several Parquet tests open files with arrow::io::ReadableFile and parquet::arrow::OpenFile, passing a raw pointer to the fixture's GetArrowPool() adaptor. Pre-buffering is enabled by default, so an Arrow IO thread can drop the last reference to a read buffer after the fixture has been torn down. ~PoolBuffer() then frees into the destroyed pool and paimon-parquet-format-test crashes with SIGSEGV, as seen in VariantParquetTest. Use arrow::default_memory_pool() for these raw readers so the pool outlives any in-flight async read. Paimon's own read path is unaffected: ArrowInputStreamAdapter already keeps its pool alive for returned buffers.
SteNicholas
force-pushed
the
PAIMON-385
branch
from
September 23, 2026 15:40
3446e4a to
1d3f63a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #385
Several Parquet tests open files directly with
arrow::io::ReadableFileandparquet::arrow::OpenFile, passing a raw pointer to the fixture'sGetArrowPool()adaptor. Pre-buffering is enabled by default, so an Arrow IO thread can drop the last reference to a read buffer after the fixture has been torn down.~PoolBuffer()then frees into the destroyed pool andpaimon-parquet-format-testcrashes with SIGSEGV, as seen inVariantParquetTest.This PR switches these raw readers to
arrow::default_memory_pool(), which outlives any in-flight async read. It also adds the error status to theReadableFile::Openassertion messages so that failures show why the open failed.Paimon's own read path is unaffected:
ArrowInputStreamAdapteralready keeps its pool alive for returned buffers.Tests
Test-only change in:
src/paimon/format/parquet/parquet_file_batch_reader_test.cppsrc/paimon/format/parquet/parquet_format_writer_test.cppsrc/paimon/format/parquet/variant_parquet_test.cppAPI and Format
No.
Documentation
No.
Generative AI tooling
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with Claude Code