fix: forward write_options when write_parquet receives ParquetWriterOptions - #1761
Open
Rodrigo-Palma wants to merge 1 commit into
Open
Rodrigo-Palma wants to merge 1 commit into
Rodrigo-Palma wants to merge 1 commit into
Conversation
…ptions
`write_parquet` takes `write_options`, documents it and declares it in
the `@overload` for the `ParquetWriterOptions` form, but that branch
calls `write_parquet_with_options(path, compression)` and drops it. The
destination accepts the parameter, so everything in
`DataFrameWriteOptions` (`partition_by`, `single_file_output`,
`insert_operation`, `sort_by`) was silently ignored for that one
spelling.
Measured with the same `DataFrameWriteOptions(partition_by="part")`:
write_parquet(path, ParquetWriterOptions(), write_options=wo)
-> ['IDuOjvMa3pdEDotb_0.parquet'] not partitioned
write_parquet_with_options(path, ParquetWriterOptions(), write_options=wo)
-> ['part=a', 'part=b']
write_parquet(path, "zstd", write_options=wo)
-> ['part=a', 'part=b']
No error and no warning: the files just land in the wrong layout.
The branch arrived in ef62fa8 (apache#1169) while `write_options` came
earlier in apache#857, so the new delegation path was written without carrying
the existing parameter over. The same `if` refuses `compression_level`
with an explicit `ValueError`, which shows that arguments incompatible
with this branch get rejected on purpose; `write_options` was not
rejected, only forgotten.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1760.
Rationale for this change
DataFrame.write_parquetacceptswrite_options, documents it and declares it in the@overloadfor theParquetWriterOptionsform, but that branch delegates without forwarding it:write_parquet_with_options(path, options, write_options=None)takes the parameter, so everything inDataFrameWriteOptions(partition_by,single_file_output,insert_operation,sort_by) was silently ignored for that one spelling. Measured with the sameDataFrameWriteOptions(partition_by="part")in all three calls:Only the
ParquetWriterOptionsbranch loses the Hive partitioning, with no error and no warning, so the files just land in the wrong layout.The branch came in ef62fa8 (#1169) while
write_optionswas added earlier in #857, so the new delegation path was written without carrying the existing parameter over. Note that the sameifrefusescompression_levelwith an explicitValueError: arguments genuinely incompatible with this branch get rejected on purpose, andwrite_optionswas not rejected, only forgotten.What changes are included in this PR?
One line:
write_optionsis passed through towrite_parquet_with_options. Plus a regression test,test_write_parquet_writer_options_keeps_write_options, next totest_write_parquet.AssertionError: assert ['9UNQ0jX8PC8GFZxz_0.parquet'] == ['part=a', 'part=b']python/tests/test_dataframe.py: 327 passed with the change against 326 on a clean tree, with the same 4 pre-existing errors in both runs (test_logical_plan,test_optimized_logical_plan,test_execution_plan,test_async_iteration_of_df), so the delta is exactly the new test.ruff 0.15.1 checkandruff format --checkare clean on both files.Measurement caveat: I ran the Python layer of this branch against the published
datafusion 54.0.0wheel rather than a locally built_internal, so the Rust side is the released one. The block being changed is byte-identical between that wheel andmain, which I verified by diffing it, so the behaviour shown above is the behaviour onmain.Are there any user-facing changes?
Yes, in the sense that
write_optionsnow takes effect for this call shape, which is what the signature, the docstring and the overload already promise. No API change: no signature, name or default is touched.