GH-45902: [Python][Parquet] Expose ParquetWriter properties and arrow_properties #47087
Conversation
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or See also: |
SoundBot
force-pushed
the
expose-parquetwrite-props
branch
3 times, most recently
from
f921baf to
db3942b
Compare
SoundBot
changed the title
|
|
There was a problem hiding this comment.
Thanks for submitting a PR!
From a quick look the test needs a @pytest.mark.pandas mark and some style changes are needed to fix linter errors:
diff --git a/cpp/src/parquet/arrow/writer.cc b/cpp/src/parquet/arrow/writer.cc
index 3724e2edf0..cdaeebc294 100644
--- a/cpp/src/parquet/arrow/writer.cc
+++ b/cpp/src/parquet/arrow/writer.cc
@@ -473,7 +473,9 @@ class FileWriterImpl : public FileWriter {
const WriterProperties& properties() const override { return *writer_->properties(); }
- const ArrowWriterProperties& arrow_properties() const override { return *arrow_properties_; }
+ const ArrowWriterProperties& arrow_properties() const override {
+ return *arrow_properties_;
+ }
::arrow::MemoryPool* memory_pool() const override {
return column_write_context_.memory_pool;
diff --git a/python/pyarrow/_parquet.pyx b/python/pyarrow/_parquet.pyx
index f1287d7b8a..6f3b9a5f[43](https://github.com/apache/arrow/actions/runs/16257450496/job/45897414651?pr=47087#step:5:44) 100644
--- a/python/pyarrow/_parquet.pyx
+++ b/python/pyarrow/_parquet.pyx
@@ -2458,7 +2458,6 @@ cdef class WriterPropertiesWrapper(_Weakrefable):
return self.props.default_column_properties().statistics_enabled()
-
cdef class ArrowWriterPropertiesWrapper(_Weakrefable):
cdef:
shared_ptr[ArrowWriterProperties] props
I will have a look at the proposed code shortly.
github-actions
bot
added
awaiting committer review
SoundBot
force-pushed
the
expose-parquetwrite-props
branch
from
db3942b to
23d3161
Compare
SoundBot
force-pushed
the
expose-parquetwrite-props
branch
from
23d3161 to
3adfc51
Compare
|
@AlenkaF thank you, I added |
| df = _test_dataframe(100) | ||
| table = pa.Table.from_pandas(df, preserve_index=False) |
There was a problem hiding this comment.
not specifically related to this PR but would be nice to have an utility function to create an Arrow table directly instead of having to go through pandas. This test doesn't really have to require pandas but the utility function is handy. We can create an issue to move some of those tests to use a new utility function and remove the @pytest.mark.pandas for some of those tests. @AlenkaF what do you think? It could be a good-first-issue
There was a problem hiding this comment.
Agree, adding a new utility function to create Arrow tables and/or batches would be great! And perfect for good-first-issue 👍
|
@github-actions crossbow submit -g python -g wheel |
github-actions
bot
added
awaiting changes
|
Revision: 3adfc51 Submitted crossbow builds: ursacomputing/crossbow @ actions-623832f922
|
|
The |
Rationale for this change
Exposes
ParquetWriterproperties viawriter.propertiesandwriter.arrow_properties(see #45902)
What changes are included in this PR?
See above
Are these changes tested?
Yes
Are there any user-facing changes?
Yes, properties are available via
writer.propertiesandwriter.arrow_properties