to_parquet() serialization options.

[betolink]

This is me not not knowing the features of this package very well, I'm thinking about writing a partitioned geoparquet file similar to this:

from pyarrow.dataset import write_dataset

write_dataset(
    items_arrow, # many STAC items, say 1M 
    output_path,
    format='parquet',
    partitioning='year', # column added to the stac items
    filesystem=fs,
    max_rows_per_file=50000,
)

Is there an equivalent with to_parquet()? the idea is that when I need to keep adding items I only touch the most recent year/partition.

EDIT: looking at the code, it seems that the API would need to change to use pyarrow.parquet.write_to_dataset (instead of pyarrow.parquet.ParquetWriter), is there a performance consideration for not using the write_to_dataset function? or would partitions be handled by a future release?

[TomAugspurger]

I don't think there were any discussions about using the dataset API.

IIUC, the main thing to ensure is that the geoparquet metadata is written correctly. If we can do that in a more general way without loosing anything then I'd be fine with using the dataset API, or offering a separate API that uses the dataset API.