Parquet Dataset API

[kylebarron]

/// Reader interface for a collection of Parquet files.
#[pyclass(module = "arro3.io", frozen)]
pub struct ParquetDataset {
    parquet_meta: ArrowReaderMetadata,
    files: HashMap<object_store::path::Path, ArrowReaderMetadata>,
    schema: SchemaRef,
    store: Arc<dyn ObjectStore>,
}

[yohplala]

Hello @kylebarron

I am managing parquet dataset to store ordered data, and I am interested in this new ParquetDataset feature.

My data worklow involves writing parquet files in different runs (one row group per file). Common metadata are thus evolving between each writing of the separate parquet files, because:

• row group list changes (files, i.e. row groups, removed or added)
• statistics change (because of removed or added files, i.e. row groups)
• modification of custom key-value metadata (adding, removing, keys or modifying value of existing keys)

Please, when reading from different parquet files with ParquetDataset, how do you consolidate these metadata (which end up to be not so "common") from different individual parquet files? Is this a use case identified?

The questions could be expanded with other items, but these are not critical in my workflow, because I think these items can be processed later, when reading each row group actual data,

• new columns appearing in new parquet files
• new categorical values appearing in column with categorical dtype

Thanks a lot for sharing these insights!

[kylebarron]

how do you consolidate these metadata

If you're continuously updating a directory with new data, then in effect the only way to handle the metadata is to read all the files' metadata when you open the dataset.

new columns appearing in new parquet files

If the schema doesn't match in all of your files, you'll get an error. If you need schema evolution, use something like iceberg or delta lake.

[yohplala]

Thank you for your replies!

how do you consolidate these metadata

If you're continuously updating a directory with new data, then in effect the only way to handle the metadata is to read all the files' metadata when you open the dataset.

Is it what will ParquetDataset do?
If I instanciate, then call statistics like this:

from arro3 import ParquetDataset

my_pds = ParquetDataset(["my.parquet", "list.parquet", "of.parquet", "files.parquet"])
my_pds.statistics   # triggers metadata reading for each file?
{
#                 rg1, rg2, rg3, rg4
 "min": {"col1": [  1,   3,   6,   9]},
 "max": {"col1": [  2,   4,   7,  10]}
}

[kylebarron]

my_pds = ParquetDataset(["my.parquet", "list.parquet", "of.parquet", "files.parquet"])

Do these files have the same schema or not? If they don't have the same schema, it'll error on this line.

And no, that initial line will trigger metadata reading for all files, before you even get to my_pds.statistics. (Though, to be clear, those statistics come from the metadata, not from the full data)

[yohplala]

my_pds = ParquetDataset(["my.parquet", "list.parquet", "of.parquet", "files.parquet"])

Do these files have the same schema or not? If they don't have the same schema, it'll error on this line.

In my use case, they will.
I mentioned the possibility some could have different schema because I am trying at the same time to assess difference with fastparquet capabilities.

And no, that initial line will trigger metadata reading for all files, before you even get to my_pds.statistics. (Though, to be clear, those statistics come from the metadata, not from the full data)

Ok, clear, thank you!

Once the feature will be out, I will give it a try.
I still wonder about the behavior with custom key-value metadata.
I see it is possible to record them
write_parquet(..., key_value_metadata: [dict] | None = None, ...)

When reading back with ParquetDataset, would it be the kvm of a specific file only (like the kvm of last file for instance?) which is made available? Or none at all? Or are they all concatenated?

Thanks again for sharing your insights!

[kylebarron]

When reading back with ParquetDataset, would it be the kvm of a specific file only (like the kvm of last file for instance?) which is made available? Or none at all? Or are they all concatenated?

In theory that's part of the schema and they should all be the same. I'm not sure what implementation details will look like