Appending to parquet files

[simonvandel]

Hi

Does arrow-rs support appending data to an existing Parquet file?

If I'm understanding https://github.com/apache/parquet-format#file-format correctly, it should theoretically be possible to implement append fairly efficiently:

• read metadata of existing file
• find the offset of the end of the last row group. Right after that will be the metadata
• insert new row groups in between the last row group and the metadata
• update the metadata to reflect the new row groups

Am I missing something in the above description? Is that already possible?

If append is not possible, the alternative is to read and decode the existing parquet file, and then reencoding the entire file again. I would like to avoid this though, and reuse the existing row groups.

Thanks.

[tustvold]

This is definitely technically feasible, and has come up before (#557). There are some challenges around how to handle other metadata such as page index information, bloom filters, etc... I'll see if I can't write up how this could be implemented

[alamb]

I think the current way to handle this is not to update the parquet file, but instead write a new parquet file and then merge them together at query time, which many query engines support

Then when enough small files accumulate to rewrite them all into a new file.

I can see that for some use cases appending would be helpful, though the metadata for the entire file probably needs to be rewritten each time

[simonvandel]

Making smaller files that somehow gets accumulated to larger ones is definitely my alternative. But if it's possible to directly grow an existing file, that would be better.

I'm storing the parquet files in S3, so querying many smaller files is much more expensive than querying a single, bigger, file once. If I can avoid having smaller files at all, I think that would make things nicer.

[tustvold]

I've created #4155 which contains an API proposal, this would not permit in-place rewrites, although given S3 doesn't support this anyway, I presume this isn't a problem? It is also worth highlighting that this won't re-encode data, which may not be desirable when merging very small files where you may want to combine row groups to get better storage efficiency