merge multiple parquet files as single dataset

[kraktus]

Hello, thanks for the lib!

Something I haven't quite figured out is how to handle parquet dataset splits. Hugging face is notorious for these, splitting those big dataset is ~150Mb parquet files each, eg: https://huggingface.co/datasets/Lichess/chess-puzzles/tree/main/data.

However I haven't found if there's a way to merge handle those separate files as one dataset with hyperparquet.

[platypii]

Hi @kraktus thanks for the question. The short answer is no there is not currently a way to combine a directory of parquet files like that.

It IS something we are working on for hyperparam-cli for use with hightable. But I think that should be at a level ABOVE hyparquet. Basically I'm saying that hyparquet is meant for parsing ONE parquet file, and then other abstractions (like multi-file) can be built on top of it. Open to suggestions though!