Performance difference with hyparquet ?

[AbdealiLoKo]

Curious if anyone has tried both parquet-wasm and hyparquet and found any performance difference

[kylebarron]

hyparquet is pure JS and loads to JS objects. So I'd expect:

• If you only need to load the metadata of Parquet files, or if you have very small Parquet files, hyparquet could be faster as hyparquet is smaller.
• If you need the absolute smallest bundle size, hyparquet may be better for your use case.
• Since parquet-wasm 1. uses a really high performance rust library, 2. is running in WebAssembly, and 3. converts to a high-performance binary memory format. If you have large files, and can use the resulting Arrow data directly without converting that to JS objects, I'd expect parquet-wasm would be significantly faster.

But I don't have hard numbers. If you do any benchmarking I'd love to hear!

[nuts-rice]

I think it'd be a bit useful to have a script for running benches (check for poetry being installed, other build chores, printing location of benny's output). Then it'd be bit easier in CI runs or whatever to elucidate performance differences.
If you think it's worthwhile I'd add one in a PR

[kylebarron]

I think that would be great, but maybe it should live in an external repository so that it can be rerun in the future with future hyparquet and parquet-wasm updates?