Virtual columns?

[severo]

On Parquet files like this one: https://source-cooperative.github.io/parquet-table/?lens=metadata&url=https://data.source.coop/youssef-harby/exiobase-3/exio3_parquet_3-9-5/IOT_1995_pxp/discovered_core/G.parquet (9802 columns, 9800 rows), hightable breaks.

One solution is to [limit the number of Another one is to virtualize the columns, as we do for the rows.

To be honest, I think it's overkill + can be complex to combine with columns autosize.

[severo]

Note that the metadata itself in this file is more than 7MB: maybe something we want to reduce too (read only part of the metadata, or lazy-load it in some way?)

[platypii]

Note that the metadata itself in this file is more than 7MB: maybe something we want to reduce too (read only part of the metadata, or lazy-load it in some way?)

Sidenote: There's a whole bunch of discussion right now in the parquet community about how to more-efficiently parse metadata. Either for many-rowgroups or many-columns. Some of the more performance-minded parquet clients (rust i think?) are experimenting with skipping the intermediate thrift representation, and writing a custom binary -> metadata decoder that is specific to parquet. Generated with macros. In theory it could skip parsing unnecessary data such as the statistics. Neat idea, but I'm not convinced we need it in javascriptland.

[platypii]

I do like the idea of using virtualized columns -- or at least being aware of which columns are in view, and only fetching data for those in view.

[severo]

already asked in #223 btw

[yharby]

Sorry 😂

[severo]

On the contrary, thanks for breaking the library :) We should handle this case gracefully, and lazy loading the columns will benefit all the datasets by reducing the transferred data and the latency.

[kojo12228]

I was just visiting the project to see if there was still demand/any traction on this, good to see there is!

[kojo12228]

I wonder if using react-window would be a good solution here, it has nice support for both row and column virtualization. Auto-sizing seems very difficult for any library, but maybe the first 1000 rows (possibly configurable by the user) cells can be rendered hidden, the widths computed and then fixed per column, and then those column widths used for all subsequent rows

[severo]

Thanks for referencing https://react-window.vercel.app/grid/grid, it looks interesting.