Understanding the behaviour of load_from_disk

[nihaljn]

I have a very large dataset with 32M examples stored as .arrow table using save_to_disk. When I use load_from_disk to load this dataset the first time (i.e., the first time after a reboot for example), it's really slow and takes > 10 minutes to complete. For every subsequent call to load_to_disk it's very fast and completes in a fraction of a second. Why does this happen? Is this due to some caching to memory? Can the cache be set to create to disk instead?

[mariosasko]

Hi! You can check #2252 to find more info on this behavior. We plan to add an option to shard arrow files on save to address this.

[mariosasko]

save_to_disk shards the arrow file to make it smaller by default now :)