Loading datasets larger than memory

[Logon27]

Hello, I was just curious if jax had any best practices or tutorials for loading datasets that are larger than memory. How is this typically accomplished? Do people normally just keep the dataset on disk and load the data in memory in batches? And is there any specific jax functionality that accomplishes this? Thanks for your time.

[chrisflesher]

This is what data loaders are for, there are some alternatives described here:
#16785

I've also used zarr and dask previously to do stuff like this.