Open large dataset without loading the whole file in RAM #8113
-
What is your issue?Is it possible to load a large dataset (larger than available RAM) by providing preconditions similar to standard indexing (for example, sel(dim_x = value))? If not, is it possible to add this functionality? I am planning to use xarray to store data gathered from neuroscience experiments that tend to by quite large and might come across this issue. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
Xarray opens files lazily by default, and our lazy classes understand indexing, so what you're asking for should already work by default. If you want to actually process larger-than-RAM datasets (not just view subsections of them) you will need to install dask. |
Beta Was this translation helpful? Give feedback.
-
|
@Dirtywind please let us know if you run into problems trying this! |
Beta Was this translation helpful? Give feedback.
-
|
All good! I wasn't aware of lazy computing and dask since I am relatively new to coding but, I got it working just as I wanted it. |
Beta Was this translation helpful? Give feedback.
Xarray opens files lazily by default, and our lazy classes understand indexing, so what you're asking for should already work by default. If you want to actually process larger-than-RAM datasets (not just view subsections of them) you will need to install dask.