Huge coordinate variable impact

[christophenoel]

Hi,

For an Earth Observation mission, we're considering generating a 2D data array shaped with 1 million x 1 million coordinates.
What would be the impact on performance (e.g. load time, query) of such huge index ?

In particular, will xarray load the 4MB variables to be able to resolve queries based on coordinates ?

Thanks for the clarification.

[TomNicholas]

Hi @christophenoel - the coordinate array can be dask-chunked, but currently you would still require two 1-million element long pandas index objects, one along each coordinate dimension. These currently do need to be in-memory, but there is discussion in #1650 as to how the upcoming flexible indexes refactor (https://github.com/pydata/xarray/projects/1) might enable chunked indexes.