Extracting valid values from large raster

[samfav]

Hi,
I have a large (40000x40000) GeoTiff file with approximately 2% of the pixels that have a valid (different from NODATA) value. I want to extract those values to a column dataset and extract the values at the same location in other aligned rasters.
The end result is a column database of points valid points.

Here is some code that I have written to perform this task :

da = rioxarray.open_rasterio(f"{BASE_PATH}large_tiff.tif")

da.where(da!=NODATA).stack(z=('x', 'y')).dropna('z').reset_index('z').to_netcdf(f"{BASE_PATH}tiff_selection.nc", engine='netcdf4')

The issue with this code is that it fills up my memory. I have tried using different chunk sizes on the opened tiff, dropping the NODATA values inside the where to no avail etc.

Has anyone performed a similar task using rioxarray and could point me to the right way of doing this?

Ultimately I want the data to be stored to parquet but that's another problem.
Thanks for your help,

Sam

[keewis]

I don't have any experience with this kind of task, but you might want to have a look at the sparse library and sparse.COO.from_numpy in particular.

[samfav]

Thanks for the input, I'll take a look at it.

[dcherian]

Hi @samfav you will probably have better luck at the rioxarray repo. Would you mind moving your question there?

[samfav]

Sure, I will ask the question there, although the issue seemed to be when processing the xarray, not really while opening the rasters.
Feel free to close this discussion!