to_netcdf memory error

[Amarounek]

Hello guys,

I have an issue with saving the dataset to NetCDF4. It is actually 499 NetCDF files, which I would like to merge and save.

Here's my code snippet:

import pandas as pd
import numpy as np
import xarray as xr
import os

files_xr = [f for f in os.listdir(os.getcwd()) if f.startswith("Precipitation") and f.endswith(".nc")]
files_xr_mer = xr.open_mfdataset(files_xr, combine='by_coords')
files_xr_mer['units'] = 'mm'
new_filename_1 = './prec_file_testing.nc'
files_xr_mer.to_netcdf(path=new_filename_1)

Traceback (most recent call last)
MemoryError: Unable to allocate 2.80 GiB for an array with shape (29, 3601, 7199) and data type float32

Is there any other way how to try to export the dataset to NetCDF4?

The open topic there: https://stackoverflow.com/questions/66139315/memoryerror-during-the-netcdf4-saving-via-xarray

I can provide info of dataset later as I am on my second computer...

[Amarounek]

So? Nobody? :/

[benoitespinola]

I am having the same error.. My NetCDF files are less than 1Gb each, I am trying to merge 6 files (them 6 files together have a total of 5.6654 GiB to be precise).

Doing the following code:
import xarray as xr

from memory_profiler import profile

@profile
def main():
    path = 'data.nc'

    data = xr.open_mfdataset(path)
    
    data = data.load()
    data = data.compute()



if __name__=='__main__':
    main()

This yields to :

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     5     71.8 MiB     71.8 MiB           1   @profile
     6                                         def main():
     7     71.8 MiB      0.0 MiB           1       path = 'data.nc'
     8                                         
     9    100.7 MiB     28.9 MiB           1       data = xr.open_mfdataset(path)
    10                                         
    11   9286.3 MiB   9185.6 MiB           1       data = data.load()
    12                                         
    13   9286.3 MiB      0.0 MiB           1       data = data.compute()

That means, loading my files consumes as much memory as the file's size.
I do not know why, when I merge files, it just fills up the available ram (in my case 185 GiB).

I think loading both files in different variables, creating a 3rd variable with the right dimensions and assigning the values would take less ram than that.

I would hope someone would approach this issue.

[benoitespinola]

Raised a bug ticket with this issue. Let's see if we manage to get some replies.