RuntimeError: NetCDF: DAP failure

[b-kode]

Hi all,

I am interested in extracting specific point and variable information from the GEOS-FC product, accessible via OpenDap.

Loading the data seems to work fine, and I can do some processing to my specific needs.
Ideally I would like to convert this selection to a dataframe, or if needed store as an intermediate file from which I can read again.

Yet when doing so, I get the following error: RuntimeError: NetCDF: DAP failure

I am not sure what is causing this? Perhaps I chunck the data in the wrong (inefficient) way? Or there is an error with the GEOS netcdf files? Or ...

Below a working code snippet.

import xarray as xr
idir_geos   = 'https://opendap.nccs.nasa.gov/dods/gmao/geos-cf/assim/chm_tavg_1hr_g1440x721_v1'

def preprocess(ds):
    ''' Rename variables and select the relevant ones. Remove lev'''
    ds = ds.rename({'pm25_rh35_gcc': 'PM2.5','no': 'NO','no2': 'NO2','o3': 'O3','so2': 'SO2','co': 'CO'})
    ds = ds[['PM2.5','NO','NO2','O3','SO2','CO']]
    ds = ds.squeeze('lev')
    return ds

ds = xr.open_mfdataset([idir_geos],preprocess=preprocess,combine='by_coords')

lat  = 51.25
lon = 4.25
pol = 'O3'
ds_sel = ds.sel(lat=lat,lon=lon,method='nearest')[pol]

df_sel = ds_sel.to_dataframe().drop(['lat','lon'],axis=1) 
#ds_sel.to_netcdf('test.nc') # Runtime error

Traceback error:

Traceback (most recent call last):
File "/home/demuzmp4/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 57, in
df_sel = ds_sel.to_dataframe().drop(['lat','lon'],axis=1)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/dataset.py", line 4285, in to_dataframe
return self._to_dataframe(self.dims)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/dataset.py", line 4273, in _to_dataframe
for k in columns
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/dataset.py", line 4273, in
for k in columns
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/variable.py", line 437, in values
return _as_array_or_item(self._data)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/variable.py", line 250, in _as_array_or_item
data = np.asarray(data)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray
return array(a, dtype, copy=False, order=order)
File "/usr/lib/python3/dist-packages/dask/array/core.py", line 1138, in array
x = self.compute()
File "/usr/lib/python3/dist-packages/dask/base.py", line 135, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/usr/lib/python3/dist-packages/dask/base.py", line 333, in compute
results = get(dsk, keys, **kwargs)
File "/usr/lib/python3/dist-packages/dask/threaded.py", line 75, in get
pack_exception=pack_exception, **kwargs)
File "/usr/lib/python3/dist-packages/dask/local.py", line 521, in get_async
raise_exception(exc, tb)
File "/usr/lib/python3/dist-packages/dask/compatibility.py", line 60, in reraise
raise exc
File "/usr/lib/python3/dist-packages/dask/local.py", line 290, in execute_task
result = _execute_task(task, data)
File "/usr/lib/python3/dist-packages/dask/local.py", line 271, in _execute_task
return func(*args2)
File "/usr/lib/python3/dist-packages/dask/array/core.py", line 72, in getter
c = np.asarray(c)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray
return array(a, dtype, copy=False, order=order)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/indexing.py", line 490, in array
return np.asarray(self.array, dtype=dtype)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray
return array(a, dtype, copy=False, order=order)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/indexing.py", line 652, in array
return np.asarray(self.array, dtype=dtype)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray
return array(a, dtype, copy=False, order=order)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/indexing.py", line 556, in array
return np.asarray(array[self.key], dtype=None)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray
return array(a, dtype, copy=False, order=order)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/coding/variables.py", line 73, in array
return self.func(self.array)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/coding/variables.py", line 142, in _apply_mask
data = np.asarray(data, dtype=dtype)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/asarray.py", line 85, in asarray
return array(a, dtype, copy=False, order=order)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/indexing.py", line 556, in array
return np.asarray(array[self.key], dtype=None)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/backends/netCDF4.py", line 72, in getitem
key, self.shape, indexing.IndexingSupport.OUTER, self.getitem
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/indexing.py", line 836, in explicit_indexing_adapter
result = raw_indexing_method(raw_key.tuple)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/backends/netCDF4.py", line 84, in _getitem
array = getitem(original_array, key)
File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/backends/common.py", line 54, in robust_getitem
return array[key]
File "netCDF4/_netCDF4.pyx", line 4408, in netCDF4._netCDF4.Variable.getitem
File "netCDF4/_netCDF4.pyx", line 5352, in netCDF4._netCDF4.Variable._get
File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: DAP failure

More info on my xarray installation:

commit: None
python: 3.6.9 (default, Jul 3 2019, 07:38:46)
[GCC 8.3.0]
python-bits: 64
OS: Linux
OS-release: 4.15.0-66-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_GB.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.3
xarray: 0.14.0
pandas: 0.25.2
numpy: 1.17.3
scipy: 1.3.1
netCDF4: 1.5.3
pydap: installed
h5netcdf: None
h5py: 2.9.0
Nio: None
zarr: 2.3.2
cftime: 1.0.4
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.0.28
cfgrib: None
iris: None
bottleneck: 1.2.1
dask: 0.16.0
distributed: None
matplotlib: 3.1.1
cartopy: 0.17.0
seaborn: 0.9.0
numbagg: None
setuptools: 41.4.0
pip: 9.0.1
conda: None
pytest: 5.2.1
IPython: 7.3.0
sphinx: 1.8.4

[max-sixty]

Hi @b-kode , thanks for the issue

Please could you paste the stack trace? And if possible adjust the example so it can be pasted (check out GH formatting documentation if needed). Thanks

[b-kode]

Hi @max-sixty,

I have added the traceback.
And apparently @dcherian already edited the example to its proper format.

Thanks!

[HarlanAndrew]

Hi all,

I encountered the same problem when trying to save data from met.no's archived MEPS data on "https://thredds.met.no/thredds/dodsC/meps25epsarchive/" when trying to save the remote dataset as a local netCDF file.

I found that it depends on the size of the dataset I want to save and it works fine up to about 5MB file size. Thus, my workaround is to chunk the data e.g. by selecting only one ensemble member at a time, save the chunk as a file to make sure everything is downloaded, then read in again and do dataset.merge() in order to get one netCDF file.

It seems to me as if the problem could be solved by writing to hard disk in chunks.

Hope this helps to find the root of this issue....

Thanks for opening this issue @b-kode!

[kuchaale]

Hi all,
I encountered the same problem when trying to download NASA's GEOS-5 data (see below). It worked occasionally but I had to restart the script several times.

import xarray as xr
import pandas as pd

URL ='https://opendap.nccs.nasa.gov/dods/GEOS-5/fp/0.25_deg/assim/inst3_3d_asm_Np'
ds = xr.open_dataset(URL,engine='netcdf4')
var_ls = ['omega', 't', 'v', 'u']
lev_ls = [1000.,  975.,  950.,  925.,  900.,  875.,  850.,  825.,  800.,
        775.,  750.,  700.,  650.,  600.,  550.,  500.,  450.,  400.,
        350.,  300.,  275.,  250.,  225.,  200.,  175.,  150.,  125.,
        100.,   70.,   50.,   30.,   10.,    5.,3, 2, 1]

time_range = pd.date_range('2021-01-02T12', '2021-01-07', freq = '6H')

for sel_date in time_range:
    ds_sel = ds[var_ls].sel(
        time = sel_date,
        lev = lev_ls,
        method = 'nearest'
    )
    ouf_date=sel_date.strftime('%Y%m%d%H')
    outfile = f'geos5_subset_{ouf_date}.nc'
    print(outfile)
    ds_sel.to_netcdf(outfile) 

[EDIT] It may have helped to add .load before to_netcdf though.