netcdf time:units issues, get around problems using preprocess?

[durack1]

Hi folks, I am an xarray noob, and have been hitting persistent issues with malformed netcdf files. I became aware of the preprocess feature and was keen to get around file issues using this technique. However, I am hitting problems, so it would be great to get some guidance (or to point me at some docs, which I've not yet found).

A minimal reproducible example is below, the file being targeted is available here (unfortunately not available through opendap).

In short, the issue is that time:units = 20O1-1-1 rather than time:units = 2001-1-1 (yes, that is the letter "O" rather than the number 0/zero) - a wonderful edge case that you will not often encounter:

Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:41:22) [Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import xarray as xr
>>> xr.__version__
'2022.6.0'
>>> fh = xr.open_dataset("sic_O1.nc")
Traceback (most recent call last):
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 270, in decode_cf_datetime
    dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar)
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 207, in _decode_datetime_with_pandas
    raise OutOfBoundsDatetime(
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, '360_day', using pandas.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 180, in _decode_cf_datetime_dtype
    result = decode_cf_datetime(example_value, units, calendar, use_cftime)
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 272, in decode_cf_datetime
    dates = _decode_datetime_with_cftime(
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 201, in _decode_datetime_with_cftime
    cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
  File "src/cftime/_cftime.pyx", line 580, in cftime._cftime.num2date
  File "src/cftime/_cftime.pyx", line 110, in cftime._cftime._dateparse
  File "src/cftime/_cftime.pyx", line 781, in cftime._cftime._parse_date
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/backends/api.py", line 531, in open_dataset
    backend_ds = backend.open_dataset(
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/backends/netCDF4_.py", line 569, in open_dataset
    ds = store_entrypoint.open_dataset(
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/backends/store.py", line 29, in open_dataset
    vars, attrs, coord_names = conventions.decode_cf_variables(
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/conventions.py", line 521, in decode_cf_variables
    new_vars[k] = decode_cf_variable(
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/conventions.py", line 369, in decode_cf_variable
    var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 682, in decode
    dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 190, in _decode_cf_datetime_dtype
    raise ValueError(msg)
ValueError: unable to decode time units 'days since 20O1-1-1' with "calendar '360_day'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

So rather, try to deal with this issue whilst calling open_mfdataset (it is not clear to me why the preprocess option is not available in open_dataset, it would be useful):

>>> def fixTime(ds):
...   print("fix time:units")
...   ds.time.encoding["units"] = "days since 2001-01-01"
...   return ds
... 
>>> fh = xr.open_dataset("sic_O1.nc", preprocess=fixTime)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/backends/api.py", line 531, in open_dataset
    backend_ds = backend.open_dataset(
TypeError: NetCDF4BackendEntrypoint.open_dataset() got an unexpected keyword argument 'preprocess'
>>> fh = xr.open_mfdataset("sic_O1.nc", preprocess=fixTime)
Traceback (most recent call last):
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 270, in decode_cf_datetime
    dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar)
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 207, in _decode_datetime_with_pandas
    raise OutOfBoundsDatetime(
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, '360_day', using pandas.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 180, in _decode_cf_datetime_dtype
    result = decode_cf_datetime(example_value, units, calendar, use_cftime)
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 272, in decode_cf_datetime
    dates = _decode_datetime_with_cftime(
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 201, in _decode_datetime_with_cftime
    cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
  File "src/cftime/_cftime.pyx", line 580, in cftime._cftime.num2date
  File "src/cftime/_cftime.pyx", line 110, in cftime._cftime._dateparse
  File "src/cftime/_cftime.pyx", line 781, in cftime._cftime._parse_date
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/backends/api.py", line 972, in open_mfdataset
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/backends/api.py", line 972, in <listcomp>
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/backends/api.py", line 531, in open_dataset
    backend_ds = backend.open_dataset(
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/backends/netCDF4_.py", line 569, in open_dataset
    ds = store_entrypoint.open_dataset(
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/backends/store.py", line 29, in open_dataset
    vars, attrs, coord_names = conventions.decode_cf_variables(
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/conventions.py", line 521, in decode_cf_variables
    new_vars[k] = decode_cf_variable(
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/conventions.py", line 369, in decode_cf_variable
    var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 682, in decode
    dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
  File "~/mambaforge/envs/xcd032/lib/python3.10/site-packages/xarray/coding/times.py", line 190, in _decode_cf_datetime_dtype
    raise ValueError(msg)
ValueError: unable to decode time units 'days since 20O1-1-1' with "calendar '360_day'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

It seems I am omitting a step, or just being a general noob that requires additional hand holding to get me rolling.

Any suggestions would be appreciated.

[kmuehlbauer]

@durack1 It might be easier than anticipated.

Just open like this:

def fixTime(ds):
    print("fix time:units")
    ds.time.encoding["units"] = "days since 2001-01-01"
    return xr.decode_cf(ds)

# make it a one-liner :-)
ds = xr.open_dataset("sic_O1.nc", decode_times=False).pipe(fixTime)

Update: refactor code

[keewis]

the decoding mechanism will take values from the .attrs dictionary, use them to actually do the decoding, and will then move those values to .encoding (while making sure not to overwrite anything). In other words, you should not modify ds.time.encoding in fixTime.

[kmuehlbauer]

OK, I can't test now, but can you try with decode_cf=False instead decode_times=False?

[durack1]

Thanks @keewis, @kmuehlbauer this seems to work:

Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:41:22) [Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import xarray as xr
>>> def fixTime(ds):
...   print("fix wonky time:units")
...   ds.time.attrs["units"] = "days since 2001-01-01"
...   return xr.decode_cf(ds)
... 
>>> ds = xr.open_dataset("sic_O1.nc", decode_times=False).pipe(fixTime)
fix wonky time:units
>>> ds.time.attrs
{'standard_name': 'time', 'long_name': 'time', 'axis': 'T', 'bounds': 'time_bnds'}
>>> ds.time.encoding
{'source': 'sic_O1.nc', 'original_shape': (1200,), 'dtype': dtype('float64'), 'units': 'days since 2001-01-01', 'calendar': '360_day'}

[keewis]

I'd probably do:

ds = (
    xr.open_dataset("sic_O1.nc", decode_times=False)
    .pipe(fixTime)
    .pipe(xr.decode_cf)
)

and remove the decode_cf call from fixTime: that way, should you ever need to do other things before decoding, you'd only have to insert another .pipe(fix_...).

[durack1]

@keewis good suggestion. This would aid fix* reuse, as you note, with a single line (e.g. in this case ds.time.attrs["units"] = "days since 2001-01-01") that could be collated across dataset pathologies