to_netcdf encoding problem

[ccoulet]

Hi
I'm stuck on a problem and doesn't find if I made a mistake or if there is a bug somewhere in xarray.to_netcdf
I come to a very basic operation, starting from a netcdf-ugrid-file.
just read and write it without any change leads to an error...

import xarray as xr
ds = xr.open_dataset('test.nc')
ds
<xarray.Dataset>
Dimensions:           (time: 133, single: 1, nele: 114981, nvertex: 3,
                       node: 60340)
Coordinates:
  * time              (time) datetime64[ns] 1928-09-12T06:00:00 ... 1928-09-1...
    X                 (node) float32 ...
    Y                 (node) float32 ...
Dimensions without coordinates: single, nele, nvertex, node
Data variables:
    mesh_topology     (single) int32 ...
    element           (nele, nvertex) int32 ...
    var1        (node) float32 ...
    var2        (node) float32 ...
    var3        (time, node) float32 ...
    var4        (time, node) float32 ...
    var5        (time, node) float32 ...
    var6        (time, node) float32 ...
    var7        (node) float32 ...
    var8        (node) float32 ...
    var9        (node) float32 ...
    var10      (node) float32 ...
Attributes:
    Title:        xxx
    Institution:  xxx
    Source:      xxx
    References:   xxx
    Conventions:  CF-1.6, UGRID-1.0
    Description:  No description has been provided 
ds.encoding
{'unlimited_dims': {'time'}, 'source': 'test.nc'}
ds['var3'].encoding
{'zlib': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': False, 'chunksizes': (1, 60340), 'source': 'test.nc', 'original_shape': (133, 60340), 'dtype': dtype('float32'), '_FillValue': -9999.0, 'scale_factor': 1.0, 'coordinates': 'X Y'}

ds.to_netcdf('out.nc', format='NETCDF4', encoding = ds.encoding)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\coulet\AppData\Local\Programs\Python\Python39\lib\site-packages\xarray\core\dataset.py", line 1901, in to_netcdf
    return to_netcdf(
  File "C:\Users\coulet\AppData\Local\Programs\Python\Python39\lib\site-packages\xarray\backends\api.py", line 1072, in to_netcdf
    dump_to_store(
  File "C:\Users\coulet\AppData\Local\Programs\Python\Python39\lib\site-packages\xarray\backends\api.py", line 1113, in dump_to_store
    variables[k].encoding = enc
KeyError: 'unlimited_dims'

There is no error if I remove encoding=ds.encoding in the last line

Any idea what's going wrong with this option?

[kmuehlbauer]

If you specify encoding kwarg you need specify a dictionary which contains the names of your variables as keys (citing from the docstring):

encoding – Nested dictionary with variable names as keys and dictionaries of variable specific encodings as values, e.g., {"my_variable": {"dtype": "int16", "scale_factor": 0.1, "zlib": True}, ...}

See to_netcdf docstring.

[ccoulet]

Thanks for your answer.
I understand the way it works when I create a netcdf file from scratch but in this case I just read it!
So why the encoding is not compliant with this rules?

[kmuehlbauer]

AFAICT this is all good.

The encoding-kwarg in to_netcdf, IIUC, is for overriding variable encoding or to add it back for specific variables after it got stripped during processing. The unlimited_dims-kwarg in to_netcdf is for specifying unlimited dimensions.

Citing from the doc-string:

unlimited_dims (iterable of hashable, optional) – Dimension(s) that should be serialized as unlimited dimensions. By default, no dimensions are treated as unlimited dimensions. Note that unlimited_dims may also be set via dataset.encoding["unlimited_dims"].

So it seems your ds has everything in the right places for writing out to netcdf

[ccoulet]

Thanks for the clarification

[dcherian]

ds.encoding is not what's expected in encoding. It wants a dictionary with {variable_name: variable_encoding_dictionary}

Also since the encoding you want is present on the dataset and variables already, you need not specify it.