Compression & to_netcdf

[snowman2]

What is the best practice when compressing a netCDF file when calling to_netcdf?

From: https://stackoverflow.com/questions/40766037/specify-encoding-compression-for-many-variables-in-xarray-dataset-when-write-to

comp = dict(zlib=True, complevel=5)
encoding = {var: comp for var in ds.data_vars}
ds.to_netcdf(filename, encoding=encoding)

However, it seems like it misses some information from the encoding that way. For example _FillValue seems to get lost.

And with:

comp = dict(zlib=True, complevel=5)
for var in ds.data_vars: 
    var.encoding.update(comp)
ds.to_netcdf(filename)

I end up with:

/usr/local/lib/python3.7/site-packages/xarray/core/dataset.py:1910: in to_netcdf
    invalid_netcdf=invalid_netcdf,
/usr/local/lib/python3.7/site-packages/xarray/backends/api.py:1078: in to_netcdf
    dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
/usr/local/lib/python3.7/site-packages/xarray/backends/api.py:1124: in dump_to_store
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
/usr/local/lib/python3.7/site-packages/xarray/backends/common.py:267: in store
    variables, check_encoding_set, writer, unlimited_dims=unlimited_dims
/usr/local/lib/python3.7/site-packages/xarray/backends/common.py:305: in set_variables
    name, v, check, unlimited_dims=unlimited_dims
/usr/local/lib/python3.7/site-packages/xarray/backends/netCDF4_.py:498: in prepare_variable
    fill_value=fill_value,
src/netCDF4/_netCDF4.pyx:2771: in netCDF4._netCDF4.Dataset.createVariable
    ???
src/netCDF4/_netCDF4.pyx:3872: in netCDF4._netCDF4.Variable.__init__
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: NetCDF: Invalid argument
src/netCDF4/_netCDF4.pyx:1950: RuntimeError

However, this works:

encoding = {}
encoding_keys = ("_FillValue", "dtype", "scale_factor", "add_offset", "grid_mapping")
for data_var in ds.data_vars:
    encoding[data_var] = {key: value for key, value in ds[data_var].encoding.items() if key in encoding_keys}
    encoding[data_var] .update(zlib=True, complevel=5)

ds.to_netcdf(filename, encoding=encoding)

Related: #6929

[jmcvey3]

Had this same issue, and searching for answers led me to this discussion, as well as the site that explains the encoding options for netcdf4: https://unidata.github.io/netcdf4-python/#Dataset.createVariable.
The original encoding that I copied contained contiguous=True and chunksizes=None. Dropping these two from the encoding dict resolved the error, though I don't have an explanation for why.

[arkanoid87]

stepped on this, too

encoding = {var: {zlib": "true"} for var in dataset}
dataset.to_netcdf("/tmp/foo.nc", format="NETCDF4", encoding=encoding)

I've enabled compression, and as a result I've lost some unrelated attributes on variables (CF spatial reference in my context).

this is confusing and should be documented

EDIT:

this worked for me

  for var in dataset: 
      dataset[var].encoding.update(dict(zlib=True, complevel=5))
  dataset.to_netcdf(temp_file_path, format="NETCDF4")

RE-EDIT:

it failed again with different set of attributes, in paritcular when "_FillValue" is set, it fails with weird error:

IndexError: index 65535 is out of bounds for axis 0 with size 2460

where 65535 is _FillValue, and 2460 is axis 0 shape (y: 2460x: 2550). I have no idea why encoding is mixing attributes with shapes (?!)

RE-RE-EDIT:

it was due to DataArray vs Dataset. This worked for me

  comp = dict(zlib=True, complevel=5)
  if type(data) is xr.Dataset:
      for var in data: 
          data[var].encoding.update(comp)
  if type(data) is xr.DataArray:
      data.encoding.update(comp)

  data.to_netcdf(temp_file_path, format="NETCDF4")

[keewis]

it would probably be worth noting in the documentation that the encoding parameter of to_* currently overrides the existing encoding of variables:

xarray/xarray/backends/api.py

Line 1272 in 2ef82c5

variables[k].encoding = enc

so it's not really surprising that the existing encoding is lost.

xref #5954

[arkanoid87]

But it seems to mess with attrs

I've never handled encoding field manually before this attemp, just attrs. Are encoding == attrs?

[keewis]

encoding are attrs that xarray used to decode a variable (like the compression parameters, or the _FillValue). After the decoding is done, the attrs that were used are moved to encoding. When encoding before writing to disk, the inverse happens: the values from encoding are used to encode the variable and are then moved to attrs.

You can see this in action if you pass decode_cf=False to open_dataset and then later call xr.decode_cf(ds).

[arkanoid87]

Thanks for the clarification.