Cannot provide _FillValue=None along with chunks for encoding settings in to_zarr()

[mliukis]

I ran across the situation when xarray is raising an exception if chunks is provided along with _FillValue=None for encoding of the data when writing to the Zarr store. It works perfectly fine if only one of _FillValue or chunks is provided though. Is it a bug?

The code below

import xarray as xr
import numpy as np
import os
import shutil

x = np.arange(0, 10)

ds = xr.Dataset(
    coords = {
        'x': ('x', x, {'name': 'x_coordinate'})
    }
)

a = xr.DataArray(
    data=np.zeros((len(x))),
    coords=[x],
    dims=['x'],
    attrs={'stardard_name': 'test_data'}
)
ds['a'] = a

output_file = 'test_a.zarr'

if os.path.exists(output_file):
    shutil.rmtree(output_file)

encoding_vals = {'a': {'dtype': 'float', 'chunks': 5, '_FillValue': None}}

# Write to Zarr
backend = ds.to_zarr(output_file, mode='w', encoding=encoding_vals, consolidated=True)

# Read back in
with xr.open_zarr(output_file, consolidated=True) as ds_from_zarr:
    ds_from_zarr.a.encoding

raises an exception:
ValueError: unexpected encoding parameters for zarr backend: ['_FillValue']

[keewis]

apologies for the pretty late reply.

I don't think this is a bug (even though the error message could certainly be clearer): the reason this fails is because None is not a valid fill value. Change _FillValue to e.g. -999 (or whatever makes sense for your dataset) and the write should succeed.

[mliukis]

@keewis Thank you for your response! We use None to indicate that _FillValue is not applicable to the data set - in a case when dataset must have values provided for all dimensions. It works just fine if we don't chunk the data on output.

[keewis]

I don't know enough about zarr to be able to tell whether (and how) this is possible, but the zarr v2 spec says this about _FillValue == null with chunks:

An uninitialized chunk MUST be treated as if it was uniformly filled with the value of the “fill_value” field in the array metadata. If the “fill_value” field is null then the contents of the chunk are undefined.

so if I understand that correctly, it means that if you have chunks you must have a valid fill value that can be used to replace uninitialized chunks (because apparently the python implementation of zarr chose to raise an error instead of arbitrarily deciding what "undefined chunk contents" means).

Edit: maybe you can use _FillValue == "NaN" instead?

Let me ping a few people who know more about zarr than I do: @rabernat, @jhamman, @shoyer, do you have any advice?

[rabernat]

This certainly feels like a bug. Changing the value of _FillValue won't change anything. Only these variables are allowed as valid encoding for Zarr.

xarray/xarray/backends/zarr.py

Lines 233 to 239 in 17933e7

	valid_encodings = {
	"chunks",
	"compressor",
	"filters",
	"cache_metadata",
	"write_empty_chunks",
	}

The way the Zarr backend handles fill value has always been confusing, because NetCDF's concept of fill value is slightly different from Zarr's. This whole thing could use a good refactoring.

@keewis - we could use more maintainers on this part of the code. Is this something you'd be interested in looking in to?

[keewis]

I'd be interested, yes. However, I have a long list of things I want to do and way less time (and energy!) than when I started out, so I can't really tell when that will happen. In general, though, I imagine we'd do that as part of the custom write backend refactor.