Encode bytes on variable attributes

[GrgMdmn]

Hello there ! I am trying to re-encode b' ' byte as an attribute value for some of my variables. The problem is that xarray does not seem to appreciate this, whereas the nc files I am opening are encoding this way (so that proves b' ' bytes are netcdf4 compatible).

Here is the easied way to reproduce the error, from a file that already has a variable attribute value which is a b' ' byte.

IN: xr.open_dataset(nc_path, engine='netcdf4', decode_times=False, decode_cf=False).to_netcdf(wd + '\\output_file.nc')
OUT: TypeError: Invalid value for attr '_FillValue': b' '. For serialization to netCDF files, its value must be of one of the following types: str, Number, ndarray, number, list, tuple

Here is a nc file example here (inside the zip), with a problem at the DC_REFERENCE key, _FillValue attribute : GL_TS_DB_21528.zip

I have tried using the encoding field, by entering this argument :
encoding = {'DC_REFERENCE': {'_FillValue' : np.bytes_(' ')}}

Another attempt to :
encoding = {'DC_REFERENCE': {'_FillValue': {'dtype': 'S1', 'data': b' '} }

For each trial, I then used this command, adding the encoding argument :
xr.open_dataset(nc_path, engine='netcdf4', decode_times=False, decode_cf=False).to_netcdf(wd + '\\output_file.nc')

But I still encounter the same error. Do you know how to proceed ?

[kmuehlbauer]

@GrgMdmn Please have a look at this issue: #7186

This comment by @tovogt shows a possible workaround: #7186 (comment) which might solve your problem.

[GrgMdmn]

Thank you for your answer. Using these two lines did not fix my problem :

nc_file.variables['DC_REFERENCE'].attrs['_FillValue'] = ' '.encoding('utf-8')
encoding = {'DC_REFERENCE': {'_FillValue': ' '.encoding('utf-8') }

[IN] : nc_file.to_netcdf(wd+'\\output_file.nc', encoding=encoding)
[OUT] : TypeError: Invalid value for attr '_FillValue': b' '. For serialization to netCDF files, its value must be of one of the following types: str, Number, ndarray, number, list, tuple

[kmuehlbauer]

@GrgMdmn I meant something along the following:

import xarray as xr
import numpy as np
ds = xr.open_dataset("GL_TS_DB_21528.nc", decode_cf=False)
ds["DC_REFERENCE"].attrs["_FillValue"] = np.array(' '.encode('utf-8'))
ds.to_netcdf("test1.nc")

[GrgMdmn]

Thank you for your reply, I forgot to use the "np.array()" function.

It seems to be working when trying to export to NETCDF4 format.
nc_file.variables[var].attrs['_FillValue']= np.array(' '.encode('utf-8'))
nc_file.to_netcdf(wd+'\\output_file.nc', encoding=encoding, format = "NETCDF4")
By the way, I do not exactly understand why calling np.array() function is indispensable.

However, it does not work when trying to export to NETCDF4_CLASSIC format (whereas the original file IS in NETCDF4_CLASSIC format, and seems to handle the string bytes values):
[IN] : new_nc_file.variables[var].attrs['_FillValue']= np.array(' '.encode('utf-8')) nc_file.to_netcdf(wd+'\\output_file.nc', encoding=encoding, format = "NETCDF4_CLASSIC")
[OUT] : TypeError: sequence item 0: expected str instance, bytes found

If I try specifying encoding conditions, this way, then I have another error :
[IN]: new_nc_file.variables[var].attrs['_FillValue']= np.array(' '.encode('utf-8'))
encoding = { 'DC_REFERENCE' : {'_FillValue': np.array(' '.encode('utf-8'))} }
nc_file.to_netcdf(wd+'\\output_file.nc', encoding=encoding, format = "NETCDF4_CLASSIC")
[OUT]: ValueError: failed to prevent overwriting existing key _FillValue in attrs on variable 'DC_REFERENCE'. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.

So I tried to delete the _FillValue attribute of the DC_REFERENCE variable:
[IN]:del new_nc_file.variables[var].attrs['_FillValue']
new_nc_file.variables[var].attrs['_FillValue']= np.array(' '.encode('utf-8'))
encoding = { 'DC_REFERENCE' : {'_FillValue': np.array(' '.encode('utf-8'))} }
nc_file.to_netcdf(wd+'\\output_file.nc', encoding=encoding, format = "NETCDF4_CLASSIC")
[OUT]: ValueError: failed to prevent overwriting existing key _FillValue in attrs on variable 'DC_REFERENCE'. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.
I am still getting the same error.

[GrgMdmn]

Hello ! I would like your help, in order to solve my problem... I have not found any answer to my problem :(

[GrgMdmn]

I found a workaround : just remove the concerned variable and rewrite it with netcdf4 library, using the "fill_value" argument when creating the variable.

# add the b' ' _FillValue attribute to DC_REFERENCE (and others than can exist)
nc =  netCDF4.Dataset(path, 'r+', format=format)
for var in variables_with_byte_FillValues:    
    # Create a new variable with the same data and attributes, including fill_value
    old_variable = nc_file.variables[var] # nc_file is xarray dataset
    
    for dim in old_variable.dims:
        if dim not in nc.dimensions:
            nc.createDimension(dim, nc_file.dims[dim])
    
    new_variable = nc.createVariable(var, old_variable.dtype, old_variable.dims, fill_value=b' ')
    
    # Copy the data from the old variable to the new variable
    new_variable[:] = old_variable[:]
    
    # Copy the attributes from the old variable to the new variable
    dict_without_fill_value = old_variable.attrs.copy()
    dict_without_fill_value.pop('_FillValue')
    new_variable.setncatts(dict_without_fill_value)

# # Save the changes and close the file
nc.close()

Did not find any solution based on xarray.