Making numpy.datetime64 coordinate variable CF-Compliant - cftime?

[openSourcerer9000]

I'm unable to find any examples on how to write a CF-compliant NetCDF with a time coordinate variable of type np.datetime64. The timesteps were originally as pd.Timestamp's but got automatically converted. I don't really think in terms of nanoseconds elapsed since the epoch or whatever, so could someone please point me to how I can make the conversion painlessly and produce a CF-compliant .nc?
Thanks

[dcherian]

A simple to_netcdf will make sensible choices for you.

See this page for how to precisely control the encoding.

[openSourcerer9000]

@dcherian OK cool. So when running it through CF Compliance Checker I'm getting:
* units for variable time must be convertible to s currently they are hours since 2019-09-02 00:00:00
I guess since its standard_name is 'forecast_reference_time', the units should match the canonical units of seconds.

So I just gave it a units of
nc.time.attrs['units'] = 'seconds since 2019-09-02 00:00:00'
and serializing to netCDF raises this, I think telling me to remove that units attr:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_68396/2439748517.py in <module>
      1 from funkshuns import *
      2 fileBackup(outnc)
----> 3 nc.to_netcdf(outnc.parent/f'{outnc.stem}_.nc')

~\Anaconda3\envs\raz\lib\site-packages\xarray\core\dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
   1900         from ..backends.api import to_netcdf
   1901 
-> 1902         return to_netcdf(
   1903             self,
   1904             path,

~\Anaconda3\envs\raz\lib\site-packages\xarray\backends\api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
   1070         # TODO: allow this work (setting up the file for writing array data)
   1071         # to be parallelized with dask
-> 1072         dump_to_store(
   1073             dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
   1074         )

~\Anaconda3\envs\raz\lib\site-packages\xarray\backends\api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
   1117         variables, attrs = encoder(variables, attrs)
   1118 
-> 1119     store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
   1120 
   1121 

~\Anaconda3\envs\raz\lib\site-packages\xarray\backends\common.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
    259             writer = ArrayWriter()
    260 
--> 261         variables, attributes = self.encode(variables, attributes)
    262 
    263         self.set_attributes(attributes)

~\Anaconda3\envs\raz\lib\site-packages\xarray\backends\common.py in encode(self, variables, attributes)
    348         # All NetCDF files get CF encoded by default, without this attempting
    349         # to write times, for example, would fail.
--> 350         variables, attributes = cf_encoder(variables, attributes)
    351         variables = {k: self.encode_variable(v) for k, v in variables.items()}
    352         attributes = {k: self.encode_attribute(v) for k, v in attributes.items()}

~\Anaconda3\envs\raz\lib\site-packages\xarray\conventions.py in cf_encoder(variables, attributes)
    853     _update_bounds_encoding(variables)
    854 
--> 855     new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}
    856 
    857     # Remove attrs from bounds variables (issue #2921)

~\Anaconda3\envs\raz\lib\site-packages\xarray\conventions.py in <dictcomp>(.0)
    853     _update_bounds_encoding(variables)
    854 
--> 855     new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}
    856 
    857     # Remove attrs from bounds variables (issue #2921)

~\Anaconda3\envs\raz\lib\site-packages\xarray\conventions.py in encode_cf_variable(var, needs_copy, name)
    267         variables.UnsignedIntegerCoder(),
    268     ]:
--> 269         var = coder.encode(var, name=name)
    270 
    271     # TODO(shoyer): convert all of these to use coders, too:

~\Anaconda3\envs\raz\lib\site-packages\xarray\coding\times.py in encode(self, variable, name)
    513                 data, encoding.pop("units", None), encoding.pop("calendar", None)
    514             )
--> 515             safe_setitem(attrs, "units", units, name=name)
    516             safe_setitem(attrs, "calendar", calendar, name=name)
    517 

~\Anaconda3\envs\raz\lib\site-packages\xarray\coding\variables.py in safe_setitem(dest, key, value, name)
    110     if key in dest:
    111         var_str = f" on variable {name!r}" if name else ""
--> 112         raise ValueError(
    113             "failed to prevent overwriting existing key {} in attrs{}. "
    114             "This is probably an encoding field used by xarray to describe "

ValueError: failed to prevent overwriting existing key units in attrs on variable 'time'. This is probably an encoding field used by xarray to describe how a variable is serialized. To proceed, remove this key from the variable's attributes manually.

[dcherian]

Use nc.time.encoding['units'] = 'seconds since 2019-09-02 00:00:00' otherwise it clashes. That error could be improved...

[openSourcerer9000]

OK, now I've seen it all
* units for variable time must be convertible to s currently they are seconds since 2019-09-02
This seems more of an issue with compliance checker though.

[DocOtak]

You are correct in that the compliance checker seems to not be doing the right things, "hours since 2019-09-02 00:00:00" should be valid CF. There are 4 examples in the latest CF document that have "hours since ..." units.

You do not need to use the exact "canonical unit" for CF compliance, things just need to be "convertible to" unambiguously. e.g. hours to seconds is fine because you can reference a definition, but liters to kilograms is not ok because you need to do density calculations/conversions. I've been mulling over proposing to CF that the "canonical units" be changed to something like "physical dimensions" and you have things like T for time, Θ for temperature etc... But I cannot convince myself that won't cause other confusion. CF tries to be descriptive rather than prescriptive, you shouldn't need to change the data you have, just attach the right attributes, though time is a funny case due to limitations of the historic netCDF format. I usually treat needing to change my data values as a code/data smell, that maybe I haven't understood something about CF, don't have the right attributes yet, or the compliance checker is going crazy, etc...