Avoiding automatic sanitization to nanoseconds in conversion to DataArray when dealing with out-of-range timestamps

[leez4rd]

Hi!
I am writing a data processing script for a large database of paleoclimate records, and many of the dates for these records extend deep into the past-- further back than 64-bit floats with nanosecond precision can store.
My current understanding is that xarray is built on top of pandas, and pandas sanitizes datetime64 objects to nanoseconds. I've trudged through date-time hell looking for ways to process the data consistently and robustly (with some success, but far more failure), but the problem I am hung up on now centers around combining a numpy array and a datetime64 array to a DataArray as illustrated in the following snippet of code:

import xarray as xr 
import numpy as np 
from datetime import datetime, timedelta

decyrs = [1260, 1320, 1640, 1980, 2005]  # years AD to convert to datetime64 objects 

dates = np.empty([5],dtype='datetime64[s]')

for i in range(5):
    start = decyrs[i]                   # this is the input (time in decimal years)
    year = int(start)
    rem = start - year
    base = datetime(year, 1, 1)
    result = base + timedelta(seconds=(base.replace(year=base.year + 1) - base).total_seconds() * rem)
    result = result.strftime("%Y-%m-%d")   # truncate to just keep YYYY-MM-DD (get rid of hours, secs, and minutes)
  
    dates[i] = np.datetime64(result).astype('datetime64[s]')


print(dates)

dataset = np.asarray([1, 2, 3, 4, 5])

dataset = xr.DataArray(dataset, coords=[dates], dims=["date"])

print(dataset)

I am sure there is a standard practice for using xarray with out-of-range timestamps, and I've encountered a number of similar discussions on here and StackOverflow revolving around the tangled mess of Python date and time libraries, but none of them were quite what I am looking for. I've thought of a few ways to get around this, but they require jumping through too many hoops for them to be the right way. I figure somebody on here with more experience dealing with dates and times can point me to the quickest solution.

Sorry if this question is vague or ill-posed-- I'm new to this stuff, so let me know if there are any other details I should include. Thanks!

[spencerkclark]

No worries -- indeed dealing with datetimes is complicated. In xarray the standard practice in this situation is to use cftime.datetime objects (see more information in xarray's documentation here). cftime.datetime objects are analogous to standard library datetimes, except they support both standard and non-standard calendars.

Here is how I would use them in your example:

import xarray as xr 
import numpy as np

from cftime import DatetimeProlepticGregorian
from datetime import timedelta

decyrs = [1260, 1320, 1640, 1980, 2005]  # years AD to convert to datetime64 objects 

dates = np.empty([5],dtype="O")

for i in range(5):
    start = decyrs[i]                   # this is the input (time in decimal years)
    year = int(start)
    rem = start - year
    base = DatetimeProlepticGregorian(year, 1, 1)
    dates[i] = base + timedelta(seconds=(base.replace(year=base.year + 1) - base).total_seconds() * rem)

dataset = np.asarray([1, 2, 3, 4, 5])
dataset = xr.DataArray(dataset, coords=[dates], dims=["date"])

dataset then looks like:

<xarray.DataArray (date: 5)>
array([1, 2, 3, 4, 5])
Coordinates:
  * date     (date) object 1260-01-01 00:00:00 ... 2005-01-01 00:00:00

where date is a CFTimeIndex-backed coordinate. As described in the linked section of the documentation, you should be able to do just about everything you can do with datetime64[ns] values in xarray with cftime.datetime objects of any calendar type, and the computations will automatically be calendar-aware.

[leez4rd]

Thank you so much!
I think I'd tried to use cftime here before but had thrown that idea on the backburner because there was a subtle issue in my application of it, which your code snippet helped me fix. Have a good one!