Coercing datetime types?

[fabianoliver]

Hi,

I'm trying to use pandera to simplify parsing text-based tables (from gherkin based tests) into schema-based dataframes. My initial hope was that I can just define an appropriate schema with coerce=True fields, and let pandera handle most of the work.

However, it doesn't seem to be quite as easy with date/datetime-like types. For example,

import pandera as pa
from pandera.typing import DataFrame, Series, DateTime

class TestSchema(pa.SchemaModel):
    date: Series[DateTime] = pa.Field(coerce=True)

df = DataFrame[TestSchema]({'date': ['2022-02-28']})

While the df is created and passes validation, its 'date' column is of dtype "object", and has string values.

Is there any way for me to specify a schema that coerces the value into an actual date(time)-like dtype?

Also somewhat oddly, manual conversion seems to have slightly funny effects. Say I afterwards do

df.date = df.date.astype('datetime64[ns]')
df.date['date'][0] # returns a Timestamp('2022-02-28 00:00:00')
df.loc[0,'date'] # returns a string '2022-02-28'

Thank you

[jeffzi]

Hi @fabianoliver

The syntax DataFrame[TestSchema] is intended to help mypy linting but you still define a "regular" pandas DataFrame (notice that you import from pandera.typing). The only additions are type hints and support for pydantic.

You need to call validate to coerce: TestSchema.validate(df). The reason why we cannot implement the behavior you expected is that Pandera would need to override every DataFrame method that outputs a DataFrame to call validate before return.

re: funny effects. Line 2 of your example does not work, I think you meant df.date[0]. It's expected to have Timestamp type in a datetime64[ns] series, the documentation says

Timestamp is the pandas equivalent of python’s Datetime and is interchangeable with it in most cases. It’s the type used for the entries that make up a DatetimeIndex, and other timeseries oriented data structures in pandas.

[fabianoliver]

Perfect, thank you! I didn't realise typing.DataFrame strictly needs a pd.DataFrame. (And yes, you're absolutely right - line 2 was my typo, that should have been df.date[0] indeed)

Going through a regular DF first, followed by a call to validate works very well!

Thanks again