Use narwhals for fivetran data adaptor logic

[williambdean]

The fivetran adaptors can be easy use case for narwhals and opens support to polars, pyspark, etc

Related to #1691

[juanitorduz]

yes! 100%!

[williambdean]

Some areas that might need some thought:

• Case where the pivot table in pandas has multi-index columns

pymc-marketing/pymc_marketing/data/fivetran.py

Lines 155 to 159 in 6c7bcae

	if isinstance(wide.columns, pd.MultiIndex):
	flat_cols = [f"{platform}_{metric}" for metric, platform in wide.columns]
	else:
	metric = value_columns[0]
	flat_cols = [f"{platform}_{metric}" for platform in wide.columns]

• normalize

pymc-marketing/pymc_marketing/data/fivetran.py

Line 221 in 6c7bcae

wide[first_col] = pd.to_datetime(wide[first_col]).dt.normalize()

pymc-marketing/pymc_marketing/data/fivetran.py

Lines 344 to 345 in 6c7bcae

	# 3) Daily bucket normalized to midnight, preserving datetime64[ns] dtype
	tmp["_date"] = tmp[date_col].dt.normalize()

That might be this : https://narwhals-dev.github.io/narwhals/api-reference/expr_dt/#narwhals.expr.ExprDateTimeNamespace.truncate

• reindex

pymc-marketing/pymc_marketing/data/fivetran.py

Lines 195 to 199 in 6c7bcae

	if include_missing_dates:
	full_index = pd.date_range(
	start=wide.index.min(), end=wide.index.max(), freq=freq
	)
	wide = wide.reindex(full_index)

[FBruzzesi]

Hey @williambdean 👋🏼 let me try to help :)

Case where the pivot table in pandas has multi-index columns

In this case we already take care of flattening multi-index to match polars output (https://github.com/narwhals-dev/narwhals/blob/28693491164a6231d96cca9ec1cf34d5b5385e00/narwhals/_pandas_like/dataframe.py#L1004)

normalize

dt.truncate(every="d") might do it

reindex

! Disclaimer: I never used reindex

We support nw.maybe_set_index but that's not always the same as .reindex, so it might require some special pathing if you want it as an index.

One additional thing to think about when heavily working with indices is "how should the index be represented for another dataframe backend?". As polars and pyarrow are not index based, where should we store these values? I guess the two options are either in a column or a series. The risk is to have a different schemas at a certain point down the line

[williambdean]

Thank you @FBruzzesi for that info. Don't think we leverage index to full extent but only use for that reindex operation. So working around should be possible

Maybe you can confirm or deny if this would be helper operation to recreate:

https://stackoverflow.com/questions/74728823/how-to-perform-pandas-reindex-in-polars

[FBruzzesi]

Hey @williambdean - again I think I never used reindex in practice, but the SO thread seems to design a fairly easy workaround to it, with operations which are supported in Narwhals since long time, namely: Expr.unique and {DataFrame, LazyFrame}.join with how={"cross", "full"}.