[core] Direct Forecasts - Incorrect Treatment of Exogenous Variables

[syrop87]

What happened + What you expected to happen

Hi,

I am a new user of MLForecast and I am highly impressed with its capabilities! However, I discovered a problematic behaviour when it comes to direct forecaasting - while it is not technically a bug, it can lead to the results are hard to justify.

Currently, the user can pass exogenous variables via X_df argument to predict method, both for recursive forecasts (one model per all horizons) and direct ones (each horizon uses separate model). However, in direct approach not all relevant entries in X_df seem to be used - even if we set horizon at 50, predict method will use only the X_df values corresponding to horizon == 1. That is problematic on many levels, e.g. if we want to use seasonality variables based on Fourier transform (so the value of sin / cos at horizon 50 should be set based on the corresponding date and not the date observed at horizon 1).

I attach the script showing the problem - the forecasts do not change when replacing meaningful values in X_df with zeros.

I realize you may not agree this is a bug (as TimeSeries.predict code includes a comment mentioning that behaviour). If so, please change the Issue tag.

Versions / Dependencies

mlforecast==1.0.2
statsforecast==2.0.1
lightgbm==4.6.0

Reproduction script

import random
import lightgbm as lgb
from mlforecast import MLForecast
from mlforecast.utils import generate_series
from utilsforecast.feature_engineering import fourier

# Forecast Horizon
H = 3

# Adding a static feature, as I don't know how to fit the model with dynamic features only (here sin & cos)
df = generate_series(3, freq="W", min_length=104, n_static_features=1)
df, _ = fourier(df, freq="W", season_length=52, k=1, h=H)

# Train and Test Split
ids = df["unique_id"].unique()
random.seed(0)
sample_ids = random.choices(ids, k=4)
sample_df = df[df["unique_id"].isin(sample_ids)]
test = sample_df.groupby("unique_id").tail(H)
train = sample_df.drop(test.index)

test_X_df = test.copy().drop(columns=["static_0", "y"])

# Replacing all entries of sin & cos in X_df with 0 where horizon > 1.
test_X_df_zeros = test_X_df.copy()
test_X_df_zeros["is_first_ds"] = test_X_df_zeros.groupby(["unique_id"])["ds"].transform(lambda x: x == x.min())
test_X_df_zeros.loc[~test_X_df_zeros["is_first_ds"], ["sin1_52", "cos1_52"]] = 0
test_X_df_zeros = test_X_df_zeros.drop(columns="is_first_ds")

fcst = MLForecast(
    models=lgb.LGBMRegressor(n_jobs=1, random_state=0, verbosity=-1),
    freq="W",
    lags=[1],
    date_features=["month"],
    num_threads=2,
)

# Direct forecasts as max_horizon is supplied to fit method
individual_fcst = fcst.fit(train, static_features=["static_0"], max_horizon=H)
individual_preds = individual_fcst.predict(h=H, X_df=test_X_df)
individual_preds_zeros = individual_fcst.predict(h=H, X_df=test_X_df_zeros)

# Forecasts on original test data and the one with zeros
# Results are the same!
individual_preds_all = individual_preds_zeros.merge(
    individual_preds, how="outer", on=["unique_id", "ds"], suffixes=("_zero", "_base")
)
print(individual_preds_all)
print(any((individual_preds_all["LGBMRegressor_zero"] - individual_preds_all["LGBMRegressor_base"]) != 0))

Issue Severity

Significant, stopping us from using Nixtla.

[jmoralez]

Hey. The predict follows the way the models are trained. During training all models use the same features but are trained to predict one step ahead, two steps, etc., so in predict they all use the features at the first timestamp to predict their respective horizons.

[syrop87]

Still, correct me if I am wrong, now there is no consistency between direct and recursive treatment of exogenous variables (in recursive, the values of exogenous series at date corresponding to horizon H will be used).

The problem becomes more evident if we use the data from the example on exogenous features - https://nixtlaverse.nixtla.io/mlforecast/docs/how-to-guides/exogenous_features.html. Imagine we use price as a regressor and want to create direct multihorizon forecasts conditional on sume future price assumptions. If forecast starting point is t, you can only change the future price assumptions about t+1. Changes in t+2, t+3, ... t+H won't matter for the forecast.

Is there maybe a plan to change the current solution (both train and predict part) - so that exgoenous variables at dates relevant to the specific horizons are taken? We would like to adopt Nixtla in a large retail series forecasting project and that 'bug' comes as a major obstacle.

[ischurov]

I agree with @syrop87. This bug makes mlforecast essentially useless if one wants to predict multi steps ahead (like, predict daily revenue in a future month, so max_horizon=30) with 'direct' strategy and efficiently use exogenous features. We spent some time trying to figure out why the time-varying feature that seemed to be highly correlated with the target variable was practically useless for our model before we realized that it is due to the fact that the X part of the training dataset generated by mlforecast is the same for all target lags, and thus exogenous features become misaligned. This becomes obvious after the fact, but initially it was highly counter-intuitive. The extreme example of this effect is that if I put a copy of target variable as one of my exogenous covariates, the model can perform poorly. This clearly contradicts my mental model: if ML system has access to correct answers, it should work nearly perfectly. Otherwise, it means that the system does not benefit from the information it is being provided, that seem to be a clear bug.

[jmoralez]

Hey. Yeah, the current implementation is wrong. I'll work on it when I have time.

[Arnechos]

@jmoralez Any updates on this? It kinda breakes the whole functionality

[JLiekenbrock]

@jmoralez I added a PR here for using future parts of X_df in the predict: f80e66c