regression & forecasting fine-tuning

[geoHeil]

I have a dataset like this:

and for each multivariate panel time-series I want to forecast n steps ahead for the target variable baed on many features (the example here only shows a single one) as well as the history of the target variable itself. How do I need to define the y_func?

So far I am simply using the sliding window panel fucntion with a specific forecast horizon

X,y = SlidingWindowPanel(window_len=216, 
                         unique_id_cols=['fnid'], 
                         stride=24, 
                         start=0, 
                         pad_remainder=False, 
                         padding_value=np.nan,
                         add_padding_feature=True,
                         get_x=['target', 'value_1'],
                         get_y='target',
                         y_func=None,
                         horizon=24,
                         seq_first=True,
                         sort_by=['time'],
                         ascending=True,
                         check_leakage=True,
                         return_key=False,
                         verbose=True
                        )(df)

Is this the correct way? Or do I need to perform any future considerations?
What I mean is: #36 (comment) I am unsure about the get_y if this is used correctly here as I want to use many dimensions of a multivariate time series to forecast a single dimension.

However, so far the results seem to be rather not well forecastable at all - the RMSE converges to 2.7 after a very short while of training.

[oguiza]

Hi,
Yes, as far as I can see, that'd be the correct way to do it (if you want to forecast the next 24 timesteps in the sequence).
I'd like to suggest a way I often use to check whether my settings are correct. I normally create some dummy data that reflects the structure of the data I'm interested in and perform the operation to check the data is correct. I've created a simple gist.
BTW, I've also demonstrated a way to apply SlidingWindow on Panel data that I've found to be much faster that SlidingWindowPanel on some data.

output = [df.groupby(['unique_id']).apply(lambda x: SlidingWindow(window_len=5,
                                                                  stride=5,
                                                                  start=0,
                                                                  pad_remainder=False,
                                                                  padding_value=np.nan,
                                                                  add_padding_feature=True,
                                                                  get_x=['var1', 'var2', 'target'],
                                                                  get_y='target',
                                                                  y_func=None,
                                                                  horizon=3,
                                                                  seq_first=True,
                                                                  sort_by=['time'],
                                                                  ascending=True,
                                                                  check_leakage=True,
                                                                  copy=True, # it's important to set copy to True when used in this way!!!
                                                                  )(x))][0].values

X = np.concatenate([oi[0] for oi in output])
y = np.concatenate([oi[1] for oi in output])

As to the training, there are multiple reasons why you are not getting the expected results. Are you scaling the data by variable using TSStandardize(by_var=True)? I've noticed the target and variable are in very different scales, which might have a large impact on the RMSE.

[oguiza]

The model is clearly overfitting, so you can use any of the techniques to try and reduce it. Jeremy Howard recommends:

1. Use more data
2. Use data augmentation
3. More generalizable architecture
4. Regularization: dropout, weight decay
5. Reduce architecture complexity

The model seems to train well for the first 10-20 epochs. Then it starts overfitting. I'd set the number of epochs to 20 (max) to reduce training time and would use 1 (if you have more data) or 2 (you can pass CutMix1d() as a callback - it works well on many occasions).

In terms of architectures, I'm not sure which one you are using. I'd test at least MiniRocket, InceptionTime and TST.

You can also add dropout (passing fc_dropout=.5 for example to any of the *Plus models in tsai.

Lastly, you can tune the model hyperparameters and reduce it's size.

[geoHeil]

When following your suggestions:

current_learner_model = create_model(InceptionTimePlus, dls=dls, **{'fc_dropout':0.5})
learn = Learner(dls, current_learner_model,  metrics=[mae, rmse], cbs=[ShowGraph(), CutMix1D()])
learn.fit_one_cycle(30, 1e-3)

the overfitting is certainly reduced. Notice: I furthermore have reduced the forecasting horizion from 7 to 1 to make the task simpler and also switched to a randomized train-test split for the sliding windows.
I guess this is moving a little bit into the right direction.

[geoHeil]

Interestingly now after restarting the notebook I start to face: #136 and cannot find any reason for it.

[oguiza]

Are you sure the learner object was saved after the issue was fixed? That's strange. To delete the learner or recreate it shouldn't have any impact on the final learner object.

[meanpenguin]

Remember that the first time it works. It’s the second time and on that it doesn’t. You probably have not noticed it if you are not looking for it. :-).