multi-sequential input - single sequence output

[razirezaei]

Hi,
Among the current TSAI models available, is there any that takes in a multi-sequential input and outputs a single sequence?
.
I'm working on a project in which there are several (let's say 13) temporal features from a fixed-term recording (let's say 120 frames) and I would like to leverage these features to predict a single sequence during that fixed-term (13by120 to 1by120). I've seen some models that take multi-seq but they don't turn a seq as the output. Please note that this is neither a classification nor a regression problem.
.
I highly appreciate your reply :)

[oguiza]

Hi @razi2021,
Yes, actually most models can do handle seq-to-seq tasks. The only thing you need to do is to pass the right target. Take a look at this example:

X, y, splits = get_regression_data('Covid3Month', split_data=False)
y_multi= y.reshape(-1,1).repeat(5,1) # this is to simulate sequences of 5 steps 
tfms = [None, TSRegression()]
batch_tfms = TSStandardize(by_sample=True)
dls = get_ts_dls(X, y_multi, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, InceptionTimePlus, metrics=mae, cbs=[ShowGraph()])
learn.fit_one_cycle(10, 1e-2)
preds = learn.get_X_preds(X)[0]
preds.shape

output: torch.Size([201, 5])
Models will automatically try to create predictions with the same shape as the target.
Please, let me know if this works for you.

[joaopalotti]

Hi @oguiza,
Thanks for the amazing package and support.

I have a similar question here.
If we change the problem to a classification instead of a regression, I get an error when using get_ts_dls.

On top of your initial example:

X, y, splits = get_regression_data('Covid3Month', split_data=False)
y_multi= y.round().reshape(-1,1).repeat(5,1) # does not really matter
tfms = [None, TSClassification()]
batch_tfms = TSStandardize(by_sample=True)
dls = get_ts_dls(X, y_multi, splits=splits, tfms=tfms, batch_tfms=batch_tfms)

TypeError: unhashable type: 'numpy.ndarray'

Am I doing something wrong?

Another question that I have is: what should I do if my sequence is not complete? Is there a way to pad sequences in tsai?
For example:

X, y, splits = get_regression_
data('Covid3Month', split_data=False)
y_multi= y.reshape(-1,1).repeat(5,1) # this is to simulate sequences of 5 steps 
y_multi[0][4] = np.nan
tfms = [None, TSRegression()]
batch_tfms = TSStandardize(by_sample=True)
dls = get_ts_dls(X, y_multi, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, InceptionTimePlus, metrics=mae, cbs=[ShowGraph()])
learn.fit_one_cycle(10, 1e-2)

Results in:

epoch | train_loss | valid_loss | mae | time
-- | -- | -- | -- | --
0 | nan | nan | nan | 00:00
1 | nan | nan | nan | 00:00
2 | nan | nan | nan | 00:00

[oguiza]

Hi @joaopalotti,

Thanks for the amazing package and support.

Thank you!

I think the first case is a bit unusual (multiclass, multistep label). I don't think I'd seen any before. In any case, I have updated TSClassification to accept this type of label. It was a simple change. I've updated the repo. So if you can install directly from GitHub using

pip install git+https://github.com/timeseriesAI/tsai.git -Uqq
# ⚠️ REMEMBER TO RESTART (NOT RECONNECT/ RESET) THE KERNEL/ RUNTIME ONCE THE INSTALLATION IS FINISHED

it should work.

For the second case, there is a batch transform called Nan2Value. But it just works on TSTensors (that is the Xs) as you don't want to pass nan values to a model. However, what you raise here is that some ys may also contain nan values.
In this case, you could use a modified loss like this:

class MaskedL1Loss(Module):
    def __init__(self):
        self.loss = nn.L1Loss()

    def forward(self, inp, targ):
        inp = inp.flatten()
        targ = targ.flatten()
        mask = torch.isnan(targ)
        inp, targ = inp[~mask], targ[~mask]
        return self.loss(inp, targ)

as you probably don't want to measure the performance of the model when the label is nan.
I haven't included it in tsai because there are many different variations.

[joaopalotti]

Hi @oguiza,

Thanks for the quick reply!

To contextualize, what I have in mind is a sleep staging problem: a series of heart rate and activity from a set of users and their respective sleep stage. For each timestamp, I have a tuple <person, hr, act, sleep_stage> and the goal is creating a f(hr, act) -> sleep_stage.

One alternative would be using a sliding window to break my long 8 hours of sleep into small pieces. Another alternative is modeling as a seq2seq task. I decided to start by this second one first and that is why I wrote the question above. If you think there is another preprocessing approach that I should try, please let me know.

The example above now generates the dataset, but does not fit correctly. It seems the batch size gets flattened somewhere:

X, _, splits = get_regression_data('Covid3Month', split_data=False)
y_multi = np.random.randint(0, 2, (201, 5), dtype=np.int)

tfms = [None, TSClassification()]
batch_tfms = TSStandardize(by_sample=True)

dls = get_ts_dls(X, y_multi, splits=splits, tfms=tfms, batch_tfms=batch_tfms)

learn = ts_learner(dls, InceptionTimePlus, cbs=[ShowGraph()])
learn.fit_one_cycle(1, 1e-2)


ValueError: Expected input batch_size (64) to match target batch_size (320).

Last, thank you very much for the MaskedL1Loss, good idea! :)

[oguiza]

I've just updated the repo with some updated code that I believe will fix this issue.
It'd be good if you could update tsai using

!pip install git+https://github.com/timeseriesAI/tsai.git -Uqq
# ⚠️ REMEMBER TO RESTART (NOT RECONNECT/ RESET) THE KERNEL/ RUNTIME ONCE THE INSTALLATION IS FINISHED

and test it again.

I have tested it with this code:

X, _, splits = get_regression_data('Covid3Month', split_data=False)
y_multi = np.random.randint(0, 2, (201, 5), dtype=np.int)

tfms = [None, TSClassification()]
batch_tfms = TSStandardize(by_sample=True)

dls = get_ts_dls(X, y_multi, splits=splits, tfms=tfms, batch_tfms=batch_tfms)

learn = ts_learner(dls, InceptionTimePlus, cbs=[ShowGraph()])
learn.fit_one_cycle(1, 1e-2)

and it seems to work well.

[joaopalotti]

Thank you very much for the fast support, Ignacio! It works now. :)