Spatial and temporal attention.

[llvm-x86]

Do the transformers find relationships between features and time?

Ie:
(_, 0, 13) finds a relationship with (_, 5, 9)?

[oguiza]

That depends on the Transformer.
Some of the models in the tsai library (like TST or TSiT) use attention between time steps (all features together). Others (like PatchTST) find relationships between patches of time steps for each feature.
None of the models currently available uses both spatial and temporal attention.

[llvm-x86]

How would I make one?

[oguiza]

Hi @shusei-kagari,
This is currently out of the scope of tsai. This would require a considerable amount of changes. You'd need to build a dataset, dataloader, and select an architecture. I'd probably start reviewing the literature to identify an architecture and see if they have a github repo where it's implemented.