What does the update_cache function in MultiHeadAttention do?

[fedorovgv]

I am currently working with the MultiHeadAttention class and found the update_cache function. As far as I understand it does nothing at this moment and is a template for the future, am I right? and if it's true, can you explain what this function will do?

https://github.com/NVIDIA/NeMo/blob/9f94649b9111b7d20fb3770b76ccccc2a2633b1f/nemo/collections/asr/parts/submodules/multi_head_attention.py#L154-L165

[titu1994]

It's used in chunk aware transformers. cc @VahidooX

[VahidooX]

The caching is being used during the inference when cache-aware streaming Conformer is being used. During the training, it is skipped.

[fedorovgv]

Thanks a lot!