Autoregressive decoding with caching

[marton-avrios]

Is caching implemented with autoregressive decoding? I mean something like in the "Transformer are RNNs" paper (and also in the huggingface library for T5). I thought the "incremental" mode in the context serves this purpose but not sure. HF T5 in pytorch is 3-4x faster for decoding then an exported saved model which makes me wonder.