Potential redundant compute

[Connor323]

I wonder why do we need to initialize the cache for every block here? Can we reuse the past_key_values from the previous model inference except the first block loop? Thanks!

[hills-code]

Hi, this is to refresh the cache after each block is decoded. If you reuse the previous past_key_values, the performance may get worse.

[Connor323]

Gotcha! To confirm, even after we refresh the cache, we will NOT update the tokens that have already been unmasked, right? If so, this refresh step is only for the better prediction of the tokens in the current block of interest? Why not updating the unmasked token as well? (for better quality)

Thanks a lot!