How does kv cache discard is done in llama.cpp?

[DavidZyy]

For example, the capicity of kv-cache is 64 token(n_ctx in examples/main/main.cpp) , when 64 is full, for example, we will discard the kv-cache of 0-31 token. I have seen the codes of rope shift of k-cache (for example, shift the position of 32-63 token to 0-31 token), but I do not see the codes about discard them (for example, set the memory of discard kv-cache to 0, the red area of matrix in the below pictrue means tokens to be discard), and seems the discard tokens still participate in attention operation when doing matrix multiply. Can someone help me?

[ggerganov]

The data remains in the memory buffer and is computed during the attention, but then it is discarded by the attention mask.

[DavidZyy]

Thanks for your reply! More details for someone who have the same question (in my understanding) :
The mask tensor inp->self_kq_mask is created at function llm_graph_context::build_attn_inp_kv_unified().
The data of mask is set at llm_graph_input_attn_kv_unified::set_input. Through the if statement kv_self->cells[i].has_seq_id(seq_id), we can judge the token that has been discarded and set the its corresponding mask data to -INF, than when doing softmax operation, it will be set to 0.

[DavidZyy]

The version of codes I used.