Anyone understand how to use "ghost attention" (GAtt) with llama 2?

[JohanAR]

Chapter 3.3 of "Llama 2: Open Foundation and Fine-Tuned Chat Models" talks about using GAtt to avoid having to repeat instructions in multi-turn dialogue. Is this something that would need to be implemented in llama.cpp, or does it already exist?

[WeirdConstructor]

As far as I understood: Gatt is a special algorithm that was used when LLaMA 2 Chat was fine tuned. The GAtt is not something that requires a change in how inference is done.

[JohanAR]

Ah, ok. I was thinking that there might be something one needed to take into consideration when writing prompts, or choosing which tokens to send to the model.

[darkman111a]

actually to utilize GAtt you must insert <> & <> between any text to modify the behavior of the entire session. Your intution was correct.