What's the use of llama_eval function in example/main.cpp?

[DavdGao]

I'm reading source code in example/main.cpp and curious about the function llama_eval. It seems like llama_eval will take a long time to deal with the prompt. My question maybe stupid, but I wonder what's the use of this function?

[Z-KN]

Just does the inference.

[DavdGao]

yes， but I wonder why we need to run llama_eval token by token. To obtain kv cache for attention layers, we only need to calculate the inference for once with all tokens, right? Is there anything I missed?

[Z-KN]

From my understanding, LLMs all produce one token after another, right? Cached KV are used for computing next token.

[DavdGao]

Sure, but it's to generate new tokens. When processing the tokens in prompt, in my opinion we shouldn't run llama_eval for each token.
However, I notice that examples/main.cpp in previous commits processes prompt by each token.
By the way, llama_eval is renamed into llama_decode in current head commit.

[aehlke]

it's how you load the tokens in so that it has context for the next token...

[rikoras]

Most of the computations are contained in this function, including a series of matrix multiplications, normalizations, and other operators.

[DavdGao]

Thanks, but I wonder why we need to run llama_eval function for each token in the prompt. To obtain kv cache for attention layers, we only need to calculate the inference for once with all tokens, right? Anything I missed?

[rikoras]

You may need to take a look at ‘autoregressive’. In a single turn, only one token is generated (you can confirm this by checking the shape of the model output). Assuming there are N tokens in the prompt, we cannot predict token N+2 until we get token N+1, as the forward attention calculation for token N+2 depends on all previous N+1 tokens.

[rikoras]

And for the initial prompt, all of the previous tokens are handled in a single eval.