How does the llama.cpp do batch processing?

[xyzhang626]

Hi guys,

I'm new to the llama.cpp and ggml, I want to understand how the code does batch processing.

I saw lines like ggml_reshape_3d(ctx0, Kcur, n_embd_head, n_head_kv, n_tokens) in build_llama, where no batch dim is considered. Could you guys help me to understand how the model forward with batch input? That will help me a lot, thanks in advance! @ggerganov

[bullno1]

If you just want to do batch decoding, see https://github.com/ggerganov/llama.cpp/tree/master/examples/parallel