How to reason about batch size in batched benchmark?

[Sunt-ing]

Although I just contributed the batched benchmark, I am confused about the batch size in the batched benchmark. I don't know the relationship between these parameters.

In the doc (https://github.com/ggerganov/llama.cpp/blob/master/examples/batched-bench/README.md), I found the following words, but the calculation does not include n_batch (batch size) and n_ubatch (physical batch size).

There are 2 modes of operation:

    prompt not shared - each batch has a separate prompt of size PP (i.e. N_KV = B*(PP + TG))
    prompt is shared - there is a common prompt of size PP used by all batches (i.e. N_KV = PP + B*TG)

Thanks for your help!

[ggerganov]

The word "batch" is not used correctly here. It should be:

There are 2 modes of operation:

    prompt not shared - each parallel sequence has a separate prompt of size PP (i.e. N_KV = N_PL*(PP + TG))
    prompt is shared - there is a common prompt of size PP used by all parallel sequences (i.e. N_KV = PP + N_PL*TG)

Here N_PL is the number of parallel sequences being decoded.
Does that help?

[Sunt-ing]

Yes, it really helps!

And what's the relationship between N_PL (or N_KV), n_batch, and n_ubatch?