Managing Performance (token/s)

[gotDaijobu]

Hi,

I'm using vllm for inference. I gave it several tries and I'm currently running a Llama3.1 70B quantized model (bitsandbytes, 4bits) on a single A100 (80GB).

I made quite a lot of tests and I realized that decreasing (or increasing) the gpu_memory_utilization has no effect on the throughput (in tokens/seconds, i'm stuck at 13tps).

So the bottleneck is elsewhere but I can't figure where.

Any idea ?

[dhruvmullick]

It's probably memory bandwidth bound.

Have you tried increasing the request concurrency?

[gotDaijobu]

Meaning --max-num-seqs ? Yes. But in this case, i'm trying to raise the throughput for a single request.

[gotDaijobu]

So I've done further tests : when I have several requests in parallel, the performances drops. I am struggling to find the cause.