The Phenomenon of Latency Jumps in Inference

[UltraTar0]

When I used vLLM to accelerate inference for Llama3 70b, I observed an interesting phenomenon. When each input request consists of 1 token, there is a latency jump every 256 requests. Does anyone know why this happens?

[SeungminHeo]

because default scheduler batch size is 256 so when reached 256 requests and nothing completed, additional requests will be Waited by scheduler within vLLM. so waiting time is added to latency, that is the phenomenon you observed i guess.

[UltraTar0]

Thank you！