Recommended setting for running vLLM for CPU

[jerin-scalers-ai]

What are the recommended settings for running vLLM on a CPU to achieve high performance? For instance, if I have a dual-socket server with 96 cores per socket, how many cores (--cpuset-cpus) should be allocated to run multiple replicas of vLLM?

[akhilreddy0703]

48 cores per instance would do fine, It's performing with almost 10 t/s throughput for single user.
Echoswift is a performance benchmark tool for self hosted LLMs, currently supports TGI,vLLM,Llamacpp and Ollama
It's very useful to perform comparative tests to find out the best container size based on the latency and throughput.