Intel Xeon CPU Max 9480 (OpenVINO) performance?

[randomqhacker]

Getting less than 4 tok/s on a 78B Qwen2.5 finetune quantized to ~43G on 1 x Intel Xeon CPU Max 9480 (HBM only) with OpenVINO backend. pcm-memory confirms that HBM throughput reaches ~170GB/s. My understanding is the practical HBM read bandwidth limit on these should be closer to 575GB/s.

Also tested Qwen2.5 7B int8, and got ~21 tok/s at ~155GB/s HBM throughput.

Is this typical throughput with vLLM + OpenVINO on Xeon CPU Max? Or Xeon in general? Any tips?