Skip to content

Update vLLM Benchmark Suite for Xeon based on 0.9.2 release #21486

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Update vLLM Benchmark Suite for Xeon based on 0.9.2 release #21486

wants to merge 2 commits into from
+439 −4

Conversation

louie-tsai
Copy link
Contributor

@louie-tsai louie-tsai commented Jul 24, 2025 •
edited by github-actions bot
Loading

Essential Elements of an Effective PR Description Checklist

  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Make sure all needed run parameters in vLLM Benchmark Suite for 0.9.2.
those new run parameters help to improve 1.3X speedup on v0.9.2 release.

Test Plan

Manually Test.

Test Result

1.3X speedup

(Optional) Documentation Update

@mergify mergify bot added ci/build performance Performance-related issues labels Jul 24, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 24, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the vLLM benchmark suite for Xeon CPUs by adding new benchmark configurations and parameters. My review identified a couple of critical issues that need to be addressed:

  • A recurring typo in an environment variable (VLLM_CPU_SGL_KERNE instead of VLLM_CPU_SGL_KERNEL) across all new and modified benchmark configuration files. This would prevent the intended performance optimizations from taking effect.
  • A logical inconsistency in one of the new test configurations (serving-tests-cpu-snc3.json), where the pipeline_parallel_size does not match the test name, which would lead to incorrect benchmark runs.

I've provided specific suggestions to correct these issues. Once these are fixed, the changes should correctly configure the benchmarks for the intended performance improvements.

.buildkite/nightly-benchmarks/tests/serving-tests-cpu-snc2.json Outdated
"VLLM_RPC_TIMEOUT": 100000,
"VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
"VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
"VLLM_CPU_SGL_KERNE": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There appears to be a typo in the environment variable name. It should be VLLM_CPU_SGL_KERNEL instead of VLLM_CPU_SGL_KERNE. This typo will cause the environment variable to be ignored, preventing the intended performance optimizations from being applied. The correct spelling is confirmed in vllm/envs.py.

	    "VLLM_CPU_SGL_KERNEL": 1,

.buildkite/nightly-benchmarks/tests/serving-tests-cpu-snc2.json Outdated
"VLLM_RPC_TIMEOUT": 100000,
"VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
"VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
"VLLM_CPU_SGL_KERNE": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There appears to be a typo in the environment variable name. It should be VLLM_CPU_SGL_KERNEL instead of VLLM_CPU_SGL_KERNE. This typo will cause the environment variable to be ignored, preventing the intended performance optimizations from being applied. The correct spelling is confirmed in vllm/envs.py.

	    "VLLM_CPU_SGL_KERNEL": 1,

.buildkite/nightly-benchmarks/tests/serving-tests-cpu-snc2.json Outdated
"VLLM_RPC_TIMEOUT": 100000,
"VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
"VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
"VLLM_CPU_SGL_KERNE": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There appears to be a typo in the environment variable name. It should be VLLM_CPU_SGL_KERNEL instead of VLLM_CPU_SGL_KERNE. This typo will cause the environment variable to be ignored, preventing the intended performance optimizations from being applied. The correct spelling is confirmed in vllm/envs.py.

	    "VLLM_CPU_SGL_KERNEL": 1,

.buildkite/nightly-benchmarks/tests/serving-tests-cpu-snc2.json Outdated
"VLLM_RPC_TIMEOUT": 100000,
"VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
"VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
"VLLM_CPU_SGL_KERNE": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There appears to be a typo in the environment variable name. It should be VLLM_CPU_SGL_KERNEL instead of VLLM_CPU_SGL_KERNE. This typo will cause the environment variable to be ignored, preventing the intended performance optimizations from being applied. The correct spelling is confirmed in vllm/envs.py.

	    "VLLM_CPU_SGL_KERNEL": 1,

.buildkite/nightly-benchmarks/tests/serving-tests-cpu-snc2.json Outdated
"VLLM_RPC_TIMEOUT": 100000,
"VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
"VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
"VLLM_CPU_SGL_KERNE": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There appears to be a typo in the environment variable name. It should be VLLM_CPU_SGL_KERNEL instead of VLLM_CPU_SGL_KERNE. This typo will cause the environment variable to be ignored, preventing the intended performance optimizations from being applied. The correct spelling is confirmed in vllm/envs.py.

	    "VLLM_CPU_SGL_KERNEL": 1,

.buildkite/nightly-benchmarks/tests/serving-tests-cpu.json Outdated
@@ -6,6 +6,7 @@
"VLLM_RPC_TIMEOUT": 100000,
"VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
"VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
"VLLM_CPU_SGL_KERNE": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There appears to be a typo in the environment variable name. It should be VLLM_CPU_SGL_KERNEL instead of VLLM_CPU_SGL_KERNE. This typo will cause the environment variable to be ignored, preventing the intended performance optimizations from being applied. The correct spelling is confirmed in vllm/envs.py.

	    "VLLM_CPU_SGL_KERNEL": 1,

.buildkite/nightly-benchmarks/tests/serving-tests-cpu.json Outdated
@@ -36,6 +39,7 @@
"VLLM_RPC_TIMEOUT": 100000,
"VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
"VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
"VLLM_CPU_SGL_KERNE": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There appears to be a typo in the environment variable name. It should be VLLM_CPU_SGL_KERNEL instead of VLLM_CPU_SGL_KERNE. This typo will cause the environment variable to be ignored, preventing the intended performance optimizations from being applied. The correct spelling is confirmed in vllm/envs.py.

	    "VLLM_CPU_SGL_KERNEL": 1,

.buildkite/nightly-benchmarks/tests/serving-tests-cpu.json Outdated
@@ -66,6 +72,7 @@
"VLLM_RPC_TIMEOUT": 100000,
"VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
"VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
"VLLM_CPU_SGL_KERNE": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There appears to be a typo in the environment variable name. It should be VLLM_CPU_SGL_KERNEL instead of VLLM_CPU_SGL_KERNE. This typo will cause the environment variable to be ignored, preventing the intended performance optimizations from being applied. The correct spelling is confirmed in vllm/envs.py.

	    "VLLM_CPU_SGL_KERNEL": 1,

.buildkite/nightly-benchmarks/tests/serving-tests-cpu.json Outdated
@@ -96,6 +105,7 @@
"VLLM_RPC_TIMEOUT": 100000,
"VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
"VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
"VLLM_CPU_SGL_KERNE": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There appears to be a typo in the environment variable name. It should be VLLM_CPU_SGL_KERNEL instead of VLLM_CPU_SGL_KERNE. This typo will cause the environment variable to be ignored, preventing the intended performance optimizations from being applied. The correct spelling is confirmed in vllm/envs.py.

	    "VLLM_CPU_SGL_KERNEL": 1,

.buildkite/nightly-benchmarks/tests/serving-tests-cpu.json Outdated
@@ -129,6 +141,7 @@
"VLLM_RPC_TIMEOUT": 100000,
"VLLM_ALLOW_LONG_MAX_MODEL_LEN": 1,
"VLLM_ENGINE_ITERATION_TIMEOUT_S": 120,
"VLLM_CPU_SGL_KERNE": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

There appears to be a typo in the environment variable name. It should be VLLM_CPU_SGL_KERNEL instead of VLLM_CPU_SGL_KERNE. This typo will cause the environment variable to be ignored, preventing the intended performance optimizations from being applied. The correct spelling is confirmed in vllm/envs.py.

	    "VLLM_CPU_SGL_KERNEL": 1,

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@louie-tsai louie-tsai force-pushed the cpu_benchmark_suite_update branch 2 times, most recently from 6e50c04 to 3b35747 Compare July 24, 2025 03:47
@louie-tsai
Update for 0.9.2 release
d33e81d 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai louie-tsai force-pushed the cpu_benchmark_suite_update branch from 3b35747 to d33e81d Compare July 24, 2025 04:09
@louie-tsai
fix a missing python module issue
2d3c3ed 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
@louie-tsai louie-tsai force-pushed the cpu_benchmark_suite_update branch from 8dfc24c to 2d3c3ed Compare July 24, 2025 04:36
bigPYJ1151
bigPYJ1151 reviewed Jul 24, 2025
View reviewed changes
.buildkite/nightly-benchmarks/scripts/run-performance-benchmarks.sh
@@ -33,7 +33,8 @@ check_gpus() {

check_cpus() {
# check the number of CPUs and NUMA Node and GPU type.
declare -g numa_count=$(python3 -c "from numa import info;numa_size = info.get_num_configured_nodes(); print(numa_size)")
last_numa_index=$(cat /sys/devices/system/node/online | cut -d'-' -f2)
declare -g numa_count=$((last_numa_index+1))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps using numa_count=$(lscpu | grep "NUMA node(s):" | awk '{print $3}') is more robust.

.buildkite/nightly-benchmarks/tests/serving-tests-cpu-snc3.json
@@ -0,0 +1,209 @@
[
{
"test_name": "serving_llama8B_pp1_sharegpt",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be serving_llama8B_pp6_sharegpt

.buildkite/nightly-benchmarks/tests/serving-tests-cpu-snc3.json
},
"server_parameters": {
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"pipeline_parallel_size": 6,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meanwhile, I recommend to use TP/PP together rather than a large PP size. For this case -tp=2 -pp=3 may be a better setting.

@bigPYJ1151
Copy link
Collaborator

0.10.0 has been released. Perhaps we should make this PR based on that :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bigPYJ1151 bigPYJ1151 bigPYJ1151 left review comments

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Assignees
No one assigned
Labels
ci/build performance Performance-related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@louie-tsai @bigPYJ1151