Setup vLLM benchmark CI for H100

[huydhn]

The new workflow can be run periodically every 2 hours or on demand by setting the commit from vLLM main branch to benchmark. It works as follows:

1. When schedule, the workflow checks the latest commits from vLLM main branch chronologically until it finds the latest commit whose vLLM CI Docker image has already been built and has not been benchmarked yet. The Docker image name is public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:<SHA>
2. When running on demand, it will just check for the request Docker image and returns early if that doesn't exist yet
3. The workflows uses the official benchmark scripts from vLLM at https://github.com/vllm-project/vllm/blob/main/.buildkite/nightly-benchmarks/scripts/run-performance-benchmarks.sh
4. Instead of using the list of models from vLLM, we are going to use those from https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks so that we can control exactly what to benchmark
5. 4xH100 currently takes 45 minutes to finish all benchmarks

Some more PRs are coming after this:

• ROCm MI300x benchmark. I need to figure out the Docker image vLLM uses for this
• Adding Llama 4 Scout and Maverick to https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks

Testing

The results are showing up on the dashboard now https://hud.pytorch.org/benchmark/llms?startTime=Fri%2C%2023%20May%202025%2019%3A19%3A35%20GMT&stopTime=Fri%2C%2030%20May%202025%2019%3A19%3A35%20GMT&granularity=day&lBranch=main&lCommit=7f21e8052b5f3948c8a59514a8dc1e9c5eef70d6&rBranch=main&rCommit=7f21e8052b5f3948c8a59514a8dc1e9c5eef70d6&repoName=vllm-project%2Fvllm&benchmarkName=&modelName=All%20Models&backendName=All%20Backends&modeName=All%20Modes&dtypeName=All%20DType&deviceName=All%20Devices&archName=All%20Platforms

[yangw-dev]

LGTM!

[yangw-dev]

for my own knowledge, is this mean instance with 4 h100?

how many of those we have now?

[huydhn]

We have 4 of them atm. Also, FYI, there is one 8xH100 runner too.