Skip to content

[Data][LLM] Bump vLLM version to support new models #51726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Apr 1, 2025
Merged

[Data][LLM] Bump vLLM version to support new models #51726

merged 11 commits into from
Apr 1, 2025

Conversation

lk-chen
Copy link
Contributor

@lk-chen lk-chen commented Mar 26, 2025

Why are these changes needed?

Vision models (e.g. pixtral) are only supported in vLLM release (0.8.2), this PR bumps vLLM requirements and adds release tests for new models.

Related issue number

#51313

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@lk-chen lk-chen added the go add ONLY when ready to merge, run all tests label Mar 26, 2025
GeneDer
GeneDer reviewed Mar 26, 2025
View reviewed changes
Copy link
Contributor

@GeneDer GeneDer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also run release test for Ray Serve LLM? just to ensure that's also compatible with the new version.

@kouroshHakha
Copy link
Contributor

@lk-chen I think we need to merge #51490 first. My understanding is that vllm has v1 on by default after some point. That PR takes care of v1 vs. v0 on serve.

@comaniac
Copy link
Contributor

@lk-chen I think we need to merge #51490 first. My understanding is that vllm has v1 on by default after some point. That PR takes care of v1 vs. v0 on serve.

We actually need to merge this PR first and rebase the other one, because that PR was based on a vLLM commit included in 0.8.2

@kouroshHakha
Copy link
Contributor

Can we run serve llm release tests and see if they fail with the upgrade? This release tests only covered the batch ones.

@lk-chen
Copy link
Contributor Author

lk-chen commented Mar 27, 2025

seems serve tests are failing. I'm turning it to draft for now.

@lk-chen lk-chen marked this pull request as draft March 27, 2025 04:28
@comaniac
Copy link
Contributor

seems serve tests are failing. I'm turning it to draft for now.

Note that 0.8.2 will use v1 by default, so you may need to combine with #51490, or explicitly add VLLM_USE_V1=0 in this PR first.

@lk-chen lk-chen force-pushed the lkchen-ray_data_llm branch from f2810c7 to f743e6b Compare March 27, 2025 18:48
aslonnie
aslonnie reviewed Mar 27, 2025
View reviewed changes
Copy link
Collaborator

@aslonnie aslonnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me know when this is ready for review (or if you have any questions)

@lk-chen lk-chen force-pushed the lkchen-ray_data_llm branch 2 times, most recently from 8691592 to ebc4b15 Compare March 27, 2025 23:27
@lk-chen lk-chen closed this Mar 28, 2025
@lk-chen lk-chen deleted the lkchen-ray_data_llm branch March 28, 2025 00:05
@lk-chen lk-chen restored the lkchen-ray_data_llm branch March 28, 2025 00:05
@lk-chen lk-chen reopened this Mar 28, 2025
@lk-chen
bump vLLM to 0.8.2
4b97d68 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen lk-chen force-pushed the lkchen-ray_data_llm branch from 115f395 to 4b97d68 Compare March 28, 2025 00:12
@comaniac comaniac mentioned this pull request Mar 28, 2025
Open
1 task
lk-chen added 2 commits March 28, 2025 11:59
@lk-chen
add test_id for test_json_mode
1c91acc 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen
force V0 in unit test
5a9f079 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen lk-chen force-pushed the lkchen-ray_data_llm branch from 281b4ac to 5a9f079 Compare March 28, 2025 21:21
@lk-chen lk-chen mentioned this pull request Mar 28, 2025
Closed
8 tasks
@lk-chen
Merge branch 'master' into lkchen-ray_data_llm
91bb7d2 
Signed-off-by: Linkun Chen <github@lkchen.net>
@comaniac comaniac mentioned this pull request Mar 28, 2025
Closed
8 tasks
lk-chen added 3 commits March 30, 2025 10:17
@lk-chen
set 90s timeout
e290330 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen
update test for V1
5a80423 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen
[llm][test] Use smolVLM in unit tests
e4d60f2 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen lk-chen mentioned this pull request Mar 31, 2025
Merged
9 tasks
@lk-chen lk-chen force-pushed the lkchen-ray_data_llm branch from 0f87863 to a5d74ea Compare March 31, 2025 19:55
@lk-chen
Merge branch 'smolvlm-for-test' into lkchen-ray_data_llm
d92f4bc 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen lk-chen marked this pull request as ready for review March 31, 2025 21:28
@lk-chen lk-chen requested a review from a team as a code owner March 31, 2025 21:28
@lk-chen
Merge branch 'master' into lkchen-ray_data_llm
152cb54 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen lk-chen force-pushed the lkchen-ray_data_llm branch from a5d74ea to 152cb54 Compare March 31, 2025 22:07
@lk-chen
revert doc change
f8aabd4 
Signed-off-by: Linkun Chen <github@lkchen.net>
@kouroshHakha
Copy link
Contributor

kicked off the release tests: https://buildkite.com/ray-project/release/builds/37586

@lk-chen lk-chen force-pushed the lkchen-ray_data_llm branch from d70a554 to 04592da Compare April 1, 2025 03:27
@lk-chen
default v0, and warning
a9351b3 
Signed-off-by: Linkun Chen <github@lkchen.net>
@lk-chen lk-chen force-pushed the lkchen-ray_data_llm branch from 04592da to a9351b3 Compare April 1, 2025 04:47
@kouroshHakha kouroshHakha enabled auto-merge (squash) April 1, 2025 05:10
@kouroshHakha kouroshHakha requested a review from aslonnie April 1, 2025 05:10
@kouroshHakha kouroshHakha merged commit 96349e8 into ray-project:master Apr 1, 2025
6 checks passed
@lk-chen lk-chen deleted the lkchen-ray_data_llm branch April 1, 2025 16:50
This was referenced Apr 2, 2025
Open
Open
pavelkushtia pushed a commit to pavelkushtia/ray that referenced this pull request Apr 6, 2025
@lk-chen
[llm] Bump vLLM version to support new models (ray-project#51726)
c8eec7c 
Signed-off-by: Linkun Chen <github@lkchen.net>
vickytsang pushed a commit to ROCm/ray that referenced this pull request May 5, 2025
@lk-chen
[llm] Bump vLLM version to support new models (ray-project#51726)
4e1c415 
Signed-off-by: Linkun Chen <github@lkchen.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GeneDer GeneDer GeneDer left review comments

@comaniac comaniac comaniac left review comments

@kouroshHakha kouroshHakha kouroshHakha approved these changes

@aslonnie aslonnie aslonnie approved these changes

@edoakes edoakes Awaiting requested review from edoakes edoakes is a code owner

@zcin zcin Awaiting requested review from zcin

@akshay-anyscale akshay-anyscale Awaiting requested review from akshay-anyscale

@richardliaw richardliaw Awaiting requested review from richardliaw richardliaw is a code owner

Assignees
No one assigned
Labels
go add ONLY when ready to merge, run all tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@lk-chen @kouroshHakha @comaniac @GeneDer @aslonnie