Does the continuous batching technology contain the concept of batch size in the vLLM online service scenario? #2259
Alkaid-Friderich
announced in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Does the continuous batching technology contain the concept of batch size in the vLLM online service scenario ? Where is the relevant code about how to set the batch size at the begin and how to resize it dynamically on the server?
Beta Was this translation helpful? Give feedback.
All reactions