VLLM output not complete

[RickyGunawan09]

hai guys,
thank you for making this super library.
i have a question about the output of vllm

i'm using GPU RTX A6000 50GB cuda 12 with model Vicuna13B-v1.5-4k from lmsys
vllm is serve with gpu_memory_utilization 0.8
the parameter that i change for request is:

max_token 4096
temperature 0
i'm make custom prompt with context from text/document.

why sometimes the output is not complete ?

[zhiqiangzhongddu]

+1, my obtained answers are always not complete. Even shorter than a sentence.

[zhiqiangzhongddu]

Hi this might be helpful for you. You can set the length of output to get complete answers.

vllm/vllm/sampling_params.py

Line 61 in ee8217e

max_tokens: Maximum number of tokens to generate per output sequence.

[jayachandrakalakutagar]

Still broken responses