Support vLLM reasoning api

[arichiardi]

Hello @karthink - picking up from #613 (comment).

I tried the gptel-make-openai backend and I still do not see the reasoning content.

Didn't debug very thoroughly but when I receive a response I dump the info structure here and see the following in all responses (note that stream is on):

:stream t :include-reasoning *scratch* :http-status 200 :status HTTP/1.1 200 OK :reasoning-block done

The :reasoning-done is something I did not see in the vllm responses: this is a sample I got with curl (just kept the last few):

...
{"id":"chatcmpl-723389392ba442f4a23d83625ea4986e","object":"chat.completion.chunk","created":1742496631,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" large"},"logprobs":null,"finish_reason":null}]}

{"id":"chatcmpl-723389392ba442f4a23d83625ea4986e","object":"chat.completion.chunk","created":1742496631,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" language"},"logprobs":null,"finish_reason":null}]}

{"id":"chatcmpl-723389392ba442f4a23d83625ea4986e","object":"chat.completion.chunk","created":1742496631,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" models"},"logprobs":null,"finish_reason":"length","stop_reason":null}]}

Where can I go from here?

[karthink]

I tried the gptel-make-gptel backend and I still do not see the reasoning content.

I presume you mean gptel-make-deepseek.

Didn't debug very thoroughly but when I receive a response I dump the info structure here and see the following in all responses (note that stream is on):

This is expected. gptel doesn't know apriori if a model supports reasoning or not, so it always checks.

Where can I go from here?

Could you share the complete response?

[arichiardi]

@karthink

This a full response (limited to 120 tokens)

curl http://<my-local-host>/v1/chat/completions -H "Content-Type: application/json" -d '{   "model": "Qwen/QwQ-32B-AWQ", "stream": true,   "messages": [     {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},     {"role": "user", "content": "Tell me something about large language models."}   ],   "temperature": 0.7,   "top_p": 0.8,   "repetition_penalty": 1.05,   "max_tokens": 10 }' 
data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":"Okay"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":","},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" the"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" user"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" wants"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" to"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" know"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" something"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" about"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-1be9da4d8a0e4028b9ccf59b7dfadaf9","object":"chat.completion.chunk","created":1743007637,"model":"Qwen/QwQ-32B-AWQ","choices":[{"index":0,"delta":{"reasoning_content":" large"},"logprobs":null,"finish_reason":"length","stop_reason":null}]}

data: [DONE]

[arichiardi]

@karthink wanted to kidly pong you on this 😅 no rush but I can help debugging what is going on...

[karthink]

Sorry, I forgot what the problem was here. IIUC,

1. You're using a reasoning model via vLLM,
2. via gptel-make-deepseek,
3. with streaming turned on,
4. but seeing no reasoning content at your desired destination (scratch buffer).

The sample output looks fine for deepseek, but there's no actual response content? I only see "reasoning_content" blocks.

[arichiardi]

Cannot recall and now I am basically on qwen3 and cannot reproduce it anymore.
I was using gptel-make-openai FWIW but I'll close this and open a new one if I see it again.

Thanks @karthink