Structured output with vLLM?

[AkshataDM]

Has anyone looked into getting structured output from a model hosted using vllm? Would like some pointers on implementing this if anyone has any suggestions.

[jeejeelee]

See: https://github.com/vllm-project/vllm/blob/v0.6.0/vllm/engine/arg_utils.py#L276

[NicolasDrapier]

Hi @AkshataDM
Yes, I enforce the model to respond with a structured output via lm-format-enforcer. In my Docker setup, I add the parameter --guided-decoding-backend lm-format-enforcer, and then I call it with the following code:

from pydantic import BaseModel
import json
from openai import OpenAI

class AnswerFormat(BaseModel):
    name: str
    lastname: str
    age: int

client = OpenAI(
    base_url="http://...../v1",
    api_key="token",
)

user_prompt = "......"
completion = client.chat.completions.create(
    model="yourmodelhere",
    messages=[
        {"role": "user", "content": user_prompt}
    ],
    extra_body={
        "guided_json": AnswerFormat.model_json_schema()
    }
)

result = json.loads(completion.choices[0].message.content)

[MohammedAlsayed]

When I enable --guided-decoding-backend, the function calling options (--enable-auto-tool-choice --tool-call-parser llama3_json --chat-template tool_chat_template_llama3.2_json.jinja) doesn't execute ... any ideas?

[datalee]

must 0.6+ support？