Issues with Guided Decoding Conformance to JSON Schema in vllm

[justHungryMan]

Hi there,

I’m currently using vllm to work with outlines, and I’m incorporating jsonschema for guided decoding. However, it seems like the responses aren’t fully conforming to the jsonschema. Here are a couple of issues I’ve encountered:

1.	Failure in JSON decoding.
2.	Responses not matching the pattern specified in the jsonschema. (like Validation Error: 'next Monday' does not match '^\\d{4}-\\d{2}-\\d{2}$')

For validation, I’m providing jsonschema as guided_json and then validating the generated data against it. Is this behavior expected, or could this be a bug?

Sample Code

completion = client.chat.completions.create(
    model="MODEL",
    messages=[{"role": "user", "content": prompt}],
    extra_body={"guided_json": sample_json_schema},
    max_tokens=2048,
    temperature=0.00001,
)

output_json = completion.choices[0].message.content
jsonschema.validate(
    instance=json.loads(output_json), schema=sample_json_schema
)

Let me know if there’s anything I might be overlooking. Thanks!

[rlouf]

Could you provide the JSON Schema?

[justHungryMan]

@rlouf Ok, here are examples

1. jsonschema

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title": "Trip Itinerary and Calendar Integration",
    "type": "object",
    "properties": {
      "intent": {
        "type": "string",
        "enum": [
          "setup_trip_itinerary",
          "schedule_meeting",
          "add_reminder"
        ],
        "description": "The primary purpose of the user's request."
      },
      "destination": {
        "type": [
          "string",
          "null"
        ],
        "description": "The destination of the trip."
      },
      "trip_start_date": {
        "type": [
          "string",
          "null"
        ],
        "format": "date",
        "description": "The start date of the trip."
      },
      "trip_end_date": {
        "type": [
          "string",
          "null"
        ],
        "format": "date",
        "description": "The end date of the trip."
      },
      "activities": {
        "type": [
          "array",
          "null"
        ],
        "items": {
          "type": "string"
        },
        "description": "List of activities planned for the trip."
      },
      "calendar_integration": {
        "type": [
          "boolean",
          "null"
        ],
        "description": "Whether the itinerary should be added to the calendar."
      }
    },
    "required": [
      "intent",
      "destination",
      "trip_start_date",
      "trip_end_date",
      "activities",
      "calendar_integration"
    ]
  }

and this is generated output

{
    "intent": "setup_trip_itinerary",
    "destination": "Tokyo",
    "trip_start_date": "2022-10-15",
    "trip_end_date": "2022-10-20",
    "activities": [
        {
            "name": "visit_museums",
            "description": "Visit some of the top museums in Tokyo"
        },
        {
            "name": "try_local_food",
            "description": "Try some of the local food in Tokyo"
        }
    ],
    "calendar_integration": true
}

2. jsonschema

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title": "Traffic Pattern Prediction and Route Optimization Request",
    "type": "object",
    "properties": {
      "intent": {
        "type": "string",
        "enum": [
          "predict traffic patterns",
          "optimize route"
        ]
      },
      "date": {
        "type": [
          "string",
          "null"
        ],
        "format": "date",
        "description": "The date for which the traffic pattern prediction is requested."
      },
      "start_location": {
        "type": [
          "string",
          "null"
        ],
        "description": "The starting location for the route."
      },
      "destination_location": {
        "type": [
          "string",
          "null"
        ],
        "description": "The destination location for the route."
      },
      "timeslot": {
        "type": [
          "string",
          "null"
        ],
        "pattern": "^([1-9]|1[0-2])(:[0-5][0-9])?\\s?(AM|PM)$",
        "description": "The time slot during which the traffic prediction and route optimization is needed, formatted as either a full hour or including minutes and meridiem (e.g., '8 AM', '8:30 AM')."
      }
    },
    "required": [
      "intent",
      "date",
      "start_location",
      "destination_location",
      "timeslot"
    ]
  }

and this is generated output

{
    "intent": "predict traffic patterns",
    "date": "tomorrow",
    "start_location": "Central Park",
    "destination_location": "Brooklyn Bridge",
    "timeslot": "morning rush hour"
}

thank you for your help.

[justHungryMan]

I think it's due to #215 and vllm is not supported v0.1.0 yet.

[justHungryMan]

by the way, can I use "pattern" in string at version v0.1.1?