[Bug]: Mistrall Small 3.2 doesn't work with images

[tomschelsen]

Your current environment

vLLM 0.9.1 (Docker)

mistralai/Mistral-Small-3.2-24B-Instruct-2506

Using mistralai/Mistral-Small-3.1-24B-Instruct-2503 with everything else unchanged works.

🐛 Describe the bug

ERROR 06-24 14:40:09 [serving_chat.py:200] Error in preprocessing prompt inputs ERROR 06-24 14:40:09 [serving_chat.py:200] Traceback (most recent call last): ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 183, in create_chat_completion ERROR 06-24 14:40:09 [serving_chat.py:200] ) = await self._preprocess_chat( ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_engine.py", line 787, in _preprocess_chat ERROR 06-24 14:40:09 [serving_chat.py:200] conversation, mm_data_future = parse_chat_messages_futures( ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1196, in parse_chat_messages_futures ERROR 06-24 14:40:09 [serving_chat.py:200] sub_messages = _parse_chat_message_content( ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1120, in _parse_chat_message_content ERROR 06-24 14:40:09 [serving_chat.py:200] result = _parse_chat_message_content_parts( ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1020, in _parse_chat_message_content_parts ERROR 06-24 14:40:09 [serving_chat.py:200] parse_res = _parse_chat_message_content_part( ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1077, in _parse_chat_message_content_part ERROR 06-24 14:40:09 [serving_chat.py:200] mm_parser.parse_image(str_content) ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 777, in parse_image ERROR 06-24 14:40:09 [serving_chat.py:200] placeholder = self._tracker.add("image", image_coro) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 586, in add ERROR 06-24 14:40:09 [serving_chat.py:200] mm_processor = mm_registry.create_processor(model_config) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/multimodal/registry.py", line 275, in create_processor ERROR 06-24 14:40:09 [serving_chat.py:200] model_cls = self._get_model_cls(model_config) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/multimodal/registry.py", line 246, in _get_model_cls ERROR 06-24 14:40:09 [serving_chat.py:200] model_cls, _ = get_model_architecture(model_config) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py", line 240, in get_model_architecture ERROR 06-24 14:40:09 [serving_chat.py:200] model_cls, arch = ModelRegistry.resolve_model_cls(architectures) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/model_executor/models/registry.py", line 489, in resolve_model_cls ERROR 06-24 14:40:09 [serving_chat.py:200] return self._raise_for_unsupported(architectures) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/model_executor/models/registry.py", line 426, in _raise_for_unsupported ERROR 06-24 14:40:09 [serving_chat.py:200] raise ValueError( ERROR 06-24 14:40:09 [serving_chat.py:200] ValueError: Model architectures ['PixtralForConditionalGeneration'] failed to be inspected. Please check the logs for more details. INFO: 10.231.0.4:36046 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[princepride]

@DarkLight1337 Can you assign this task to me, I noticed that the Mistral 3.2 seems different from Mistral 3.1, I reproduced the problem and noticed some code works on Mistral 3.1 but not works on Mistral 3.2, I will try to solve it.

[gaby]

@princepride Both small-3.2 and magistral require the latest mistral-common>=1.6.2 package.

While support for that was merged, there hasn't been any releases in 2 weeks, so when people use the latest docker image it's using an older package without the new mistral tokenizer.

[princepride]

I see, but when I change the model from mistral3.1 to mistral3.2 and run the example code of offline vision inference, got another error.

[gaby]

Interesting, I did notice I can't get function calling to work properly with small/devstral and vLLM. But it's related to vLLM using regex to perse JSON and not handling single quotes.

[princepride]

I know where the problem lies. Mistral has its own tokenizer function, and the online reasoning code follows their logic, but the offline reasoning in the example does not configure Mistral's own tokenizer function. Fortunately, Mistral 3.1 has a complete set of standard transformers format tokenizer configuration files, but Mistral 3.2 lacks it. This results in the same example code being able to run in Mistral 3.1, but not in Mistral 3.2. I have seen that many users have requested that the huggingface repository of Mistral 3.2 also provide a complete set of transformers configuration files. By the way, I think we should add ignore_patterns=["consolidated.safetensors"] in the example, because now all the weight files in the huggingface repository will be downloaded, which is close to 100G.

[DarkLight1337]

cc @patrickvonplaten

[erikmargaronis]

I stumbled upon this thread when trying out Mistral Small 3.2 with vLLM v0.9.1 but both vision and tool calling seem to work fine for me. For tool calling I had to specify the chat template suggested by vLLM here: https://docs.vllm.ai/en/stable/features/tool_calling.html#mistral-models-mistral

My compose file:

services:
  vllm-service:
    image: vllm/vllm-openai:v0.9.1
    networks:
      - common_network
    environment:
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - VLLM_FLASH_ATTN_VERSION=3
      - VLLM_USE_V1=0
    command:
      - "--host=0.0.0.0"
      - "--port=8000"
      - "--model=/app/Mistral-Small-3.2-24B-Instruct-2506"
      - "--served-model-name=Mistral-Small-3.2-24B-Instruct-2506"
      - "--enforce-eager"
      - "--max-model-len=65535"
      - "--tokenizer_mode=mistral"
      - "--load_format=mistral"
      - "--config_format=mistral"
      - "--dtype=half"
      - "--enable-auto-tool-choice"
      - "--tool-call-parser=mistral"
      - "--limit-mm-per-prompt"
      - "image=10"
      - "--chat-template=/templates/tool_chat_template_mistral_parallel.jinja"
    volumes:
      - "/model-share/mistralai/Mistral-Small-3.2-24B-Instruct-2506:/app/Mistral-Small-3.2-24B-Instruct-2506"
      - "/model-share/chat_templates/tool_chat_template_mistral_parallel.jinja:/templates/tool_chat_template_mistral_parallel.jinja"
    ports:
      - "5008:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [ gpu ]
              device_ids: [ '0' ]

[gaby]

@erikmargaronis that image doesnt have the latest mistral tokenizer and it still works for you?

[erikmargaronis]

@gaby Yes, it seems to work fine. Do you have an example that fails for you that I can try?

[princepride]

You can refer to the example code of offline vision inference, but you need change model from mistral 3.1 to mistral 3.2

…

On Thu, Jul 3, 2025 at 9:09 PM erikmargaronis ***@***.***> wrote: *erikmargaronis* left a comment (vllm-project/vllm#20025) <#20025 (comment)> @gaby <https://github.com/gaby> Yes, it seems to work fine. Do you have an example that fails for you that I can try? — Reply to this email directly, view it on GitHub <#20025 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHDXVGAZ4HUBLS67AN747OT3GUTP7AVCNFSM6AAAAACAAUSDUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMZSGIYDQMZQGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[erikmargaronis]

@princepride I'm not sure which example you mean. To be clear, I'm using the docker image v0.9.1 to serve an OpenAI compatible server, if that makes any difference.

Here's some more info from my configuration:

INFO 07-03 06:43:10 [llm_engine.py:230] Initializing a V0 LLM engine (v0.9.1) with config: model='/app/Mistral-Small-3.2-24B-Instruct-2506', speculative_config=None, tokenizer='/app/Mistral-Small-3.2-24B-Instruct-2506', skip_tokenizer_init=False, tokenizer_mode=mistral, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=65535, download_dir=None, load_format=LoadFormat.MISTRAL, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='xgrammar', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=None, served_model_name=Mistral-Small-3.2-24B-Instruct-2506, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":false,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}, use_cached_outputs=True,

And for what it's worth, here some code that runs fine for me:

from openai import OpenAI

client = OpenAI(
    api_key="test",
    base_url="http://myserver:8000/v1/",
)

messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe these images"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg",
                    }
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/7/77/002_The_lion_king_Snyggve_in_the_Serengeti_National_Park_Photo_by_Giles_Laurent.jpg",
                    }
                },
            ],
        }
    ]

response = client.chat.completions.create(
    messages=messages,
    model="Mistral-Small-3.2-24B-Instruct-2506"
)

print(response.choices[0].message.content)

Response:

Sure, let's break down each image description:

1. First Image - Duck:

   • Main Subject: A duck floating on water.
   • Water: The background shows a blue, rippled water surface indicating gentle movements.
   • Duck:
     • Head: The duck has a distinctive green head with a yellowish beak.
     • Body: Its body is a mix of brown and white with some shading, giving an impression of smooth, waterproof feathers. The softly textured feathers glistening slightly under the light suggest the variegated plumage typical of many duck species.
     • Posture: The duck appears calm and at ease, with its body almost entirely submerged but for the top part, showcasing the natural swimming ability of ducks.

2. Second Image - Lion:

   • Main Subject: A lion.
   • Environment: The background consists of dry, tall grasses indicative of a savanna or grassland habitat.
   • Lion:
     • Mane: The lion has a thick, dark mane around its head, which is a characteristic of male lions.
     • Facial Features: It appears calm with an intense gaze.
     • Body Posture: The lion is seated in the middle of tall grasses, exuding a calm and attentive posture.

These images represent a quiet moment in nature, capturing the reflective duck on water and the majestic lion in its natural environment.

Edit: @princepride I see now that you made an investigation and found issues in the offline code, while, if I understand you correctly, the online version works fine. I'm running the online version using the official vllm/vllm-openai:v0.9.1 image. I understood that the OP also uses the Docker image so I let my posts remain if it helps anyone.

[PedroMiolaSilva]

@erikmargaronis I've used the example that you shared here and now mistral3.2 is working with images. However, when it comes to stream with tool calling, I'm still getting an error:

ERROR 07-03 11:30:24 [mistral_tool_parser.py:365] Error trying to handle streaming tool call.
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365] Traceback (most recent call last):
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/tool_parsers/mistral_tool_parser.py", line 220, in extract_tool_calls_streaming
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]     tool_call_arr: list[dict] = partial_json_parser.loads(
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]   File "/usr/local/lib/python3.12/dist-packages/partial_json_parser/core/api.py", line 15, in parse_json
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]     return parser(ensure_json(json_string, allow_partial, use_fast_fix))
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]   File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]     return _default_decoder.decode(s)
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]   File "/usr/lib/python3.12/json/decoder.py", line 338, in decode
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]   File "/usr/lib/python3.12/json/decoder.py", line 356, in raw_decode
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365]     raise JSONDecodeError("Expecting value", s, err.value) from None
ERROR 07-03 11:30:24 [mistral_tool_parser.py:365] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Could you confirm if streaming tool calls work for you?

I think the only thing different here is because I'm using a quantized version of the model. I'm also using the suggested jinja template:

source ../.env
export MODEL_ID=unsloth/Mistral-Small-3.2-24B-Instruct-2506-FP8
export MODEL_ID_PORT=8000
export MODEL_ID_GPU=0

docker run \
--runtime nvidia \
-e VLLM_USE_V1=0 \
--gpus all \
--ipc=host \
-p "${MODEL_ID_PORT}:8000" \
--env "HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}" \
--env "HF_HUB_OFFLINE=0" \
-v "${HF_HOME}:/root/.cache/huggingface" \
-v ~/.cache/vllm/torch_compile_cache:/root/.cache/vllm/torch_compile_cache \
-v "$(pwd):/app" \
vllm/vllm-openai:latest \
--model ${MODEL_ID} \
--enforce-eager \
--tool-call-parser mistral \
--enable-auto-tool-choice \
--chat-template /app/template.jinja \
--tokenizer_mode mistral \
--config_format mistral \
--load_format mistral \
--max-model-len 26000 \
--gpu-memory-utilization 0.8