Mistral Medium with image_url

[enrico-stauss]

• Package Name: azure-ai-inference
• Package Version: 1.0.0b9
• Operating System: Windows 11
• Python Version: 3.10

Describe the bug
Deployment of multimodal model in the Azure AI Foundry raises internal model_error when sending messages containing image blocks in form of image_url

To Reproduce
Steps to reproduce the behavior:

1. Deploy Mistral Medium to AI Foundry
2. Execute this example: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-inference/samples/sample_chat_completions_from_input_dict_with_image_url.py

Expected behavior
An image description should be returned.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dargilco @jhakulin @trangevi.

[trangevi]

Mistral Medium does not support taking an image as a url pointer. Please look to one of our samples providing the image as base 64 encoded data, as that will work with this model: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-inference/samples/sample_chat_completions_with_image_data.py

I've filed a separate ticket on our backend team to improve the failure message here.

[github-actions]

Hi @enrico-stauss. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[enrico-stauss]

Mistral Medium supports this through their official API and I figured that since you have an example here where an image url is passed to Mistral Small, this should be supported by Mistral Medium, too.

[enrico-stauss]

/unresolve