[Feature]: Support Mistral OCR

[MalteHB]

The Feature

Support for sending stuff to Mistral OCR

Motivation, pitch

I am working on a solution where OCR would be very helpful, and given that Mistral has the SOTA, it would be awesome to be able to use them.

Are you a ML Ops Team?

No

Twitter / LinkedIn details

No response

[wagnerjt]

Maybe you can use the custom provider and write against the rest client. The REST spec is here

[antonkulaga]

I will be super-useful if it will be implemened

[ishaan-jaff]

hi @antonkulaga, @MalteHB what API spec should mistral OCR follow on litellm ?

Do you want to call it in the native mistral api spec ? If it's on the proxy, we can add a mistral api pass through like we do for cohere: https://docs.litellm.ai/docs/pass_through/cohere

[qdrddr]

This potentially can be used with /files endpoint or a pass through I think

[krrishdholakia]

So it looks like a new endpoint we need to onboard - /ocr

the response doesn't look openai like - how would you expect to call this?

[krrishdholakia]

Passthrough for mistral might make the most sense here. Let me know if that works for you? @qdrddr @antonkulaga @MalteHB

/mistral/v1/ocr

[krrishdholakia]

Bump on this?

[MoussaSak]

Hello @krrishdholakia , is it possible to add support of Mistral OCR on Azure AI Foundry as well ?

[M4teo]

o/
With the help of this doc : https://docs.litellm.ai/docs/proxy/pass_through
we finally configured liteLLM as AI gateway for mistral-ocr hosted by Azure.
The config looks like :

    pass_through_endpoints:
      - path: "/v1/ocr"                                
        target: "${model_mistral_ocr_api_base}v1/ocr?api-version=2024-05-01-preview"
        headers:
          Authorization: "Bearer ${model_mistral_ocr_api_key}"
          content-type: "application/json"  
          model: "mistral-ocr-2503-xxxxx"
        forward_headers: false