[Bug]: litellm.APIConnectionError: 'str' object has no attribute 'get'

[tjfroese]

What happened?

What happened?
I'm using litellm as proxy for frigate to descripe the events.
By sending the events to litellm the error Type: APIConnectionError, Message: litellm.APIConnectionError: 'str' object has no attribute 'get' raised

GenAI Config frigate:
genai:
enabled: true
provider: openai
api_key: sk-xxxxxxxxxxxxxxxxxxx
base_url: xxxxxxxxxxxxxxxxxx
model: moondream
prompt: "Describe the {label} in the sequence of images with as much detail as possible. Do not describe the background
object_prompts:
cat: "My special cat prompt."
bird: "My special bird prompt."
person: "My special person prompt."

Request from frigate:
{
"model": "moondream",
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABA ...... (truncated 10359 chars)",
"detail": "low"
}
},
"My special person prompt."
]
}

Is this a bug in litellm or for figrate?

Relevant log output

Type:
APIConnectionError
Message:
litellm.APIConnectionError: 'str' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/litellm/main.py", line 2929, in completion
    response = base_llm_http_handler.completion(
        model=model,
    ...<13 lines>...
        client=client,
    )
  File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 307, in completion
    data = provider_config.transform_request(
        model=model,
    ...<3 lines>...
        headers=headers,
    )
  File "/usr/lib/python3.13/site-packages/litellm/llms/ollama/completion/transformation.py", line 340, in transform_request
    modified_prompt = ollama_pt(model=model, messages=messages)
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/prompt_templates/factory.py", line 206, in ollama_pt
    if m.get("type", "") == "image_url":
       ^^^^^
AttributeError: 'str' object has no attribute 'get'
. Received Model Group=moondream
Available Model Group Fallbacks=None LiteLLM Retried: 1 times, LiteLLM Max Retries: 2

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.71.1

Twitter / LinkedIn details

No response

[VigneshwarRajasekaran]

Hi @tjfroese
Litellm requires the request for vision models to be a dictionary following the OpenAIs vision API , but your request contains String.
} }, "My special person prompt." ] }

Hence the error.

If you think this is resolved, please close this issue and open a new issue requesting this feature.

Cheers!

[tjfroese]

Hi Vigneshwar,
thanks a lot! Your open my eyes to the right spot.

I've analyse the problem more in detail on frigate site. Therefore Version with the latest tag (0.15.0) has this problem.

Problem solved by using the newst version >0.16