Passing Multi-Modal Content to LiteLLM

[tianqizhao-louis]

Hello everyone,

I'm wondering if llama-index supports passing multi-modal contents like images to its LiteLLM wrapper.

In the code, it seems it just converts all messages' TextBlock.

Appreciate any help ahead!

def content(self) -> str | None:
    """Keeps backward compatibility with the old `content` field.

    Returns:
        The cumulative content of the TextBlock blocks, None if there are none.
    """
    content = ""
    for block in self.blocks:
        if isinstance(block, TextBlock):
            content += block.text

    return content or None

def to_openai_message_dict(message: ChatMessage) -> dict:
    """Convert generic message to OpenAI message dict."""
    message_dict = {
        "role": message.role,
        "content": message.content,
    }

    # NOTE: openai messages have additional arguments:
    # - function messages have `name`
    # - assistant messages have optional `function_call`
    message_dict.update(message.additional_kwargs)

    return message_dict

[dosubot[bot]]

LlamaIndex does support passing multi-modal contents like images to its LiteLLM wrapper. The integration with Reka's multi-modal language models allows you to use image input capabilities in your LlamaIndex applications. The MultiModalCompleteComponent class processes inputs that include image_documents, which must be a list of ImageNode objects. This framework can handle both text prompts and image documents, enabling chat and text completions with images, and even streaming responses, by using the RekaMultiModalLLM class ^[1][2][3].

The to_openai_message_dict function you mentioned is designed to convert a ChatMessage to a format compatible with OpenAI's API. It handles additional content types beyond text, such as images, by checking the type of each block in the message.blocks. For ImageBlock, it can append a dictionary with the type "image_url" and the URL, or encode the image in base64 if no URL is provided ^[4][5].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[logan-markewich]

@tianqizhao-louis litellm would need to be updated to support actually passing the content blocks, similar to Anthropic, Ollama, Openai, Google GenAI, etc.

We've been slowly rolling out this change and I haven't updated litellm yet, would be amazing to get a PR for this