What is the recommended way to add image to LLM chat history after calling a tool?

[101scholar]

If a tool calling returns an image instead of texts, how to add it properly to the chat history? My workaround is

def load_screenshot(tool_context: ToolContext):
    """ Load the system screenshot from clipboard.
    """
    if image_bytes := get_clipboard_image():
        tool_context._invocation_context.user_content.parts.append(
            types.Part.from_bytes(
                data=image_bytes,
                mime_type="image/png"
            )
        )
        return "Successfully load screenshot from clipboard."
    else:
        return "Failed to load screenshot from clipboard."

Considering _invocation_context is an internal variable, I guess it is not the traditional way to do this. Then what is the recommended way for this purpose?

[seanzhou1023]

I don't think it's a right way to do it. Mainly:

1. you are not supposed to modify invocation context yourself
2. the content you added won't relate to the original function call

I don't think we support blob as function response, maybe you can try base64 encoding ?

[seanzhou1023]

since response is of type dict[str, Any], you may try put types.Blob as one of the value in the dict.