Support for Displaying Image Outputs in Langserve Playground for Text-to-Image Tasks

[amizi93130]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from typing import Any
from pydantic import BaseModel, Field
from langchain_core.runnables import RunnableLambda
from openai import OpenAI
import base64
import requests


class Request(BaseModel):
    raw_prompt: str = Field(description="Enter Your Prompt")

    @classmethod
    def model_json_schema(cls, *args, **kwargs) -> dict[str, Any]:
        schema = super().model_json_schema(*args, **kwargs)
        required_fields = ["raw_prompt"]
        optional_fields = []
        if "properties" in schema:
            for field in optional_fields:
                if field in schema["properties"]:
                    model_type = schema["properties"][field].pop("type")
                    schema["properties"][field]["type"] = [model_type, "null"]
        if "required" in schema:
            schema["required"] = required_fields
        return schema


def generate_image(enriched_prompt):
    client = OpenAI()
    response = client.images.generate(
        model="dall-e-3",
        prompt=enriched_prompt,
        size="1024x1024",
        quality="standard",
        n=1,
    )
    image_url = response.data[0].url

    print(image_url)
    return image_url


def load_image(image_path: str) -> dict:
    response = requests.get(image_path)
    response.raise_for_status()

    image_base64 = base64.b64encode(response.content).decode("utf-8")

    html_image = f'<img src="data:image/png;base64,{image_base64}" alt="Generated Image" width="512"/>'

    return {"image": html_image}



def route(request: Request) -> str:
    from enrichment.prompt_enrichment import enrichment_chain
    chain = enrichment_chain | RunnableLambda(generate_image) | RunnableLambda(load_image)

    return chain


image_generator_chain = RunnableLambda(route)

Description

I am working on a text-to-image task using Langserve, and I am encountering an issue where the Langserve playground does not support displaying images directly as outputs. While Langchain excels at handling various multimodal tasks, this limitation in the playground environment makes it challenging to visualize image outputs in real-time, especially for models generating images from text prompts.
I want to convert the user's prompt into an image with Dall-E after enrichment it with another chain that I imported it.

System Info

pydantic > 2
langserve == 0.3.0

[kientv]

Why don't you use tool for drawing the image