v0.10.0

🧪 New Experiments

🖼️ Multimodal Text Generation

We are adding support for passing images in user messages and other multimodal features.

from haystack_experimental.dataclasses import ImageContent, ChatMessage
from haystack_experimental.components.generators.chat import OpenAIChatGenerator

image_url = "https://cdn.britannica.com/79/191679-050-C7114D2B/Adult-capybara.jpg"
image_content = ImageContent.from_url(image_url)

message = ChatMessage.from_user(
    content_parts=["Describe the image in short.", image_content]
)

llm = OpenAIChatGenerator(model="gpt-4o-mini")

print(llm.run([message])["replies"][0].text)

For the list of implemented features, see #302.

For more usage examples, check out the example: 📓 Introduction to Multimodal Text Generation.

Related PRs

• feat: ImageContent dataclass by @anakin87 in #286
• feat: Add ImageFileToImageContent and PDFToImageContent converters by @sjrl in #290
• feat: multimodal support in OpenAIChatGenerator by @anakin87 in #292
• chore: improve Image Converters pydoc config by @anakin87 in #295
• feat: add convenience class methods to Imagecontent by @anakin87 in #294
• chore: move ImageContent to a separate module by @anakin87 in #296
• feat: add Jinja2 ChatMessage extension by @anakin87 in #297
• feat: ImageContent visualization by @anakin87 in #300
• feat: extend ChatPromptBuilder to support string templates by @anakin87 in #299
• chore: update README with multimodal experiment by @anakin87 in #303
• fix: move IPython import by @anakin87 in #304
• feat: ImageContent validation by @anakin87 in #305

🐛 Bug Fixes

• fix: Update __init__.py to use double underscore by @sjrl in #288
• fix: preserve initialization parameters in debug state when run params are not supplied by @Amnah199 in #293

✅ Adopted Experiments

• chore: update/clean up experimental by @anakin87 in #285
• chore: Remove SuperComponent and pre-made super components. Update Readme by @sjrl in #287
• chore: remove dependencies needed for MultiFileConverter by @anakin87 in #298

Other Updates

• Update issue template for adding new experiments by @bilgeyucel in #283
• docs: add missing pydocs by @dfokina in #291

Full Changelog: v0.9.0...v0.10.0