Azure AI Search Multimodal RAG Demo

Table of Contents

• Azure AI Search Multimodal RAG Demo
• Azure AI Search Portal: Bring your own index and resources
• Getting Started
  • General Requirements
• Environment setup
  • Github codespaces
  • Local development setup (Windows or Linux)
  • Provision resources and deploy working app
  • Debug app locally
  • Bring your own data (supports .pdf only)
• Azure Services Used for Deployment
  • Role Mapping for the Application
• End-to-end app diagram
• Troubleshooting

Welcome to the Azure AI Search Multimodal RAG Demo. This repository contains the code for an application designed to showcase multimodal Retrieval-Augmented Generation (RAG) techniques using Azure AI Search. This demo combines AI capabilities to create custom copilots / RAG applications that can query, retrieve, and reason over both text and image data.

With multimodal RAG, you can:

• Extract relevant information from documents, screenshots, and visuals (like diagrams, charts, workflows, etc.).
• Preserve and understand the relationships between entities in complex images to enable reasoning over structured content.
• Generate grounded, accurate responses using Large Language Models (LLMs), integrating insights from both textual and visual modalities.

This demo is intentionally kept lean and simple, providing a hands-on experience with multimodal AI techniques. While not intended for production use, it serves as a powerful starting point for exploring how multimodal RAG can unlock new possibilities in building smarter, more context-aware applications.

Note that currently this sample doesn't have support for table extraction as a structure, but tables are extracted as plain text.

Text citations Image citations

Azure AI Search Portal: Bring your own index and resources

You can create an index using the AI Search portal's quick wizard for the multimodal scenario. Once the index is successfully created, you can integrate it with the app by running the following steps:

• Checkout a code space based on main branch

• Run az login --use-device-code

• Run

  scripts/portal-2-app.ps1 `
       -SearchIndexName "my-index" `
       -SearchServiceEndpoint "https://myservice.search.windows.net" `
       -StorageAccountUrl "https://myaccount.blob.core.windows.net" `
       -KnowledgeStoreContainerName "knowledgestore-artifacts" `
       -DataSourcesContainerName "data-sources" `
       -AzureOpenAiEndpoint "https://myopenai.openai.azure.com" `
       -AzureOpenAiDeploymentName "my-deployment" `
       -AzureOpenAiEndpointChatCompletionModelName "gpt-4o"

  Replace the placeholders (<...>) with your specific values. This script will configure the app to use the newly created index.
  Assumption: For app simplicity, ensure 'KnowledgeStoreContainerName' and 'DataSourcesContainerName' must be from same storage account.

• Ensure your Azure Entra ID user object ID has been granted the necessary permissions for all required resources. See Role Mapping for the Application for details.

• Run:

     src/start.sh

Getting Started

General Requirements

To deploy and run this application, you will need the following:

1. Azure Account

   • If you're new to Azure, you can sign up for a free Azure account and receive some free credits to get started.
   • Follow the guide to deploy using the free trial if applicable.

2. Azure Account Permissions

   • Your Azure account must have sufficient permissions to perform deployments. Specifically, you need:
     • Microsoft.Authorization/roleAssignments/write permissions, such as those granted by the Role Based Access Control (RBAC) Administrator, User Access Administrator, or Owner roles.
     • Subscription-level permissions. Alternatively, if you don’t have subscription-level permissions, you must be granted RBAC access for an existing resource group where you’ll deploy the application.
     • Microsoft.Resources/deployments/write permissions at the subscription level.

3. Local Deployment Environment (Optional)

   • If a local deployment of the application is required, ensure you have one of the following operating systems set up:
     • Windows OS
     • Linux OS

---

Environment setup

Github codespaces

• Checkout a code space based on main branch

Local development setup (Windows or Linux)

Install the below tools

• Python 3.12.7
• Node.js > v.18
• az cli latest
• azd latest
• Powershell 7 (Windows and Linux)

Provision resources and deploy working app

• Run below commands (One time setup per environment)
  • Run az login --use-device-code
  • Run azd auth login
  • Run azd env new <YOUR_ENVIRONMENT_NAME>
  • Run azd env set AZURE_PRINCIPAL_ID <USER_OBJECT_ID> (This needs to user's object ID from Azure Entra ID. Alternate you can use command from your local development box az ad signed-in-user show --query id -o tsv )
  • Run azd up. This command will
    • Provision the azure resources
    • Package the application
    • Injest data into azure search index
    • Deploy the working app to webApp services
  • NOTE: You might encounter provisioning errors on cohere. Please visit troubleshooting section for more details.
  • Once deployment succeeds, you can use the app.

NOTE: It may take 5-10 minutes after you see 'SUCCESS' for the application to be fully deployed. If you see a "Python Developer" welcome screen or an error page, then wait a bit and refresh the page.

Debug app locally

• You need to provision all the resources before your start to debug app locally

• To launch the app locally, run the below command. The website will open automatically and be served at localhost:5000.

• On Windows:

  src/start.ps1

• On Linux:

  src/start.sh

Bring your own data (supports .pdf only)

• To index your own data,
  • Place pdf's under /data folder
  • Run scripts\prepdocs.ps1
• You could also use different indexer strategies ["indexer-image-verbal", "self-multimodal-embedding"]
• To create new index with a different strategy
  • Run azd set SEARCH_INDEX_NAME <new-index-name>
  • On Windows Run scripts\prepdocs.ps1 -IndexerStrategy indexer-image-verbal
  • On Linux Run scripts\prepdocs.sh indexer-image-verbal

---

Azure Services Used for Deployment

The following Azure services are used as part of this deployment. Ensure you verify their billing and pricing details as part of the setup:

1. Azure AI Search

   • Service used for search functionalities within the application. Review pricing.

2. Azure AI Document Intelligence

   • Service used for processing and extracting information from documents. Review pricing.

3. Your provided:

   • LLM Deployment: For running the large language model (LLM) for verbalization and used by the RAG orchestrator.
   • Embedding Model Deployment: Used for creating embeddings for vector search and other tasks.
   • Ensure you check the pricing for both LLM and embedding deployments.
   • This sample currently supports gpt-4o, (AOAI) text-embedding-large, cohere-serverless-v3

4. Azure Blob Storage Account

   • Used to store extracted images and other data. Verify the pricing for storage and associated operations. Review pricing.

5. Azure App Service

   • Used to host and run the application in the cloud. Review pricing.

Role Mapping for the Application

The following table maps the roles used by the application to their respective functions:

Role ID	Built-in Role Name	Purpose
5e0bd9bd-7b93-4f28-af87-19fc36ad61bd	Cognitive Services OpenAI User	Read-only access to models, files, and deployments in an Azure OpenAI resource. Allows running completion/embedding/image-generation calls.
a97b65f3-24c7-4388-baec-2e87135dc908	Cognitive Services User	Provides read access to an Azure Cognitive Services resource and the ability to list its access keys. (No write or manage permissions.)
ba92f5b4-2d11-453d-a403-e96b0029c9fe	Storage Blob Data Contributor	Allows read, upload, modify, and delete operations on blobs and containers within an Azure Storage account (data-plane only).
7ca78c08-252a-4471-8644-bb5ff32d4ba0	Search Service Contributor	Enables management of the Azure Cognitive Search service (e.g., create, scale, delete). Does not provide access to index data itself.
8ebe5a00-799e-43f5-93ac-243d3dce84a7	Search Index Data Contributor	Provides full create, read, update, and delete access to all
64702f94-c441-49e6-a78b-ef80e0188fee	Azure AI Developer	Provides full create, read access to AI foundry projects.

End-to-end app diagram

Troubleshooting

• What is the region availability for Azure OpenAI service?
  Please visit available regions
• What is the region availability for Cohere Serverless?
  Cohere serverless is supported only in 5 regions
• Deployment fails for 'Cohere' in marketplace subscription
  • Ensure your subscription is supported or enabled for Marketplace deployment Learn more
  • There is a known issue of conflict operation between Marketplace subscription and endpoint deployment. Rerun deployment to fix it