Try out Model Router and Foundry Local with these simple Gradio samples

[guygregory]

Introduction

At Microsoft Build 2025, we announced a number of exciting new preview features, including Foundry Local and Model Router.

Foundry Local unlocks instant, on-device AI by allowing users to run generative AI models directly on their local machines, without relying on cloud-based inference. This approach offers enhanced privacy, lower latency, and greater control over AI workloads, empowering developers and organizations to build and experiment with AI locally.

The Model Router feature within Azure OpenAI is a deployable model trained to select the best large language model (LLM) to respond to a given prompt in real time. By assessing factors such as prompt complexity, cost, and performance, Model Router dynamically routes requests to the most suitable underlying model.

To accelerate adoption of these new features, I've included a couple of sample applications, which provide a simple Gradio front-end to make it easier to test and create custom demos:

Foundry Local samples
Model Router samples

Features and Screenshots

Screenshots:

Technical Details

Technologies Used ...

• Python
• Gradio
• Dotenv
• Azure OpenAI Service
• Foundry Local
• OpenAI Python SDK

Challenges and Solutions

Problems Encountered & Solutions Implemented ...

• Main challenge was around the rendering of LaTeX within Gradio - this probably needs some additional work, but the problem is purely cosmetic.
• I also had a challenge around optimum selection of models by the model router selection when used in a multi-turn conversation. For that reason, I changed the model router sample to remove the conversation history.

[amynic]

@guygregory I got a great question in Discord (https://aka.ms/azureaifoundry/discord) : what is the cost of the router model? if every request has to go through the router first.

Today:
When you use model router today, you're only billed for the use of the underlying models as they're recruited to respond to prompts: the model routing function itself doesn't incur any extra charges. Starting August 1, the model router usage will be charged as well.

You can monitor the costs of your model router deployment in the Azure portal.
https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/model-router

Thought that was worth sharing here 👍

[guygregory]

Good catch, thanks for sharing!