Foundry Local AMA

[leestott]

Azure AI Foundry Discord Community AMA on Foundry Local 12th June 2025.

The following is a summary of the discussions, questions and answers that were discussed.

🔍 What is Foundry Local?

Foundry Local is a local version of Azure AI Foundry that enables developers to run large language models (LLMs) directly on their Windows or macOS devices. It supports on-device inference, agent-based workflows, and model management via CLI and REST APIs.

---

🧠 Key Features & Capabilities

✅ Local Model Execution

• Run LLMs locally for privacy, cost-efficiency, and offline availability — Foundry Local Docs and Resources
• Supports CPU, GPU, and NPU execution via ONNX Runtime.
• CLI tools foundry model list and foundry cache list help manage models.
• Installation (Windows): winget install Microsoft.FoundryLocal
• Installation (macOS):

  brew tap microsoft/foundrylocal
  brew install foundrylocal

---

✅ Minimum System Requirements

Component	Requirement
Operating System	Windows 10 (x64), Windows 11 (x64/ARM), Windows Server 2025, macOS
RAM	Minimum: 8 GB • Recommended: 16 GB
Disk Space	Minimum: 15 GB • Recommended: 30 GB
Network	Internet connection required for initial model download (optional afterward)
Permissions	Administrative privileges required for installation

Foundry Local will download the appropriate model variant (CPU, GPU, or NPU) based on your system configuration automatically.

---

🚀 Optional Hardware Acceleration

Foundry Local supports automatic model variant selection based on your system. For best performance:

Accelerator Type	Supported Hardware
GPU (CUDA)	NVIDIA 2000 series or newer
NPU	Snapdragon X Elite
Apple Silicon	M1, M2, M3, M4 (via WebGPU)
CPU	Supported fallback when no accelerator is available

---

✅ Hardware Acceleration

ONNX Runtime Execution Providers supported:

• CUDA – for NVIDIA GPUs
• WebGPU – for Apple Silicon

Performance: Up to 6× faster than raw PyTorch and 20% faster than Ollama on Windows 11.

✅ ONNX Optimization

• Uses ONNX for portability and optimization.
• Supports quantized models (e.g., INT8) for faster inference.

---

🛠️ Developer Lifecycle with Foundry Local

1. Install Foundry Local
   Get Started Guide

2. Model Discovery
   Use foundry model list

3. Model Caching
   Use foundry cache list

4. Inference
   Run models via CLI or REST API

5. Agent Development
   Use the Azure AI Foundry VS Code extension

6. Local to Network Inference
   Switch seamlessly to cloud endpoints when ready

---

🧩 Integration with MS Dev Proxy

To publicly test Foundry Local endpoints:

• Use Microsoft Dev Proxy to forward local ports
• Example:

  devproxy --forward http://localhost:5000

---

🧪 Foundry Local vs. Azure AI Foundry

Feature	Foundry Local	Azure AI Foundry
Deployment	On-device (Windows/macOS)	Cloud-based
Model Access	Local + Azure Catalog	Azure-hosted only
Hardware Support	CPU, GPU, NPU	CPU, GPU
Use Case	Edge AI, offline, privacy	Scalable cloud AI

---

🤖 Foundry Local vs. Ollama

Feature	Foundry Local	Ollama
Platform	Windows/macOS	macOS/Linux/Windows
Model Format	ONNX	GGUF
Hardware Support	CPU, GPU, NPU	CPU, GPU
Performance	Up to 6× faster than PyTorch	Optimized for Apple Silicon
Integration	CLI, VS Code SDK	CLI only

---

🧠 Foundry Local vs. Windows AI Foundry vs. Windows ML

A breakdown of Microsoft’s local AI tooling stack and how each component fits into the developer and platform ecosystem.

---

🔍 Overview

Foundry Local

• Cross-platform runtime (Windows/macOS) for running LLMs locally.
• Supports CPU, GPU, and NPU acceleration.
• CLI, REST API, and SDKs (Python, JS, C#) for integration.
• Ideal for edge AI, offline scenarios, and privacy-first applications.

Windows AI Foundry

• Windows-specific packaging of Foundry Local.
• Integrated with Windows 11 and Copilot+ PCs.
• Includes access to Windows ML APIs and Phi Silica models.
• Enables hybrid AI workflows (local + cloud).

Windows ML

• Native Windows runtime for machine learning.
• Provides APIs for AI tasks like OCR, image recognition, and background removal.
• Wraps ONNX Runtime and DirectML for hardware acceleration.
• Supports model optimization via Olive.

---

🧩 Architecture Relationship

Layer	Component	Role
Cloud	Azure AI Foundry	Model catalog, training, fine-tuning, cloud inference
Platform	Windows AI Foundry	Windows-integrated AI runtime
Runtime	Foundry Local	Local inference engine (cross-platform)
API Layer	Windows ML	Windows-native APIs for AI tasks

---

🔄 Feature Comparison

Feature	Foundry Local	Windows AI Foundry	Windows ML
Platform	Windows/macOS	Windows 11 only	Windows 10/11
Use Case	Local AI apps	Windows-integrated AI	ML API access
Model Sources	Azure Foundry, Hugging Face	Azure Foundry, Copilot	ONNX, Olive
Hardware Support	CPU, GPU, NPU	CPU, GPU, NPU	CPU, GPU, NPU
Interfaces	CLI, REST, SDKs	Windows APIs, VS Code	Windows APIs
Integration	VS Code, GitHub, CLI	Windows Shell, Copilot	Windows apps

---

🧠 Developer Guidance

• Use Foundry Local if you're building cross-platform apps or want full control over local inference.
• Use Windows AI Foundry if you're targeting Windows 11 and want seamless integration with the OS and Copilot.
• Use Windows ML if you're building native Windows apps and want to call AI tasks via system APIs.

If you're building something today, Foundry Local is the most flexible for prototyping and privacy, while Windows AI Foundry gives you the full access to AI PC + Windows Phi Silica and Windows AI PC APIs

🧠 Agent Mode & Preview Signup

Foundry Local Agent allows developers to easily create and run local agent using local models and MCP servers.

• Sign up for Agent Mode Preview
• Download AI Toolkit Extension

---

💡 Tips & Tricks

• 🧪 Use quantized ONNX models (INT8) for faster inference
• 🧰 Foundry Local uses:
  • ~/.foundry for model storage
  • ~/.aitk for diagnostics
• 🧼 Use foundry service diag --logs for debugging
• 🧠 Use the VS Code extension to build, test, and deploy agents