Skip to content

Commit 88c201f

Browse files
behroozazarkhalilibehroozazarkhalili
committed
Update DSPy GEPA notebook with inline resource links
- Add resource links in introduction section - Link to DSPy, NuminaMath dataset, and OpenRouter - Add GEPA optimizer documentation link
1 parent eb4e702 commit 88c201f

File tree

1 file changed

+147
-11
lines changed

1 file changed

+147
-11
lines changed

‎notebooks/en/dspy_gepa.ipynb‎

Lines changed: 147 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,21 +4,61 @@
44
"cell_type": "markdown",
55
"id": "5aa65d7d",
66
"metadata": {},
7-
"source": "# Optimizing Language Models with DSPy GEPA\n\n_Authored by: [Behrooz Azarkhalili](https://github.com/behroozazarkhalili)_\n\nThis notebook demonstrates how to use [DSPy](https://dspy.ai/)'s GEPA (Generalized Error-driven Prompt Augmentation) optimizer to improve language model performance on mathematical reasoning tasks. We'll work with the [NuminaMath-1.5 dataset](https://huggingface.co/datasets/AI-MO/NuminaMath-1.5) and show how GEPA can boost accuracy through automated prompt optimization.\n\n**What you'll learn:**\n- Setting up DSPy with language models ([OpenRouter](https://openrouter.ai/)) \n- Processing and filtering mathematical problem datasets\n- Building a baseline Chain-of-Thought reasoning program\n- Optimizing prompts with GEPA using error-driven feedback\n- Evaluating improvements in model accuracy\n\n\nGEPA works by analyzing errors, generating targeted feedback, and automatically refining prompts to address common failure patterns. This makes it particularly effective for complex reasoning tasks where prompt quality significantly impacts performance.\n\n**Key Resources:**\n- [DSPy Documentation](https://dspy.ai/learn/programming/)\n- [Chain-of-Thought Prompting Paper](https://arxiv.org/abs/2201.11903)\n- [GEPA Optimizer Guide](https://dspy.ai/api/optimizers/GEPA/)"
7+
"source": [
8+
"# Prompt Optimization for Language Models with DSPy GEPA\n",
9+
"\n",
10+
"_Authored by: [Behrooz Azarkhalili](https://github.com/behroozazarkhalili)_\n",
11+
"\n",
12+
"This notebook demonstrates how to use [DSPy](https://dspy.ai/)'s GEPA (Generalized Error-driven Prompt Augmentation) optimizer to improve language model performance on mathematical reasoning tasks. We'll work with the [NuminaMath-1.5 dataset](https://huggingface.co/datasets/AI-MO/NuminaMath-1.5) and show how GEPA can boost accuracy through automated prompt optimization.\n",
13+
"\n",
14+
"**What you'll learn:**\n",
15+
"- Setting up DSPy with language models ([OpenRouter](https://openrouter.ai/)) \n",
16+
"- Processing and filtering mathematical problem datasets\n",
17+
"- Building a baseline Chain-of-Thought reasoning program\n",
18+
"- Optimizing prompts with GEPA using error-driven feedback\n",
19+
"- Evaluating improvements in model accuracy\n",
20+
"\n",
21+
"\n",
22+
"GEPA works by analyzing errors, generating targeted feedback, and automatically refining prompts to address common failure patterns. This makes it particularly effective for complex reasoning tasks where prompt quality significantly impacts performance.\n",
23+
"\n",
24+
"**Key Resources:**\n",
25+
"- [DSPy Documentation](https://dspy.ai/learn/programming/)\n",
26+
"- [Chain-of-Thought Prompting Paper](https://arxiv.org/abs/2201.11903)\n",
27+
"- [GEPA Optimizer Guide](https://dspy.ai/api/optimizers/GEPA/)"
28+
]
829
},
930
{
1031
"cell_type": "markdown",
1132
"id": "99b369f9",
1233
"metadata": {},
13-
"source": "## Installation and Setup\n\nInstall required dependencies and import libraries for DSPy, dataset processing, and model configuration.\n\n**Installation Options:**\n- **uv** - Fast Python package installer ([documentation](https://docs.astral.sh/uv/))\n- **pip** - Traditional Python package manager\n\n**Key Dependencies:**\n- `dspy` - DSPy framework for language model programming\n- `datasets` - Hugging Face datasets library for loading NuminaMath-1.5\n- `python-dotenv` - Environment variable management for API keys"
34+
"source": [
35+
"## Installation and Setup\n",
36+
"\n",
37+
"Install required dependencies and import libraries for DSPy, dataset processing, and model configuration.\n",
38+
"\n",
39+
"**Installation Options:**\n",
40+
"- **uv** - Fast Python package installer ([documentation](https://docs.astral.sh/uv/))\n",
41+
"- **pip** - Traditional Python package manager\n",
42+
"\n",
43+
"**Key Dependencies:**\n",
44+
"- `dspy` - DSPy framework for language model programming\n",
45+
"- `datasets` - Hugging Face datasets library for loading NuminaMath-1.5\n",
46+
"- `python-dotenv` - Environment variable management for API keys"
47+
]
1448
},
1549
{
1650
"cell_type": "code",
51+
"execution_count": null,
1752
"id": "6lfe42g2q12",
18-
"source": "# Install with uv (recommended - faster)\n!uv pip install dspy datasets python-dotenv\n\n# Alternative: Install with pip\n# !pip install dspy datasets python-dotenv",
1953
"metadata": {},
20-
"execution_count": null,
21-
"outputs": []
54+
"outputs": [],
55+
"source": [
56+
"# Install with uv (recommended - faster)\n",
57+
"!uv pip install dspy datasets python-dotenv\n",
58+
"\n",
59+
"# Alternative: Install with pip\n",
60+
"# !pip install dspy datasets python-dotenv"
61+
]
2262
},
2363
{
2464
"cell_type": "code",
@@ -99,8 +139,43 @@
99139
{
100140
"cell_type": "markdown",
101141
"id": "f2i6lg4sa0o",
102-
"source": "### Understanding GEPA's Two-Model Architecture\n\nGEPA's breakthrough innovation lies in its **dual-model approach** for reflective prompt optimization, which fundamentally differs from traditional single-model optimizers.\n\n**Why Two Models?**\n\nTraditional prompt optimizers rely on scalar metrics (accuracy scores) to guide improvements, essentially using trial-and-error without understanding *why* predictions fail. GEPA introduces a revolutionary approach by separating concerns:\n\n**1. Student LM (Inference Model)**\n- **Role**: Primary model that executes tasks and generates predictions\n- **Characteristics**: Fast, cost-efficient, handles high-volume inference\n- **Usage Pattern**: ~90-95% of all API calls during optimization\n- **In This Notebook**: `openrouter/openai/gpt-4.1-nano`\n\n**2. Reflection LM (Meta-Cognitive Model)**\n- **Role**: Analyzes failures, identifies patterns, and generates prompt improvements\n- **Characteristics**: Stronger reasoning, analytical depth, interpretability\n- **Usage Pattern**: ~5-10% of API calls (only during reflection phases)\n- **In This Notebook**: `openrouter/qwen/qwen3-next-80b-a3b-thinking`\n\n**The Reflective Optimization Cycle:**\n\n```\n1. Student LM solves training problems → predictions\n2. Metric provides rich textual feedback on failures\n3. Reflection LM analyzes batches of failures → identifies patterns\n4. Reflection LM generates improved prompt instructions\n5. Student LM tests new prompts → validation\n6. Repeat until convergence\n```\n\n**Research Foundation:**\n\nThis approach is detailed in the paper [\"Reflective Prompt Evolution Can Outperform Reinforcement Learning\"](https://arxiv.org/abs/2507.19457), which demonstrates that reflective optimization with textual feedback outperforms reinforcement learning approaches on complex reasoning tasks.",
103-
"metadata": {}
142+
"metadata": {},
143+
"source": [
144+
"### Understanding GEPA's Two-Model Architecture\n",
145+
"\n",
146+
"GEPA's breakthrough innovation lies in its **dual-model approach** for reflective prompt optimization, which fundamentally differs from traditional single-model optimizers.\n",
147+
"\n",
148+
"**Why Two Models?**\n",
149+
"\n",
150+
"Traditional prompt optimizers rely on scalar metrics (accuracy scores) to guide improvements, essentially using trial-and-error without understanding *why* predictions fail. GEPA introduces a revolutionary approach by separating concerns:\n",
151+
"\n",
152+
"**1. Student LM (Inference Model)**\n",
153+
"- **Role**: Primary model that executes tasks and generates predictions\n",
154+
"- **Characteristics**: Fast, cost-efficient, handles high-volume inference\n",
155+
"- **Usage Pattern**: ~90-95% of all API calls during optimization\n",
156+
"- **In This Notebook**: `openrouter/openai/gpt-4.1-nano`\n",
157+
"\n",
158+
"**2. Reflection LM (Meta-Cognitive Model)**\n",
159+
"- **Role**: Analyzes failures, identifies patterns, and generates prompt improvements\n",
160+
"- **Characteristics**: Stronger reasoning, analytical depth, interpretability\n",
161+
"- **Usage Pattern**: ~5-10% of API calls (only during reflection phases)\n",
162+
"- **In This Notebook**: `openrouter/qwen/qwen3-next-80b-a3b-thinking`\n",
163+
"\n",
164+
"**The Reflective Optimization Cycle:**\n",
165+
"\n",
166+
"```\n",
167+
"1. Student LM solves training problems → predictions\n",
168+
"2. Metric provides rich textual feedback on failures\n",
169+
"3. Reflection LM analyzes batches of failures → identifies patterns\n",
170+
"4. Reflection LM generates improved prompt instructions\n",
171+
"5. Student LM tests new prompts → validation\n",
172+
"6. Repeat until convergence\n",
173+
"```\n",
174+
"\n",
175+
"**Research Foundation:**\n",
176+
"\n",
177+
"This approach is detailed in the paper [\"Reflective Prompt Evolution Can Outperform Reinforcement Learning\"](https://arxiv.org/abs/2507.19457), which demonstrates that reflective optimization with textual feedback outperforms reinforcement learning approaches on complex reasoning tasks."
178+
]
104179
},
105180
{
106181
"cell_type": "code",
@@ -2217,13 +2292,74 @@
22172292
"cell_type": "markdown",
22182293
"id": "skmsf5j36v",
22192294
"metadata": {},
2220-
"source": "### Understanding the Optimization Results\n\n**Performance Improvement:**\n- **Baseline Accuracy**: 52.2% (47/90 correct)\n- **Optimized Accuracy**: 57.8% (52/90 correct)\n- **Improvement**: +5.6 percentage points (~11% relative improvement)\n\n**What Changed:**\nSee the instruction GEPA developed above.\n\n**Why the Modest Improvement?**\n\nThe ~6% gain is expected given:\n1. **Small Training Set**: Only 112 training examples (0.025% of full dataset)\n2. **Light Optimization**: Using `auto=\"light\"` for faster iteration\n3. **Simple Baseline**: Chain-of-Thought already provides decent reasoning structure\n4. **Model Limitations**: GPT-4.1 Nano's mathematical capabilities are the ceiling\n\n**Cost Efficiency:**\n\nThis entire experiment (baseline evaluation, GEPA optimization, and final evaluation on 224 examples) cost **less than $0.50** thanks to:\n- GPT-4.1 Nano's low pricing ($0.10/M input, $0.40/M output)\n- Asymmetric architecture (cheap model for 99% of calls, smart model for 1%)\n- Small sample size for demonstration purposes\n\n**Key Takeaway:**\n\nEven with limited data and light optimization, GEPA successfully identified failure patterns and generated targeted prompt improvements. With more training data (`sample_fraction=0.01` or higher) and heavier optimization (`auto=\"medium\"` or `\"heavy\"`), we'd expect 15-25% improvements, potentially reaching 65-70% accuracy."
2295+
"source": [
2296+
"### Understanding the Optimization Results\n",
2297+
"\n",
2298+
"**Performance Improvement:**\n",
2299+
"- **Baseline Accuracy**: 52.2% (47/90 correct)\n",
2300+
"- **Optimized Accuracy**: 57.8% (52/90 correct)\n",
2301+
"- **Improvement**: +5.6 percentage points (~11% relative improvement)\n",
2302+
"\n",
2303+
"**What Changed:**\n",
2304+
"See the instruction GEPA developed above.\n",
2305+
"\n",
2306+
"**Why the Modest Improvement?**\n",
2307+
"\n",
2308+
"The ~6% gain is expected given:\n",
2309+
"1. **Small Training Set**: Only 112 training examples (0.025% of full dataset)\n",
2310+
"2. **Light Optimization**: Using `auto=\"light\"` for faster iteration\n",
2311+
"3. **Simple Baseline**: Chain-of-Thought already provides decent reasoning structure\n",
2312+
"4. **Model Limitations**: GPT-4.1 Nano's mathematical capabilities are the ceiling\n",
2313+
"\n",
2314+
"**Cost Efficiency:**\n",
2315+
"\n",
2316+
"This entire experiment (baseline evaluation, GEPA optimization, and final evaluation on 224 examples) cost **less than $0.50** thanks to:\n",
2317+
"- GPT-4.1 Nano's low pricing ($0.10/M input, $0.40/M output)\n",
2318+
"- Asymmetric architecture (cheap model for 99% of calls, smart model for 1%)\n",
2319+
"- Small sample size for demonstration purposes\n",
2320+
"\n",
2321+
"**Key Takeaway:**\n",
2322+
"\n",
2323+
"Even with limited data and light optimization, GEPA successfully identified failure patterns and generated targeted prompt improvements. With more training data (`sample_fraction=0.01` or higher) and heavier optimization (`auto=\"medium\"` or `\"heavy\"`), we'd expect 15-25% improvements, potentially reaching 65-70% accuracy."
2324+
]
22212325
},
22222326
{
22232327
"cell_type": "markdown",
22242328
"id": "cuj307bhp8f",
2225-
"source": "## Learn More\n\nThis notebook introduced DSPy's GEPA optimizer for automated prompt improvement. Here are additional resources to deepen your understanding:\n\n### DSPy Framework\n- **[DSPy Documentation](https://dspy.ai/)** - Official documentation and guides\n- **[DSPy GitHub Repository](https://github.com/stanfordnlp/dspy)** - Source code and examples\n- **[DSPy Research Paper](https://arxiv.org/abs/2310.03714)** - \"DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines\"\n- **[DSPy Tutorial Series](https://dspy.ai/learn/programming/)** - Step-by-step learning path\n\n### Prompt Optimization\n- **[GEPA Optimizer Documentation](https://dspy.ai/api/optimizers/GEPA/)** - Technical details on GEPA\n- **[Chain-of-Thought Prompting](https://arxiv.org/abs/2201.11903)** - Foundational paper on CoT reasoning\n- **[Automatic Prompt Engineering](https://arxiv.org/abs/2211.01910)** - \"Large Language Models Are Human-Level Prompt Engineers\"\n- **[DSPy Optimizers Comparison](https://dspy.ai/api/optimizers/)** - Overview of different optimization strategies\n\n### Mathematical Reasoning\n- **[NuminaMath Dataset](https://huggingface.co/datasets/AI-MO/NuminaMath-1.5)** - The dataset used in this notebook\n- **[GSM8K Dataset](https://huggingface.co/datasets/gsm8k)** - Grade school math word problems benchmark\n- **[MATH Dataset](https://huggingface.co/datasets/hendrycks/competition_math)** - Competition-level mathematics problems\n- **[Mathematical Reasoning with LLMs](https://arxiv.org/abs/2206.14858)** - Survey of techniques\n\n### Related Techniques\n- **[Few-Shot Learning](https://arxiv.org/abs/2005.14165)** - \"Language Models are Few-Shot Learners\" (GPT-3 paper)\n- **[Self-Consistency](https://arxiv.org/abs/2203.11171)** - Improving reasoning via multiple sampling paths\n- **[ReAct Prompting](https://arxiv.org/abs/2210.03629)** - Reasoning and Acting in language models\n\n### Tools and Platforms\n- **[OpenRouter](https://openrouter.ai/)** - Unified API for multiple LLM providers\n- **[Hugging Face Datasets](https://huggingface.co/docs/datasets/)** - Dataset loading and processing\n- **[DSPy Optimizers Guide](https://dspy.ai/deep-dive/optimizers/)** - Deep dive into optimization strategies",
2226-
"metadata": {}
2329+
"metadata": {},
2330+
"source": [
2331+
"## Learn More\n",
2332+
"\n",
2333+
"This notebook introduced DSPy's GEPA optimizer for automated prompt improvement. Here are additional resources to deepen your understanding:\n",
2334+
"\n",
2335+
"### DSPy Framework\n",
2336+
"- **[DSPy Documentation](https://dspy.ai/)** - Official documentation and guides\n",
2337+
"- **[DSPy GitHub Repository](https://github.com/stanfordnlp/dspy)** - Source code and examples\n",
2338+
"- **[DSPy Research Paper](https://arxiv.org/abs/2310.03714)** - \"DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines\"\n",
2339+
"- **[DSPy Tutorial Series](https://dspy.ai/learn/programming/)** - Step-by-step learning path\n",
2340+
"\n",
2341+
"### Prompt Optimization\n",
2342+
"- **[GEPA Optimizer Documentation](https://dspy.ai/api/optimizers/GEPA/)** - Technical details on GEPA\n",
2343+
"- **[Chain-of-Thought Prompting](https://arxiv.org/abs/2201.11903)** - Foundational paper on CoT reasoning\n",
2344+
"- **[Automatic Prompt Engineering](https://arxiv.org/abs/2211.01910)** - \"Large Language Models Are Human-Level Prompt Engineers\"\n",
2345+
"- **[DSPy Optimizers Comparison](https://dspy.ai/api/optimizers/)** - Overview of different optimization strategies\n",
2346+
"\n",
2347+
"### Mathematical Reasoning\n",
2348+
"- **[NuminaMath Dataset](https://huggingface.co/datasets/AI-MO/NuminaMath-1.5)** - The dataset used in this notebook\n",
2349+
"- **[GSM8K Dataset](https://huggingface.co/datasets/gsm8k)** - Grade school math word problems benchmark\n",
2350+
"- **[MATH Dataset](https://huggingface.co/datasets/hendrycks/competition_math)** - Competition-level mathematics problems\n",
2351+
"- **[Mathematical Reasoning with LLMs](https://arxiv.org/abs/2206.14858)** - Survey of techniques\n",
2352+
"\n",
2353+
"### Related Techniques\n",
2354+
"- **[Few-Shot Learning](https://arxiv.org/abs/2005.14165)** - \"Language Models are Few-Shot Learners\" (GPT-3 paper)\n",
2355+
"- **[Self-Consistency](https://arxiv.org/abs/2203.11171)** - Improving reasoning via multiple sampling paths\n",
2356+
"- **[ReAct Prompting](https://arxiv.org/abs/2210.03629)** - Reasoning and Acting in language models\n",
2357+
"\n",
2358+
"### Tools and Platforms\n",
2359+
"- **[OpenRouter](https://openrouter.ai/)** - Unified API for multiple LLM providers\n",
2360+
"- **[Hugging Face Datasets](https://huggingface.co/docs/datasets/)** - Dataset loading and processing\n",
2361+
"- **[DSPy Optimizers Guide](https://dspy.ai/deep-dive/optimizers/)** - Deep dive into optimization strategies"
2362+
]
22272363
}
22282364
],
22292365
"metadata": {
@@ -2252,4 +2388,4 @@
22522388
},
22532389
"nbformat": 4,
22542390
"nbformat_minor": 5
2255-
}
2391+
}

0 commit comments

Comments
 (0)