Skip to content

[Bug] Setting max_bootstrapped_demos and max_labeled_demos to 0 still results in few shot examples being generated for MiproV2 #8508

New issue
New issue
Open
#8909
Open
[Bug] Setting max_bootstrapped_demos and max_labeled_demos to 0 still results in few shot examples being generated for MiproV2#8508
#8909
Assignees
isaacbmiller
Labels
bugSomething isn't working
@ejing42

Description

@ejing42
ejing42
opened on Jul 8, 2025

What happened?

I'm trying to do 0-shot instruction optimization using MiproV2, but even if I set max_bootstrapped_demos=0 and max_labeled_demos=0, the optimizer still generates few-shot examples and use them for optimization. Example:


optimized_pipeline = optimizer.compile(AnswerResearch(), trainset=train,
                           max_bootstrapped_demos=0, max_labeled_demos=0,
                           requires_permission_to_run=False)

RUNNING WITH THE FOLLOWING MEDIUM AUTO RUN SETTINGS:
num_trials: 18
minibatch: False
num_fewshot_candidates: 12
num_instruct_candidates: 6
valset size: 11

2025[/07/08](http://localhost:8888/07/08) 10:08:54 INFO dspy.teleprompt.mipro_optimizer_v2: 
==> STEP 1: BOOTSTRAP FEWSHOT EXAMPLES <==
2025[/07/08](http://localhost:8888/07/08) 10:08:54 INFO dspy.teleprompt.mipro_optimizer_v2: These will be used for informing instruction proposal.

2025[/07/08](http://localhost:8888/07/08) 10:08:54 INFO dspy.teleprompt.mipro_optimizer_v2: Bootstrapping N=12 sets of demonstrations...
Bootstrapping set 1/12
Bootstrapping set 2/12
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 3[/3](http://localhost:8888/3) [00:04<00:00,  1.67s[/it](http://localhost:8888/it)]
Bootstrapped 3 full traces after 2 examples for up to 1 rounds, amounting to 3 attempts.
Bootstrapping set 3/12
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 3[/3](http://localhost:8888/3) [00:09<00:00,  3.15s[/it](http://localhost:8888/it)]
Bootstrapped 3 full traces after 2 examples for up to 1 rounds, amounting to 3 attempts.
Bootstrapping set 4/12
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 3[/3](http://localhost:8888/3) [00:11<00:00,  3.68s[/it](http://localhost:8888/it)]
Bootstrapped 3 full traces after 2 examples for up to 1 rounds, amounting to 3 attempts.
Bootstrapping set 5/12
 33%|███████████████████████████████████                                                                      | 1[/3](http://localhost:8888/3) [00:02<00:04,  2.03s[/it](http://localhost:8888/it)]
Bootstrapped 1 full traces after 1 examples for up to 1 rounds, amounting to 1 attempts.
Bootstrapping set 6/12
 33%|███████████████████████████████████                                                                      | 1[/3](http://localhost:8888/3) [00:01<00:02,  1.50s[/it](http://localhost:8888/it)]
Bootstrapped 1 full traces after 1 examples for up to 1 rounds, amounting to 1 attempts.
Bootstrapping set 7/12
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 3[/3](http://localhost:8888/3) [00:06<00:00,  2.02s[/it](http://localhost:8888/it)]
Bootstrapped 3 full traces after 2 examples for up to 1 rounds, amounting to 3 attempts.
Bootstrapping set 8/12
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 3[/3](http://localhost:8888/3) [00:04<00:00,  1.56s[/it](http://localhost:8888/it)]
Bootstrapped 3 full traces after 2 examples for up to 1 rounds, amounting to 3 attempts.
Bootstrapping set 9/12
 67%|██████████████████████████████████████████████████████████████████████                                   | 2[/3](http://localhost:8888/3) [00:02<00:01,  1.46s[/it](http://localhost:8888/it)]
Bootstrapped 2 full traces after 2 examples for up to 1 rounds, amounting to 2 attempts.
Bootstrapping set 10/12
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 3[/3](http://localhost:8888/3) [00:04<00:00,  1.57s[/it](http://localhost:8888/it)]
Bootstrapped 3 full traces after 2 examples for up to 1 rounds, amounting to 3 attempts.
Bootstrapping set 11/12
 67%|██████████████████████████████████████████████████████████████████████                                   | 2[/3](http://localhost:8888/3) [00:05<00:02,  2.58s[/it](http://localhost:8888/it)]
Bootstrapped 2 full traces after 2 examples for up to 1 rounds, amounting to 2 attempts.
Bootstrapping set 12/12
 67%|██████████████████████████████████████████████████████████████████████                                   | 2[/3](http://localhost:8888/3) [00:03<00:01,  1.73s[/it](http://localhost:8888/it)]
2025[/07/08](http://localhost:8888/07/08) 10:09:50 INFO dspy.teleprompt.mipro_optimizer_v2: 
==> STEP 2: PROPOSE INSTRUCTION CANDIDATES <==
2025[/07/08](http://localhost:8888/07/08) 10:09:50 INFO dspy.teleprompt.mipro_optimizer_v2: We will use the few-shot examples from the previous step, a generated dataset summary, a summary of the program code, and a randomly selected prompting tip to propose instructions.
2025[/07/08](http://localhost:8888/07/08) 10:09:50 INFO dspy.teleprompt.mipro_optimizer_v2: 
Proposing N=6 instructions...

Bootstrapped 2 full traces after 2 examples for up to 1 rounds, amounting to 2 attempts.
2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: Proposed Instructions for Predictor 0:

2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: 0: Given the fields `question`, `context`, `answer_criteria`, produce the fields `answer`.

2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: 1: <redacted>

2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: 2: <redacted>

2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: 3: <redacted>

2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: 4: <redacted>

2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: 5: <redacted>

2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: 

2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: ==> STEP 3: FINDING OPTIMAL PROMPT PARAMETERS <==
2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: We will evaluate the program over a series of trials with different combinations of instructions and few-shot examples to find the optimal combination using Bayesian Optimization.

2025[/07/08](http://localhost:8888/07/08) 10:10:53 INFO dspy.teleprompt.mipro_optimizer_v2: == Trial 1 [/](http://localhost:8888/) 18 - Full Evaluation of Default Program ==
Average Metric: 5.55 / 11 (50.5%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:01<00:00,  5.96it/s]
2025[/07/08](http://localhost:8888/07/08) 10:10:55 INFO dspy.evaluate.evaluate: Average Metric: 5.55 [/](http://localhost:8888/) 11 (50.5%)
2025[/07/08](http://localhost:8888/07/08) 10:10:55 INFO dspy.teleprompt.mipro_optimizer_v2: Default program score: 50.45

2025[/07/08](http://localhost:8888/07/08) 10:10:55 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 2 [/](http://localhost:8888/) 18 =====

Average Metric: 9.50 / 11 (86.4%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:05<00:00,  1.84it/s]
2025[/07/08](http://localhost:8888/07/08) 10:11:01 INFO dspy.evaluate.evaluate: Average Metric: 9.5 [/](http://localhost:8888/) 11 (86.4%)
2025[/07/08](http://localhost:8888/07/08) 10:11:01 INFO dspy.teleprompt.mipro_optimizer_v2: Best full score so far! Score: 86.36
2025[/07/08](http://localhost:8888/07/08) 10:11:01 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 86.36 with parameters ['Predictor 0: Instruction 1', 'Predictor 0: Few-Shot Set 6'].
2025[/07/08](http://localhost:8888/07/08) 10:11:01 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36]
2025[/07/08](http://localhost:8888/07/08) 10:11:01 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 86.36
2025[/07/08](http://localhost:8888/07/08) 10:11:01 INFO dspy.teleprompt.mipro_optimizer_v2: ========================


2025[/07/08](http://localhost:8888/07/08) 10:11:01 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 3 [/](http://localhost:8888/) 18 =====

Average Metric: 9.05 / 11 (82.3%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:07<00:00,  1.51it/s]
2025[/07/08](http://localhost:8888/07/08) 10:11:08 INFO dspy.evaluate.evaluate: Average Metric: 9.05 [/](http://localhost:8888/) 11 (82.3%)
2025[/07/08](http://localhost:8888/07/08) 10:11:08 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 82.27 with parameters ['Predictor 0: Instruction 4', 'Predictor 0: Few-Shot Set 2'].
2025[/07/08](http://localhost:8888/07/08) 10:11:08 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27]
2025[/07/08](http://localhost:8888/07/08) 10:11:08 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 86.36
2025[/07/08](http://localhost:8888/07/08) 10:11:08 INFO dspy.teleprompt.mipro_optimizer_v2: ========================


2025[/07/08](http://localhost:8888/07/08) 10:11:08 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 4 [/](http://localhost:8888/) 18 =====

Average Metric: 6.80 / 11 (61.8%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:15<00:00,  1.38s/it]
2025[/07/08](http://localhost:8888/07/08) 10:11:23 INFO dspy.evaluate.evaluate: Average Metric: 6.8 [/](http://localhost:8888/) 11 (61.8%)
2025[/07/08](http://localhost:8888/07/08) 10:11:24 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 61.82 with parameters ['Predictor 0: Instruction 0', 'Predictor 0: Few-Shot Set 6'].
2025[/07/08](http://localhost:8888/07/08) 10:11:24 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82]
2025[/07/08](http://localhost:8888/07/08) 10:11:24 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 86.36
2025[/07/08](http://localhost:8888/07/08) 10:11:24 INFO dspy.teleprompt.mipro_optimizer_v2: ========================


2025[/07/08](http://localhost:8888/07/08) 10:11:24 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 5 [/](http://localhost:8888/) 18 =====

Average Metric: 8.50 / 11 (77.3%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:09<00:00,  1.20it/s]
2025[/07/08](http://localhost:8888/07/08) 10:11:33 INFO dspy.evaluate.evaluate: Average Metric: 8.5 [/](http://localhost:8888/) 11 (77.3%)
2025[/07/08](http://localhost:8888/07/08) 10:11:33 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 77.27 with parameters ['Predictor 0: Instruction 2', 'Predictor 0: Few-Shot Set 4'].
2025[/07/08](http://localhost:8888/07/08) 10:11:33 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27]
2025[/07/08](http://localhost:8888/07/08) 10:11:33 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 86.36
2025[/07/08](http://localhost:8888/07/08) 10:11:33 INFO dspy.teleprompt.mipro_optimizer_v2: ========================


2025[/07/08](http://localhost:8888/07/08) 10:11:33 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 6 [/](http://localhost:8888/) 18 =====

Average Metric: 9.65 / 11 (87.7%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:09<00:00,  1.19it/s]
2025[/07/08](http://localhost:8888/07/08) 10:11:42 INFO dspy.evaluate.evaluate: Average Metric: 9.65 [/](http://localhost:8888/) 11 (87.7%)
2025[/07/08](http://localhost:8888/07/08) 10:11:42 INFO dspy.teleprompt.mipro_optimizer_v2: Best full score so far! Score: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:11:42 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 87.73 with parameters ['Predictor 0: Instruction 3', 'Predictor 0: Few-Shot Set 5'].
2025[/07/08](http://localhost:8888/07/08) 10:11:42 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73]
2025[/07/08](http://localhost:8888/07/08) 10:11:42 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:11:42 INFO dspy.teleprompt.mipro_optimizer_v2: ========================


2025[/07/08](http://localhost:8888/07/08) 10:11:42 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 7 [/](http://localhost:8888/) 18 =====

Average Metric: 7.80 / 11 (70.9%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:06<00:00,  1.83it/s]
2025[/07/08](http://localhost:8888/07/08) 10:11:48 INFO dspy.evaluate.evaluate: Average Metric: 7.8 [/](http://localhost:8888/) 11 (70.9%)
2025[/07/08](http://localhost:8888/07/08) 10:11:48 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 70.91 with parameters ['Predictor 0: Instruction 4', 'Predictor 0: Few-Shot Set 6'].
2025[/07/08](http://localhost:8888/07/08) 10:11:48 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91]
2025[/07/08](http://localhost:8888/07/08) 10:11:48 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:11:48 INFO dspy.teleprompt.mipro_optimizer_v2: ========================


2025[/07/08](http://localhost:8888/07/08) 10:11:48 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 8 [/](http://localhost:8888/) 18 =====

Average Metric: 7.80 / 11 (70.9%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:07<00:00,  1.39it/s]
2025[/07/08](http://localhost:8888/07/08) 10:11:56 INFO dspy.evaluate.evaluate: Average Metric: 7.8 [/](http://localhost:8888/) 11 (70.9%)
2025[/07/08](http://localhost:8888/07/08) 10:11:56 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 70.91 with parameters ['Predictor 0: Instruction 5', 'Predictor 0: Few-Shot Set 1'].
2025[/07/08](http://localhost:8888/07/08) 10:11:56 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91]
2025[/07/08](http://localhost:8888/07/08) 10:11:56 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:11:56 INFO dspy.teleprompt.mipro_optimizer_v2: ========================


2025[/07/08](http://localhost:8888/07/08) 10:11:56 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 9 [/](http://localhost:8888/) 18 =====

Average Metric: 8.60 / 11 (78.2%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:11<00:00,  1.02s/it]
2025[/07/08](http://localhost:8888/07/08) 10:12:07 INFO dspy.evaluate.evaluate: Average Metric: 8.6 [/](http://localhost:8888/) 11 (78.2%)
2025[/07/08](http://localhost:8888/07/08) 10:12:07 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 78.18 with parameters ['Predictor 0: Instruction 3', 'Predictor 0: Few-Shot Set 3'].
2025[/07/08](http://localhost:8888/07/08) 10:12:07 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18]
2025[/07/08](http://localhost:8888/07/08) 10:12:07 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:12:07 INFO dspy.teleprompt.mipro_optimizer_v2: ========================


2025[/07/08](http://localhost:8888/07/08) 10:12:07 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 10 [/](http://localhost:8888/) 18 =====

Average Metric: 7.70 / 11 (70.0%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:08<00:00,  1.23it/s]
2025[/07/08](http://localhost:8888/07/08) 10:12:16 INFO dspy.evaluate.evaluate: Average Metric: 7.7 [/](http://localhost:8888/) 11 (70.0%)
2025[/07/08](http://localhost:8888/07/08) 10:12:16 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 70.0 with parameters ['Predictor 0: Instruction 3', 'Predictor 0: Few-Shot Set 10'].
2025[/07/08](http://localhost:8888/07/08) 10:12:16 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18, 70.0]
2025[/07/08](http://localhost:8888/07/08) 10:12:16 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:12:16 INFO dspy.teleprompt.mipro_optimizer_v2: =========================


2025[/07/08](http://localhost:8888/07/08) 10:12:16 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 11 [/](http://localhost:8888/) 18 =====

Average Metric: 9.55 / 11 (86.8%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:02<00:00,  3.99it/s]
2025[/07/08](http://localhost:8888/07/08) 10:12:19 INFO dspy.evaluate.evaluate: Average Metric: 9.55 [/](http://localhost:8888/) 11 (86.8%)
2025[/07/08](http://localhost:8888/07/08) 10:12:19 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 86.82 with parameters ['Predictor 0: Instruction 3', 'Predictor 0: Few-Shot Set 5'].
2025[/07/08](http://localhost:8888/07/08) 10:12:19 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18, 70.0, 86.82]
2025[/07/08](http://localhost:8888/07/08) 10:12:19 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:12:19 INFO dspy.teleprompt.mipro_optimizer_v2: =========================


2025[/07/08](http://localhost:8888/07/08) 10:12:19 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 12 [/](http://localhost:8888/) 18 =====

Average Metric: 9.50 / 11 (86.4%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:04<00:00,  2.54it/s]
2025[/07/08](http://localhost:8888/07/08) 10:12:23 INFO dspy.evaluate.evaluate: Average Metric: 9.5 [/](http://localhost:8888/) 11 (86.4%)
2025[/07/08](http://localhost:8888/07/08) 10:12:23 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 86.36 with parameters ['Predictor 0: Instruction 3', 'Predictor 0: Few-Shot Set 5'].
2025[/07/08](http://localhost:8888/07/08) 10:12:23 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18, 70.0, 86.82, 86.36]
2025[/07/08](http://localhost:8888/07/08) 10:12:23 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:12:23 INFO dspy.teleprompt.mipro_optimizer_v2: =========================


2025[/07/08](http://localhost:8888/07/08) 10:12:23 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 13 [/](http://localhost:8888/) 18 =====

Average Metric: 7.70 / 11 (70.0%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:17<00:00,  1.56s/it]
2025[/07/08](http://localhost:8888/07/08) 10:12:41 INFO dspy.evaluate.evaluate: Average Metric: 7.7 [/](http://localhost:8888/) 11 (70.0%)
2025[/07/08](http://localhost:8888/07/08) 10:12:41 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 70.0 with parameters ['Predictor 0: Instruction 5', 'Predictor 0: Few-Shot Set 5'].
2025[/07/08](http://localhost:8888/07/08) 10:12:41 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18, 70.0, 86.82, 86.36, 70.0]
2025[/07/08](http://localhost:8888/07/08) 10:12:41 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:12:41 INFO dspy.teleprompt.mipro_optimizer_v2: =========================


2025[/07/08](http://localhost:8888/07/08) 10:12:41 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 14 [/](http://localhost:8888/) 18 =====

Average Metric: 8.80 / 11 (80.0%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:11<00:00,  1.04s/it]
2025[/07/08](http://localhost:8888/07/08) 10:12:52 INFO dspy.evaluate.evaluate: Average Metric: 8.8 [/](http://localhost:8888/) 11 (80.0%)
2025[/07/08](http://localhost:8888/07/08) 10:12:52 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 80.0 with parameters ['Predictor 0: Instruction 3', 'Predictor 0: Few-Shot Set 9'].
2025[/07/08](http://localhost:8888/07/08) 10:12:52 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18, 70.0, 86.82, 86.36, 70.0, 80.0]
2025[/07/08](http://localhost:8888/07/08) 10:12:52 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:12:52 INFO dspy.teleprompt.mipro_optimizer_v2: =========================


2025[/07/08](http://localhost:8888/07/08) 10:12:52 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 15 [/](http://localhost:8888/) 18 =====

Average Metric: 7.90 / 11 (71.8%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:10<00:00,  1.10it/s]
2025[/07/08](http://localhost:8888/07/08) 10:13:02 INFO dspy.evaluate.evaluate: Average Metric: 7.9 [/](http://localhost:8888/) 11 (71.8%)
2025[/07/08](http://localhost:8888/07/08) 10:13:02 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 71.82 with parameters ['Predictor 0: Instruction 1', 'Predictor 0: Few-Shot Set 5'].
2025[/07/08](http://localhost:8888/07/08) 10:13:02 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18, 70.0, 86.82, 86.36, 70.0, 80.0, 71.82]
2025[/07/08](http://localhost:8888/07/08) 10:13:02 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:13:02 INFO dspy.teleprompt.mipro_optimizer_v2: =========================


2025[/07/08](http://localhost:8888/07/08) 10:13:02 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 16 [/](http://localhost:8888/) 18 =====

Average Metric: 7.60 / 11 (69.1%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:05<00:00,  2.13it/s]
2025[/07/08](http://localhost:8888/07/08) 10:13:07 INFO dspy.evaluate.evaluate: Average Metric: 7.6 [/](http://localhost:8888/) 11 (69.1%)
2025[/07/08](http://localhost:8888/07/08) 10:13:07 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 69.09 with parameters ['Predictor 0: Instruction 3', 'Predictor 0: Few-Shot Set 8'].
2025[/07/08](http://localhost:8888/07/08) 10:13:07 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18, 70.0, 86.82, 86.36, 70.0, 80.0, 71.82, 69.09]
2025[/07/08](http://localhost:8888/07/08) 10:13:07 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:13:07 INFO dspy.teleprompt.mipro_optimizer_v2: =========================


2025[/07/08](http://localhost:8888/07/08) 10:13:07 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 17 [/](http://localhost:8888/) 18 =====

Average Metric: 7.20 / 11 (65.5%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:08<00:00,  1.27it/s]
2025[/07/08](http://localhost:8888/07/08) 10:13:16 INFO dspy.evaluate.evaluate: Average Metric: 7.2 [/](http://localhost:8888/) 11 (65.5%)
2025[/07/08](http://localhost:8888/07/08) 10:13:16 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 65.45 with parameters ['Predictor 0: Instruction 2', 'Predictor 0: Few-Shot Set 5'].
2025[/07/08](http://localhost:8888/07/08) 10:13:16 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18, 70.0, 86.82, 86.36, 70.0, 80.0, 71.82, 69.09, 65.45]
2025[/07/08](http://localhost:8888/07/08) 10:13:16 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:13:16 INFO dspy.teleprompt.mipro_optimizer_v2: =========================


2025[/07/08](http://localhost:8888/07/08) 10:13:16 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 18 [/](http://localhost:8888/) 18 =====

Average Metric: 6.85 / 11 (62.3%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:18<00:00,  1.68s/it]
2025[/07/08](http://localhost:8888/07/08) 10:13:35 INFO dspy.evaluate.evaluate: Average Metric: 6.85 [/](http://localhost:8888/) 11 (62.3%)
2025[/07/08](http://localhost:8888/07/08) 10:13:35 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 62.27 with parameters ['Predictor 0: Instruction 0', 'Predictor 0: Few-Shot Set 5'].
2025[/07/08](http://localhost:8888/07/08) 10:13:35 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18, 70.0, 86.82, 86.36, 70.0, 80.0, 71.82, 69.09, 65.45, 62.27]
2025[/07/08](http://localhost:8888/07/08) 10:13:35 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:13:35 INFO dspy.teleprompt.mipro_optimizer_v2: =========================


2025[/07/08](http://localhost:8888/07/08) 10:13:35 INFO dspy.teleprompt.mipro_optimizer_v2: ===== Trial 19 [/](http://localhost:8888/) 18 =====

Average Metric: 7.30 / 11 (66.4%): 100%|████████████████████████████████████████████████████████████████████| 11/11 [00:18<00:00,  1.72s/it]
2025[/07/08](http://localhost:8888/07/08) 10:13:54 INFO dspy.evaluate.evaluate: Average Metric: 7.3 [/](http://localhost:8888/) 11 (66.4%)
2025[/07/08](http://localhost:8888/07/08) 10:13:54 INFO dspy.teleprompt.mipro_optimizer_v2: Score: 66.36 with parameters ['Predictor 0: Instruction 3', 'Predictor 0: Few-Shot Set 11'].
2025[/07/08](http://localhost:8888/07/08) 10:13:54 INFO dspy.teleprompt.mipro_optimizer_v2: Scores so far: [50.45, 86.36, 82.27, 61.82, 77.27, 87.73, 70.91, 70.91, 78.18, 70.0, 86.82, 86.36, 70.0, 80.0, 71.82, 69.09, 65.45, 62.27, 66.36]
2025[/07/08](http://localhost:8888/07/08) 10:13:54 INFO dspy.teleprompt.mipro_optimizer_v2: Best score so far: 87.73
2025[/07/08](http://localhost:8888/07/08) 10:13:54 INFO dspy.teleprompt.mipro_optimizer_v2: =========================


2025[/07/08](http://localhost:8888/07/08) 10:13:54 INFO dspy.teleprompt.mipro_optimizer_v2: Returning best identified program with score 87.73!

I'm wondering if this is the expected behavior? Thank you!

Steps to reproduce

class QA(dspy.Signature):
    question = dspy.InputField(desc='Question to answer based on product information')
    context = dspy.InputField(desc='Product information from doing a product research')
    answer_criteria = dspy.InputField(desc='Answer criteria')
    answer = dspy.OutputField(desc='Answer to the question')

class AnswerResearch(dspy.Module):
    def __init__(self):
        self.qa = dspy.ChainOfThought(QA)
        
    def forward(self, question, context, answer_criteria):
        return self.qa(question=question, context=context, answer_criteria=answer_criteria)
 
optimizer = dspy.MIPROv2(metric=metric, auto='medium', num_threads=12)  # use fewer threads if your rate limit is small

optimized_pipeline = optimizer.compile(AnswerResearch(), trainset=train,
                           max_bootstrapped_demos=0, max_labeled_demos=0,
                           requires_permission_to_run=False)

DSPy version

2.6.27

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions