Skip to content

Fix BootstrapFinetune example in index doc and add basic tests for bootstrap_finetune. #8435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

Hangzhi
Copy link
Contributor

@Hangzhi Hangzhi commented Jun 23, 2025

Why

Followup for #8395 since to make the example is not runnable after the ChainOfThoughtWithHint fix.

By running the example code in https://dspy.ai/#__tabbed_3_3

Attribution error would be thrown.

dspy/dspy/teleprompt/bootstrap_finetune.py", line 89, in compile
    logger.info(f"Using {len(train_data)} data points for fine-tuning the model: {pred.lm.model}")
                                                                                  ^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'model'

What

  1. Add the .set_lm(lm) for the example code.
  2. Add basic tests for BootstrapFinetune module and better error handling in BootstrapFinetune
  3. Link a full tutorial in the BootstrapFinetune index example.

After

I can get the result as below with the example code by adding .set_lm.

2025/06/26 10:07:42 INFO dspy.teleprompt.bootstrap_finetune: Updating the student program with the fine-tuned LMs...
2025/06/26 10:07:42 INFO dspy.teleprompt.bootstrap_finetune: BootstrapFinetune has finished compiling the student program
Prediction(
    reasoning='A pending cash withdrawal refers to a cash withdrawal transaction that has been initiated but is still being processed and has not been completed yet.',
    label='pending_cash_withdrawal'
)

@Hangzhi Hangzhi changed the title done Fix bootstrap_finetune without by reading lm from settings and add basic tests for bootstrap_finetune. Jun 23, 2025
@Hangzhi Hangzhi changed the title Fix bootstrap_finetune without by reading lm from settings and add basic tests for bootstrap_finetune. Fix bootstrap_finetune by reading lm from settings and add basic tests for bootstrap_finetune. Jun 23, 2025
@Hangzhi Hangzhi force-pushed the yiwei-dai_data/cot-fintune branch from fafc262 to bd0d3bb Compare June 23, 2025 15:35
@chenmoneygithub
Copy link
Collaborator

@dilarasoylu Do you remember the context for us always requiring the pred.lm to be set?

all_predictors_have_lms(student)

@dilarasoylu
Copy link
Collaborator

@dilarasoylu Do you remember the context for us always requiring the pred.lm to be set?

all_predictors_have_lms(student)

Oh, yes, we always want to ensure that predictor level LMs are set for the finetune optimizers, otherwise we might end up modifying a globally set LM as part of the finetune process, which might be confusing for the users. We use the 'set_lm(...)' method on the program to be finetuned before calling any optimizer that modifies the weights in order to ensure that predictor level LMs are set.

@Hangzhi Hangzhi force-pushed the yiwei-dai_data/cot-fintune branch from 72e11b4 to 6cd9a32 Compare June 26, 2025 16:42
@Hangzhi Hangzhi changed the title Fix bootstrap_finetune by reading lm from settings and add basic tests for bootstrap_finetune. Fix BootstrapFinetune example in index doc and add basic tests for bootstrap_finetune. Jun 26, 2025
@Hangzhi
Copy link
Contributor Author

Hangzhi commented Jun 26, 2025

@chenmoneygithub @dilarasoylu
Thanks for the suggestions. I updated the example code and set_lm works.


# Optimize via BootstrapFinetune.
optimizer = dspy.BootstrapFinetune(metric=(lambda x, y, trace=None: x.label == y.label), num_threads=24)
optimized = optimizer.compile(classify, trainset=trainset)

optimized(text="What does a pending cash withdrawal mean?")

# For a complete fine-tuning tutorial, see: https://dspy.ai/tutorials/classification_finetuning/
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chenmoneygithub let me know if there are better way refer to another doc page.

@okhat
Copy link
Collaborator

okhat commented Jul 5, 2025

@Hangzhi This looks great to me, and I was about to merge. But why are we catching all exceptions from the teacher? I think the original behavior should have been to let it be handled by dspy.settings.max_errors essentially (through dspy.Evaluate). This allows more fine-grained control and alerts the user if the program seems broken.

@Hangzhi
Copy link
Contributor Author

Hangzhi commented Jul 5, 2025

@okhat That makes sense. Thanks for the clarification.
I removed the call exception catching logic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants