-
Notifications
You must be signed in to change notification settings - Fork 2k
Fix BootstrapFinetune example in index doc and add basic tests for bootstrap_finetune
.
#8435
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
bootstrap_finetune
without by reading lm from settings and add basic tests for bootstrap_finetune
.
bootstrap_finetune
without by reading lm from settings and add basic tests for bootstrap_finetune
. bootstrap_finetune
by reading lm from settings and add basic tests for bootstrap_finetune
.
fafc262
to
bd0d3bb
Compare
@dilarasoylu Do you remember the context for us always requiring the
|
Oh, yes, we always want to ensure that predictor level LMs are set for the finetune optimizers, otherwise we might end up modifying a globally set LM as part of the finetune process, which might be confusing for the users. We use the 'set_lm(...)' method on the program to be finetuned before calling any optimizer that modifies the weights in order to ensure that predictor level LMs are set. |
72e11b4
to
6cd9a32
Compare
bootstrap_finetune
by reading lm from settings and add basic tests for bootstrap_finetune
. bootstrap_finetune
.
@chenmoneygithub @dilarasoylu |
|
||
# Optimize via BootstrapFinetune. | ||
optimizer = dspy.BootstrapFinetune(metric=(lambda x, y, trace=None: x.label == y.label), num_threads=24) | ||
optimized = optimizer.compile(classify, trainset=trainset) | ||
|
||
optimized(text="What does a pending cash withdrawal mean?") | ||
|
||
# For a complete fine-tuning tutorial, see: https://dspy.ai/tutorials/classification_finetuning/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chenmoneygithub let me know if there are better way refer to another doc page.
@Hangzhi This looks great to me, and I was about to merge. But why are we catching all exceptions from the teacher? I think the original behavior should have been to let it be handled by dspy.settings.max_errors essentially (through dspy.Evaluate). This allows more fine-grained control and alerts the user if the program seems broken. |
@okhat That makes sense. Thanks for the clarification. |
Why
Followup for #8395 since to make the example is not runnable after the ChainOfThoughtWithHint fix.
By running the example code in https://dspy.ai/#__tabbed_3_3
Attribution error would be thrown.
What
BootstrapFinetune
module and better error handling in BootstrapFinetuneAfter
I can get the result as below with the example code by adding
.set_lm
.