Add async support for dspy.Evaluate

[chenmoneygithub]

We are introducing:

• _execute_with_multithreading for running evaluation in multithreading
• _execute_with_event_loop for running evaluation in async concurrency.

The weird part is although technically async eval should run faster than multithreading because evaluation is IO bound task, but I am not noticing it consistently.

Testing script is pasted below:

import asyncio

import dspy
from dspy.datasets.gsm8k import GSM8K

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", cache=False))


# Load math questions from the GSM8K dataset.
gsm8k = GSM8K()
gsm8k_trainset, gsm8k_devset = gsm8k.train[:50], gsm8k.dev[:100]


cot = dspy.ChainOfThought("question->answer")


def my_metric(args, pred):
    return 1.0 if pred.answer == args.answer else 0.0


evaluator = dspy.Evaluate(devset=gsm8k_devset, num_threads=50, display_table=False)


import time

start_time = time.time()
result = evaluator(cot, metric=my_metric)
end_time = time.time()
print(f"Time taken with multithreading: {end_time - start_time} seconds")
print(result)


async def main():
    return await evaluator.acall(cot, metric=my_metric)


start_time = time.time()
result = asyncio.run(main())
end_time = time.time()
print(f"Time taken with async: {end_time - start_time} seconds")
print(result)

70% time async runs faster than multithreading insignificantly, and 30% time it's the reverse. Two potential theories:

• Bottleneck happens on provider side, like rate limiting.
• Behind the scene litellm acompletion is not true async, I need to dig the code a bit.

[TomeHirata]

q: Does acyncio.gather use multiple threads with this setup?