High latency for batch evaluations (Foundry Evaluations SDK)

[jacobreesmontgomery]

I'm experiencing really high latency when running batch evaluations on as little as 10 Q&A sets.

Here is one error I'm repeatedly seeing in particular:

2025-08-04 15:06:48 -0400   72737 execution          ERROR    23/36 flow run failed, indexes: [10,16,14,13,11,15,12,17,19,18,20,23,21,22,24,25,26,29,27,33,28,31,30], exception of index 10: (UserError) OpenAI API hits APIConnectionError: Connection error. [Error reference: https://platform.openai.com/docs/guides/error-codes/api-errors]
======= Run Summary =======

Run name: "azure_ai_evaluation_evaluators_response_completeness_20250804_150051_814454"
...
    raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.

I used to never see this, so I'm unclear as to why this is suddenly an issue. It'd be one thing if the batch evaluations always failed, but in this case some are succeeding and others failing due to the above APIConnectionError. Because of that, it's hard for me to believe that it's a network configuration issue as suggested by the OpenAI error documentation.

Any help would be very much appreciated!

[nitya]

Hi @jacobreesmontgomery - was responding to your other message and noted this one.

Is the behavior recent (e.g., did you update to a newer evaluation SDK when you started seeing this)? Are you using a Foundry project (vs. Hub project) for running your evaluations? And are the issues happening consistently for the same subset of evaluators? Just trying to get more context.

Given the evaluations SDK is evolving fast, might be best to file the issue on the repo and provide more context so they can follow up as well.

Evaluations Issues Tracker:
https://github.com/Azure/azure-sdk-for-python/issues?q=state%3Aopen%20label%3AEvaluation

[jacobreesmontgomery]

The behavior is quite recent, but I am not aware of any recent change to the library version of 1.8.0 that I'm currently using for the evaluations SDK. We're currently using a Hub project but will soon be moving to a Foundry project. The issue itself is happening very consistently.

On my end, I want to see if others on my team are experiencing the same issues, and I also want to test without response completeness and on the main branch to try to drill down on this more.

[jacobreesmontgomery]

I just tested more and it definitely doesn't have to do with any particular evaluation (e.g., response completeness). I believe the issue stems from OpenAI limits in our resourcing if I had to guess. The issues only arise when we start sending more requests during batch evaluations.

[jacobreesmontgomery]

Okay, well I posted an issue here.

[nitya]

Thanks so much @jacobreesmontgomery - I see the team has already assigned a person to the issue (and I know the engineer) so will share context there if useful as well.

The fact you see more when requests go up seems to indicate a rate limit issue - the one thing I can check in with team is whether batch evals does an auto-retry (so it takes longer but will complete).

In the meantime, I will mark the issue statement above as the answer and close the issue here so folks who discover this thread can instead contribute to the issue created directly.