Skip to content

Add Tool Call Accuracy Evaluator Bugbash Notebook #42121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 44 commits into
base: users/singankit/agent_evaluators_v2_bug_bash
Choose a base branch
from
Open

Add Tool Call Accuracy Evaluator Bugbash Notebook #42121

wants to merge 44 commits into from
+191,028 −994,233

Conversation

salma-elshafey
Copy link
Contributor

Description

Please add an informative description that covers that changes made by the pull request and link all relevant issues.

If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

Salma Elshafey and others added 30 commits June 22, 2025 22:40
support 5 levels, evaluate all tools at once
41de91a
Update sample notebook and change log
6a1e2b3
Add missing import
0dad199
Modify test cases to match the new output format
e4b1a37
Modify other test file to match the new output format
a40c91b
Fixed parsing of results
ed0ecf9
Change key name in output
9bc900b
Spell check fixes
eaf493a
Minor prompt update
1965639
Update result key to tool_call_accuracy
8865240
@salma-elshafey
Delete test_new_evaluator.ipynb
fcd1cb8
Added field names and messages as constants
67fc87d
Merge branch 'selshafey/improve_tool_call_accuracy' of https://github…
080f941 
….com/salma-elshafey/azure-sdk-for-python into selshafey/improve_tool_call_accuracy
Additional note in prompt
fd2429f
Re-add the temperature to the prompty file
6c9e342
Removed 'applicable' field and print statement
d0f637e
Move excess/missing tool calls fields under additional details
4c27dff
Typo fix and removal of redundant field in the prompt
3fa14f0
Modify per_tool_call_details field's name to details
2c3ce50
@howieleung
Made response_format option type public and update docstr (Azure#41991)
9d7aea0 
* Made response_format option type public and update docstr

* resolved comment

* update

* Update CHANGELOG.md
@azure-sdk
Increment package version after release of azure-cosmos (Azure#42006)
e646d9e
@Copilot @singankit
[evaluation] Fix lazy loading for optional dependency imports to avoi…
d76b8ba 
…d noisy messages (Azure#41852)

* Initial plan

* Fix lazy loading for optional dependency imports to avoid noisy messages

Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>

* Refactor lazy import mechanism to use generalized function

- Replace separate _try_import_aiagentconverter and _try_import_skagentconverter functions with a single _create_lazy_import function
- The new function accepts class_name, module_path, and dependency_name as parameters
- Reduces code duplication and makes the pattern more maintainable
- Maintains exact same functionality and error messages
- Updates corresponding unit tests to reflect the new implementation

Addresses feedback from @singankit to generalize the lazy import pattern.

Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>

* Remove [INFO] prefix from error messages in lazy imports

Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>

* Simplify lazy import error handling to let ImportError propagate naturally

Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>

* Implement lazy loading for red_team module to avoid noisy import messages

Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>

* Revert red_team module to traditional import pattern as requested

Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>

* Change red_team module to raise ImportError instead of print statement

Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>

* Apply black formatting to fix code style issues

Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>

* Remove AIAgentConverter test since azure-ai-projects is always in dev requirements

Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>

* Skip lazy import tests when semantic-kernel is installed

Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: singankit <30610298+singankit@users.noreply.github.com>
@pvaneck
[Identity] Update VisualStudioCodeCredential to be broker-based (Azur…
79ccfa6 
…e#41822)

Signed-off-by: Paul Van Eck <paulvaneck@microsoft.com>
@azure-sdk @ChenxiJiang333
[AutoRelease] t2-dashboard-2025-06-10-06558(can only be merged by SDK…
08b1156 
… owner) (Azure#41496)

* code and test

* update-tasecase

* Update CHANGELOG.md

---------

Co-authored-by: azure-sdk <PythonSdkPipelines>
Co-authored-by: ChenxiJiang333 <v-chenjiang@microsoft.com>
Co-authored-by: ChenxiJiang333 <119990644+ChenxiJiang333@users.noreply.github.com>
@azure-sdk @ChenxiJiang333
[AutoRelease] t2-containerservice-2025-07-02-17859(can only be merged…
b926f21 
… by SDK owner) (Azure#41859)

* code and test

* update changelog

* Update CHANGELOG.md

---------

Co-authored-by: azure-sdk <PythonSdkPipelines>
Co-authored-by: ChenxiJiang333 <v-chenjiang@microsoft.com>
Co-authored-by: ChenxiJiang333 <119990644+ChenxiJiang333@users.noreply.github.com>
@pvaneck @Copilot
[Identity Broker] Allow default account usage in WSL (Azure#42005)
81df908 
* [Identity Broker] Allow default account usage in WSL

Signed-off-by: Paul Van Eck <paulvaneck@microsoft.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Signed-off-by: Paul Van Eck <paulvaneck@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@azure-sdk @ChenxiJiang333
[AutoRelease] t2-hardwaresecuritymodules-2025-07-02-39085(can only be…
4268dd3 
… merged by SDK owner) (Azure#41861)

* code and test

* update changelog

* Update CHANGELOG.md

---------

Co-authored-by: azure-sdk <PythonSdkPipelines>
Co-authored-by: ChenxiJiang333 <v-chenjiang@microsoft.com>
Co-authored-by: ChenxiJiang333 <119990644+ChenxiJiang333@users.noreply.github.com>
@slister1001 @Copilot
[evaluation] add retry logic for high concurrency scenarios in Advers…
0c44507 
…arailSimulator, _SafetyEvaluation (Azure#41978)

* Add retry logic for high concurrency scenarios

* Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/simulator/_model_tools/_proxy_completion_model.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* black fixes

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@Copilot @slister1001
Fix red team scan output_path issue - prevent interim evaluation over…
56db2d7 
…writes (Azure#42024)

* Initial plan

* Fix red team scan output_path issue - prevent interim evaluation overwrites

Co-authored-by: slister1001 <103153180+slister1001@users.noreply.github.com>

* Update CHANGELOG.md to document red team scan output_path bug fix

Co-authored-by: slister1001 <103153180+slister1001@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: slister1001 <103153180+slister1001@users.noreply.github.com>
@howieleung
Use paramtype for keyword params (Azure#42008)
3b2723b
@github-actions github-actions bot added AI Agents Azure.Identity Community Contribution Community members are working on the issue Cosmos customer-reported Issues that are reported by GitHub users external to the Azure organization. Evaluation Issues related to the client library for Azure AI Evaluation Health Deidentification labels Jul 21, 2025
@github-actions GitHub Actions
Copy link

Thank you for your contribution @salma-elshafey! We will review the pull request and get back to you soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ChenxiJiang333 ChenxiJiang333 Awaiting requested review from ChenxiJiang333 ChenxiJiang333 is a code owner

@msyyc msyyc Awaiting requested review from msyyc msyyc is a code owner

@alexathomases alexathomases Awaiting requested review from alexathomases alexathomases is a code owner

@dargilco dargilco Awaiting requested review from dargilco dargilco is a code owner

@jhakulin jhakulin Awaiting requested review from jhakulin jhakulin is a code owner

@trangevi trangevi Awaiting requested review from trangevi trangevi is a code owner

@glharper glharper Awaiting requested review from glharper glharper is a code owner

@nick863 nick863 Awaiting requested review from nick863 nick863 is a code owner

@howieleung howieleung Awaiting requested review from howieleung howieleung is a code owner

@pvaneck pvaneck Awaiting requested review from pvaneck pvaneck is a code owner

@xiangyan99 xiangyan99 Awaiting requested review from xiangyan99 xiangyan99 is a code owner

@maorleger maorleger Awaiting requested review from maorleger maorleger is a code owner automatically assigned from Azure/azure-sdk-write-identity

@christothes christothes Awaiting requested review from christothes christothes is a code owner automatically assigned from Azure/azure-sdk-write-identity

@KarishmaGhiya KarishmaGhiya Awaiting requested review from KarishmaGhiya KarishmaGhiya is a code owner automatically assigned from Azure/azure-sdk-write-identity

@chlowell chlowell Awaiting requested review from chlowell chlowell is a code owner automatically assigned from Azure/azure-sdk-write-identity

@minhanh-phan minhanh-phan Awaiting requested review from minhanh-phan minhanh-phan is a code owner automatically assigned from Azure/azure-sdk-write-identity

Assignees
No one assigned
Labels
AI Agents Azure.Identity Community Contribution Community members are working on the issue Cosmos customer-reported Issues that are reported by GitHub users external to the Azure organization. Evaluation Issues related to the client library for Azure AI Evaluation Health Deidentification
Projects
Azure Identity SDK Improvements
Status: Untriaged
CosmosDB Python Eco-System
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@salma-elshafey @howieleung @azure-sdk @Copilot @pvaneck @slister1001 @jovinson-ms @glharper