Python: Include Bing Grounding Tool call results in invoke_stream responses

[KanchiShimono]

Motivation and Context

According to the Terms of Service for Grounding with Bing Search, the tool’s call results must be displayed in their original format.
Currently, in the AzureAIAgent invoke/invoke_stream responses, search results are included in the AnnotationContent, but the search query itself is not.
This PR updates the response of invoke/invoke_stream to include the Bing Grounding Tool’s call results, so that the Bing search query can also be displayed.

Description

• Fixe an issue where errors occurred when handling Bing Grounding Tool results in streaming mode
• Update invoke/invoke_stream responses to include the Bing Grounding Tool call results

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[moonbox3]

Thanks for this, @KanchiShimono. This is using the azure-ai-agents>=1.0.0 and azure-ai-projects>=1.0.0b11 packages, right? Looks like they added support to get this info, because in the current way to handle the tool, we didn't have this info.

[KanchiShimono]

@moonbox3
Thank you for your review.

As you mentioned, I'm using the azure-ai-agents==1.0.0 and azure-ai-projects==1.0.0b11 packages.

By the way, for content that needs special handling, such as Bing search queries, I think it would be ideal to have a dedicated content type like BingGroundingCallContent.
With FunctionCallContent, we have to check the function_name, which can make handling the results a bit more complicated.

[moonbox3]

@moonbox3 Thank you for your review.

As you mentioned, I'm using the azure-ai-agents==1.0.0 and azure-ai-projects==1.0.0b11 packages.

By the way, for content that needs special handling, such as Bing search queries, I think it would be ideal to have a dedicated content type like BingGroundingCallContent. With FunctionCallContent, we have to check the function_name, which can make handling the results a bit more complicated.

Thanks for the comment, @KanchiShimono. I understand that it would be ideal to have a more specific BingGroundingCallContent but that's tricky for the SK SDK, where we aim to have content types that have representation across different Agents/AI Services/Providers, etc. The BingGroundingCallContent has such a slim touch-point (Azure AI Agent), that it's difficult to justify introducing a brand new content type just for that. Could you help me understand, please, what is so tricky about using FunctionCallContent with the Bing Grounding Tool? Thank you.

[KanchiShimono]

@moonbox3
Thank you for your comment.

I agree with your point that adding a highly specific type like BingGroundingCallContent can be difficult. I want to clarify that I’m not strongly requesting the addition of BingGroundingCallContent right now,it was just a casual suggestion.

Here’s what I find tricky about using FunctionCallContent in this context:

• To know that the function_name in FunctionCallContent should be bing_grounding, we have to look into the SK source code, it’s not obvious from the outside.
• If the Bing Grounding Tool result returned by azure-ai-agents (RunStepDeltaToolCall.type) ever changes from bing_grounding, string-based comparisons could easily miss that change until runtime, making it harder and riskier to upgrade package versions.
• With a dedicated type like BingGroundingCallContent, type checking could absorb such changes within the SK SDK itself. This would let users keep their code unchanged in many cases. Also, changes or removals of the type can be detected before runtime using static type checkers (like mypy or Pylance), which reduces the risk of breakage.

[moonbox3]

Hi @KanchiShimono, thanks for the additional comments, they’re very helpful. I agree with your points, and we’ll discuss your suggestion about more scoped content types internally with the SK team.

Now that the agent service is GA, we don’t expect any further breaking changes (for example, bing_grounding being renamed). For type safety, we should be able to type check against the Azure SDK’s BingGroundingToolCall (or whatever the final type is named), to make sure that we’re handling the correct tool call at that layer.

However, once tool calls are converted to SK’s FunctionCallContent, things become less strict since plugins and functions can use arbitrary names. Given this flexibility, the most practical approach for now is to make sure the Bing Tool Call type is well documented, so developers understand how the tool’s function_name will appear within SK’s function call content.

Let me know if there’s anything I missed or if you have further suggestions.

[KanchiShimono]

@moonbox3
Thank you for your comments.
Your explanation helped me better understand some practical approaches to handling Bing Grounding-like content with SK.
I really appreciate how well-organized the SK documentation and samples are, and I think adding this information there would be extremely helpful for developers.

I don’t have any further comments at this time.
Thank you for sincerely listening to and engaging with user feedback.

[moonbox3]

Thanks @KanchiShimono. I will make a new issue to track adding the updated Bing Tool content to the AzureAIAgent page in our learn site, once this change goes in. Appreciate the help with this.

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
agents/azure_ai
agent_content_generation.py	210	50	76%	72–73, 75, 93, 95, 217, 220, 260, 285–286, 294, 302, 304, 312, 319, 355, 394–395, 397, 399–402, 412, 425–426, 428, 430–434, 443, 451, 477–484, 490–493, 498, 503–504, 511
TOTAL	26990	4567	83%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
3616	18 💤	0 ❌	0 🔥	1m 52s ⏱️