.Net: Trouble using ResponseTool with Kernel Plugins

[thomasjlittle]

Hi, I am trying to give an OpenAIResponseAgent access to Kernel plugins as well as the new ResponseTools (like file and web search) but it seems like as soon as I give the agent a ResponseTool, it loses access to the Kernel plugins, despite them getting loaded into the kernel. I am wondering if this is an issue with my implementation or if this is a bug. I will paste in a short code sample below that demonstrates this behavior.

This code sample is adapted from this SK github sample.

Here is the code:

using System.ClientModel;
using ConsoleApp4;
using Microsoft.Extensions.Configuration;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Agents.OpenAI;
using Microsoft.SemanticKernel.ChatCompletion;
using OpenAI.Responses;

var configuration = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json")
    .Build();

await AddPluginsAsync();
async Task AddPluginsAsync()
{
    // Set vars
    string openAiKey = configuration["openAIKey"] ?? throw new InvalidOperationException("OpenAI key is not set in appsettings.json");
    
    // Create the responses agent 
    OpenAIResponseClient client = new OpenAIResponseClient("gpt-4.1", new ApiKeyCredential(openAiKey));
    OpenAIResponseAgent responseAgent = new(client);
    
    // Create a plugin that defines the tools to be used by the agent.
    KernelPlugin plugin = KernelPluginFactory.CreateFromType<MenuPlugin>();
    var tools = plugin.Select(f => f.ToToolDefinition(plugin.Name));
    responseAgent.Kernel.Plugins.Add(plugin);
    
    // Add a response tool
    ResponseCreationOptions creationOptions = new();
    creationOptions.Tools.Add(ResponseTool.CreateWebSearchTool());
    var options = new OpenAIResponseAgentInvokeOptions()
    {
        ResponseCreationOptions = creationOptions
    };

    ICollection<ChatMessageContent> messages =
    [
        new ChatMessageContent(AuthorRole.User, "What is the special soup and its price? Do not use the web search tool."),
        new ChatMessageContent(AuthorRole.User, "What is the special drink and its price?"),
    ];
    foreach (ChatMessageContent message in messages)
    {
        Console.WriteLine(message);
    }

    // Invoke the agent and output the response
    var responseItems = responseAgent.InvokeAsync(messages, options: options);
    await foreach (ChatMessageContent responseItem in responseItems)
    {
        Console.WriteLine(responseItem);
    }

}

where the MenuPlugin is copied from here.

This code snippet returns something similar to the following:

"What is the special soup and its price? Do not use the web search tool.
What is the special drink and its price?
I don't have access to live or local restaurant menus unless you provide them, so I can't give you the exact information about today's special soup or special drink and their prices without more context.

If you let me know the name of the restaurant or provide a menu, I can help you identify the special soup or drink and their prices! If you have a photo of the menu, you can upload it, and I'll assist you further."

However, when you remove the options: options param from the InvokeAsync call, it calls the menu plugin as expected.

I am using the following package versions:
Microsoft.SemanticKernel 1.60.0
Microsoft.SemanticKernel.Agents.OpenAI 1.60.0
Microsoft.SemanticKernel.Core 1.60.0

[crickman]

@markwallace-microsoft - ResponseCreationOptionsFactory isn't merging tools. The short-circuit on line 19 is incorrect logic.

https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Agents/OpenAI/Internal/ResponseCreationOptionsFactory.cs#L19

[markwallace-microsoft]

Hi @thomasjlittle
I have a fix for this issue and it will be included in the next SK release which should happen in the next few days

[thomasjlittle]

Awesome, thank you!