How to create multi-agent-user chat in Semantic Kernel?

[anirudhv]

I am creating a semantic kernel application. I want the user to be an active part of the conversation; after the initial message/request, the user shouldn't remain silent while the two agents take over.

Here is an example transcript -

User: I want to change my primary address. 
Agent A: Sure, let me verify your identity. Please provide your ID. 
User: ID001
Agent A: Perfect! Now provide your full name. (If the ID is invalid, Agent A keeps asking the user to try again until they enter a valid ID)
User: Joe Smith 
Agent A: Wonderful, now please confirm your current primary address. (If the name entered by the user does not match the name associated with the ID, the agent keeps asking the user to try again until they get it right)
User: 123 Random St San Francisco, CA 95234 
Agent A: Perfect. Your identity has been verified. Here are all the addresses on your file. Let me know which one you want to make your new permanent address. 
1. ...
2. ...
3. ...
User: 2
Agent A: Give me a second to process your request. [calling Agent B]
Agent B: [Your new primary address has a higher risk rating than your current one. Your request will be under further review.]
Agent A: Your request is under review.

Notice how the user is an active part of the conversation.

If I specify when the user should provide another input in the selection function, the kernel will mistake the user for an agent called user (which doesn't exist) and return an error.

There doesn't seem to be an easy way to determine when a user should be able to speak in an agent group chat in semantic kernel. The selection function is only used to determine when an agent should speak. It seems like AgentGroupChat is designed to be a conversation between agents with no human intervention after the initial user message.

Is AgentGroupChat what I should be using given that I want the user to be an active part of the conversation with the agents? What framework/libraries would I even use to simulate the transcript?

[leestott]

Implementing User-in-the-Loop Conversations in Semantic Kernel

You're right that AgentGroupChat isn't designed for user-in-the-loop interactions. It's primarily for agent-to-agent conversations after the initial user message.

Solution: Custom Conversation Orchestrator

The best approach is to implement a custom conversation orchestrator that explicitly handles user interactions:

public class ConversationOrchestrator
{
    private readonly Kernel _kernel;
    private readonly KernelAgent _agentA;
    private readonly KernelAgent _agentB;
    private List<ChatMessageContent> _history = new();
    
    public ConversationOrchestrator(Kernel kernel, KernelAgent agentA, KernelAgent agentB)
    {
        _kernel = kernel;
        _agentA = agentA;
        _agentB = agentB;
    }
    
    public async Task<string> ProcessUserInput(string userInput)
    {
        // Add user message to history
        _history.Add(new ChatMessageContent(AuthorRole.User, userInput));
        
        // Determine next action based on conversation state
        // This is where your state machine logic would go
        
        // For example, if we're in the ID verification phase
        if (IsInVerificationPhase())
        {
            var response = await _agentA.InvokeAsync(_kernel, _history);
            _history.Add(response);
            
            // Determine if we need to call Agent B
            if (ShouldCallAgentB(response.Content))
            {
                var agentBInput = PrepareAgentBInput(_history);
                var agentBResponse = await _agentB.InvokeAsync(_kernel, agentBInput);
                _history.Add(agentBResponse);
                
                // Agent A processes Agent B's response
                var finalResponse = await _agentA.InvokeAsync(_kernel, _history);
                _history.Add(finalResponse);
                return finalResponse.Content;
            }
            
            return response.Content;
        }
        
        // Other conversation phases...
        
        return "I'm not sure how to respond.";
    }
    
    // Helper methods for state management
    private bool IsInVerificationPhase() { /* ... */ }
    private bool ShouldCallAgentB(string content) { /* ... */ }
    private List<ChatMessageContent> PrepareAgentBInput(List<ChatMessageContent> history) { /* ... */ }
}

Implementation Strategy

1. Define conversation states (ID verification, name verification, address selection, etc.)
2. Create state transitions based on message content
3. Implement agent prompts specific to each state
4. Control when to expect user input by returning a response that asks a question

Example Application Flow

var kernel = Kernel.CreateBuilder()
    .AddOpenAIChatCompletion("gpt-4", "your-api-key")
    .Build();

var verifierAgent = new KernelAgent("VerifierAgent", 
    "You verify user identity before changing addresses...");
    
var riskAgent = new KernelAgent("RiskAgent", 
    "You assess risk levels of address changes...");

var orchestrator = new ConversationOrchestrator(kernel, verifierAgent, riskAgent);

// In your UI/console app
while (true)
{
    Console.Write("User: ");
    var userInput = Console.ReadLine();
    
    var response = await orchestrator.ProcessUserInput(userInput);
    Console.WriteLine($"Agent: {response}");
    
    // If the conversation is complete, break
    if (IsConversationComplete(response))
        break;
}

[arafattehsin]

Process Framework within Semantic Kernel addresses the human in the loop scenarios. If you want to checkout the working examples on how the Agent Group chat works with proper selection and termination strategies then you may checkout this notebook where I have put a few examples for the same.