How to handle ToolCall that contains real tool (Tools Node) and another agent call (Agent Node)

[azidanit]

How to handle ToolCall that contains real tool (Tools Node) and another agent call (Agent Node), I am using agent as tools pattern

[dosubot[bot]]

Hello @azidanit! I'm Dosu, your friendly bot here to help you solve bugs, answer questions, and navigate the world of contributions while you wait for a human maintainer. Let's tackle your issue together!

To handle a ToolCall that involves both a real tool (Tools Node) and another agent call (Agent Node) using the agent as tools pattern, you can set up a workflow using a StateGraph that cycles between the agent and tool nodes. Here's a step-by-step guide:

1. Define the Function to Execute Tools: Implement the call_tool function to execute a tool based on the last message's function call. This function constructs a ToolInvocation from the function call and uses a tool_executor to invoke the tool, adding the response to the conversation state.

   def call_tool(state):
       print("---EXECUTE RETRIEVAL---")
       messages = state["messages"]
       last_message = messages[-1]
       action = ToolInvocation(
           tool=last_message.additional_kwargs["function_call"]["name"],
           tool_input=json.loads(
               last_message.additional_kwargs["function_call"]["arguments"]
           ),
       )
       response = tool_executor.invoke(action)
       function_message = FunctionMessage(content=str(response), name=action.tool)
       return {"messages": [function_message]}

2. Define the Workflow Graph: Use StateGraph to define a workflow that cycles between the agent and tool nodes. Add nodes for the agent (call_model) and the tool (call_tool).

   from langgraph.graph import END, StateGraph
   
   # Define a new graph
   workflow = StateGraph(AgentState)
   
   # Define the nodes we will cycle between
   workflow.add_node("agent", call_model)  # agent
   workflow.add_node("action", call_tool)  # retrieval

3. Set the Entry Point: Set the entry point of the workflow to the agent node to decide whether to retrieve or not.

   workflow.set_entry_point("agent")

Additionally, you can use the create_tool_calling_agent function from LangChain to streamline the process of creating an agent that uses tools. This function helps bind tools to a language model and format messages appropriately, ensuring smooth integration between agent actions and tool outputs ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[azidanit]

the thing is LLM model can call multiple tools and agent at a time, then I want to go to the tool_node first to get all response tool from the actual tool, then pass it to other agent and so on

[azidanit]

@dosu the thing is LLM model can call multiple tools and agent at a time, then I want to go to the tool_node first to get all response tool from the actual tool, then pass it to other agent and so on

[dosubot[bot]]

To achieve this setup where an LLM model calls multiple tools and agents, and you want to first get responses from the tools before passing them to another agent, you can follow these steps using LangChain:

1. Define and Bind Tools: Create the tools you need and bind them to the LLM model using the @tool decorator and .bind_tools.

   from langchain_core.tools import tool
   
   @tool
   def add(a: int, b: int) -> int:
       return a + b
   
   @tool
   def multiply(a: int, b: int) -> int:
       return a * b
   
   tools = [add, multiply]
   
   from langchain_anthropic import ChatAnthropic
   
   model = ChatAnthropic(model="claude-3-sonnet-20240229")
   model_with_tools = model.bind_tools(tools)

2. Create a Tool-Calling Agent: Set up an agent that can manage tool calls using create_tool_calling_agent.

   from langchain import hub
   from langchain.agents import create_tool_calling_agent
   from langchain_openai.chat_models import ChatOpenAI
   
   llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
   prompt = hub.pull("hwchase17/openai-tools-agent")
   agent = create_tool_calling_agent(llm, tools, prompt)

3. Use AgentExecutor for Tool Calls: Utilize AgentExecutor to handle the sequence of tool usage and pass outputs between tools.

   from langchain.agents import AgentExecutor
   
   agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
   
   result = agent_executor.invoke({
       "input": "What is 3 * 12? Also, what is 11 + 49?"
   })

4. Pass Tool Outputs to Another Agent: Use ToolMessages to pass the results of tool invocations back to the model, and then to another agent.

   from langchain_core.messages import HumanMessage, ToolMessage
   
   messages = [HumanMessage(query)]
   ai_msg = model_with_tools.invoke(messages)
   messages.append(ai_msg)
   for tool_call in ai_msg.tool_calls:
       selected_tool = {"add": add, "multiply": multiply}[tool_call["name"].lower()]
       tool_output = selected_tool.invoke(tool_call["args"])
       messages.append(ToolMessage(tool_output, tool_call_id=tool_call["id"]))
   model_with_tools.invoke(messages)

5. Chain Tool Outputs: Ensure your tool functions are designed to accept and return the necessary data for chaining outputs. The AgentExecutor will manage this chaining.

This setup allows you to first gather responses from the tools and then pass these responses to another agent for further processing ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}