Passing Tool Output back to the Deprecated OllamaFunctions

[noggynoggy]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_experimental.llms.ollama_functions import OllamaFunctions
from langchain_core.messages import AIMessage, SystemMessage, HumanMessage
from langchain.tools import tool

@tool
def multiply(a: float, b: float) -> str:
    """Multiplies a*b and returns the product"""
    af = float(a)
    bf = float(b)
    return f"{af} * {bf} = {a*b}"

tools = [ multiply ]

llm = OllamaFunctions(
    model="gemma2:2b",
    format="json",
).bind_tools(tools)

messages = [
    SystemMessage("You are a helpful AI assistant"),
    HumanMessage("What is 24123 times 323423423")
]

ai_msg = llm.invoke(messages) # promt once
print("AI:     " + str(ai_msg))

messages += [ ai_msg ] # append the answer

try:
    calls = ai_msg.tool_calls
    for call in calls:
        tool_msg = multiply.invoke(call) # get the tool answer
        messages.append(tool_msg)        # append the tool answer
        print("Tool:   " + str(tool_msg))
        ai_msg = llm.invoke(messages)    # promt for a conversational answer
                                         # but its always the same answer again, 
                                         # like the two messages (ai_msg, tool_msg) dont exists
except AttributeError: 
    print(ai_msg.content)
    exit(1) # llm didnt call a tool

# ai_msg.content is always empty here, since it is is always another tool call
print("AI:     " + str(ai_msg))

Description

To use extremely small models like Gemma or tinyllama using the deprecated OllamaFuncitons is nessesary, since they do not support function calling natively.

• here, the last print statement should be a conversational answer to the human question
• but it is always just another tool call like the first ai_msg

Is there a "non-hacky" way (so not forging a system message like "answer conversationally to this sequence") to pass back tool output to the deprecated OllamaFuncitons?

System Info

System Information

OS: Linux
OS Version: #1-NixOS ZEN SMP PREEMPT_DYNAMIC Tue Jan 1 00:00:00 UTC 1980
Python Version: 3.11.9 (main, Apr 2 2024, 08:25:04) [GCC 13.2.0]

Package Information

langchain_core: 0.2.32
langchain: 0.2.14
langchain_community: 0.2.12
langsmith: 0.1.96
langchain_experimental: 0.0.64
langchain_ollama: 0.1.1
langchain_text_splitters: 0.2.2
langgraph: 0.2.3

Optional packages not installed

langserve

Other Dependencies

aiohttp: 3.10.0
async-timeout: Installed. No version info available.
dataclasses-json: 0.6.7
jsonpatch: 1.33
langgraph-checkpoint: 1.0.2
numpy: 1.26.4
ollama: 0.3.1
orjson: 3.10.6
packaging: 24.1
pydantic: 2.8.2
PyYAML: 6.0.1
requests: 2.32.3
SQLAlchemy: 2.0.31
tenacity: 8.5.0
typing-extensions: 4.12.2

[dosubot[bot]]

Yes, there is a "non-hacky" way to pass back tool output to the deprecated OllamaFunctions so that the last print statement is a conversational answer to the human question instead of another tool call. The provided code already includes a mechanism to handle this by appending a default response function (DEFAULT_RESPONSE_FUNCTION) to the list of functions. This function is designed to provide a conversational response if no other tools should be called.

Here is the relevant part of the code that ensures a conversational response is returned:

DEFAULT_RESPONSE_FUNCTION = {
    "name": "__conversational_response",
    "description": (
        "Respond conversationally if no other tools should be called for a given query."
    ),
    "parameters": {
        "type": "object",
        "properties": {
            "response": {
                "type": "string",
                "description": "Conversational response to the user.",
            },
        },
        "required": ["response"],
    },
}

# In the _generate method
functions.append(DEFAULT_RESPONSE_FUNCTION)

# Later in the _generate method
if (
    called_tool is None
    or called_tool["name"] == DEFAULT_RESPONSE_FUNCTION["name"]
):
    if (
        "tool_input" in parsed_chat_result
        and "response" in parsed_chat_result["tool_input"]
    ):
        response = parsed_chat_result["tool_input"]["response"]
    elif "response" in parsed_chat_result:
        response = parsed_chat_result["response"]
    else:
        raise ValueError(
            f"Failed to parse a response from {self.model} output: "
            f"{chat_generation_content}"
        )
    return ChatResult(
        generations=[
            ChatGeneration(
                message=AIMessage(
                    content=response,
                )
            )
        ]
    )

This code ensures that if no specific tool is called or if the __conversational_response function is called, a conversational response is generated and returned.

Additionally, LangChain provides a structured way to handle tool outputs and pass them back to the model using ToolMessages. Here is an example of how you can pass tool outputs back to the model:

from langchain_core.messages import HumanMessage, ToolMessage

messages = [HumanMessage(query)]
ai_msg = llm_with_tools.invoke(messages)
messages.append(ai_msg)
for tool_call in ai_msg.tool_calls:
    selected_tool = {"add": add, "multiply": multiply}[tool_call["name"].lower()]
    tool_output = selected_tool.invoke(tool_call["args"])
    messages.append(ToolMessage(tool_output, tool_call_id=tool_call["id"]))
messages

This approach ensures that the tool outputs are properly passed back to the model, allowing it to generate a final conversational response ^[1][2].

To continue talking to Dosu, mention @dosu.