Build Agent with Ollama locally, or is there a better approach?

[trevorstr]

I am using MacOS, and installed Ollama locally. My objective is to develop an Agent using Langchain, that can take actions on inputs from LLM conversations, and execute various scripts or one-off shell commands.

I asked https://chat.langchain.com about this, and it responded with the following:

For agents, LangChain provides an experimental OllamaFunctions wrapper that gives Ollama the same API as OpenAI Functions. This allows you to:
- Bind functions defined with JSON Schema parameters to the model 3
- Call those functions and get JSON output matching the schema 3
- Use this for structured data extraction or other tasks 3

https://python.langchain.com/docs/integrations/chat/ollama_functions

Is this the best way of developing a custom Agent, without using a managed service for LLM? Or is there another mechanism I should be considering?

Under the Agent Types documentation, I don't see any references to Ollama. Perhaps that's because Ollama Functions are an Agent Type that is still experimental, and hasn't been incorporated into the release documentation?

[Orbifold]

Ollama Functions is one way to bind tools, the alternative is via ReAct. Here's an example of a ReAct agent running locally with Ollama (llama3):

from langchain_core.tools import tool
from langchain.pydantic_v1 import BaseModel, Field
# ============================================================
# Define a custom but dummy tool
# ============================================================
class SearchInput(BaseModel):
    location: str = Field(description="location to search for")
@tool(args_schema=SearchInput)
def weather_forecast(location: str):
    """Weather forecast tool."""
    print(f"Weather for  {location}")
    return f"A dummy forecast for {location}"
# ============================================================
# Define the agent
# ============================================================
from langchain_community.chat_models import ChatOllama
llm = ChatOllama(model="llama3")

from langchain_core.tools import render_text_description
from langchain.agents import AgentExecutor, create_react_agent
from langchain import hub
tools = [weather_forecast]
# a very special prompt embodying the essence of ReAct
prompt = hub.pull("hwchase17/react-json")
prompt = prompt.partial(
    tools=render_text_description(tools),
    tool_names=", ".join([t.name for t in tools]),
)
agent = create_react_agent(llm, tools, prompt)
# ============================================================
# Test the agent
# note: 
# - returning the parsing error will lead to the agent trying again
# - if you set verbose True you can see the agent's internal reflection flow
# ============================================================
agent_executor = AgentExecutor(agent=agent, tools=tools, handle_parsing_errors=True, verbose=False, format="json")
print(agent_executor.invoke({"input":"What is the weather in Paris?"}))

[Genesis1231]

React is way too buggy for Ollama, even when I use llama 3 70b, my guess is to use langgraph and execute the tool separately

[zenocode-org]

I need to test this, but I think using YAML instead of JSON is working better for function call. It's less likely to generate bad formatted JSON, also based on this article it reduces the number of tokens. I had experiences in using OpenAI and also Claude models and for the comparison Claude was terrible in outputting JSON.
I think it's worth a shot to test this ReAct framework with llama model but using a YAML function calling approach.

[Genesis1231]

JSON worked out fine for me with a pydantic parser and prompt emphasis of Json output. (even for llama3 8b)
it is just easier to use than YAML,

langgraph toolnode actually works, and you have more control over how the tools are executed

[CoreyB84]

So is langchain, crewai, and llama.index(GPT) the only ways to build an agent?

[rutwik777]

LangChain has recently introduced Agent execution of Ollama models, its there on their youtube, (there was a Gorq and pure Ollama) tutorials.
I personally feel the agent tools in form of functions gives great flexibility to AI Engineers.