Need advice on agent applications scale out

[YogeshSaini85]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

agent = create_react_agent(llm, tools, prompt)
agent_executor = AgentExecutor.from_agent_and_tools(
        agent=agent,
        tools=tools,
        verbose=True,
        handle_parsing_errors=True,
        max_iterations=4000,
        max_execution_time=60000
    )
agent_executor.invoke("my_query")

Description

I have created an react agent based application which builds a "process".
It has a tool to take user input and other tools to add/remove/update process object.
It starts by detecting items for the process from the initial input and then using appropriate tools to add these items.
Then it uses the 'User Input' tool to get further items about the process and adds it to the process.
This goes on for a while until it has all the items required to create a process, at which point it ends.

We have to put it in production where 1000s of users will be interacting concurrently with the application on 20 nodes:
My manager is worried about:

1. Application memory growing, because agent keeps the state information (intermediate steps, agent_scratchpad) in memory. He wants to make the application stateless between user inputs for the same agent invoke and keep the state in redis cache.
2. In case of node crash, some other node should be able to pick up from the saved state.

So he is asking me to write my own agent code where instead of keeping the state in memory I would save it in redis like checkpoint.

Is this is the best way to go about it? Are there better ways to handle his worries? Are these worries justified?

Seems like re-inventing the wheel writing the agent code when it's already written.

System Info

langchain==0.2.16
langchain-community==0.2.16
langchain-core==0.2.38
langchain-openai==0.1.7
langchain-text-splitters==0.2.4

[dosubot[bot]]

I found a similar solved discussion that might be helpful: What's the best way to save a retriever and an established chain in Redis or any other database?. The discussion suggests using a single retriever and namespacing the documents according to which user owns them. This approach can make the instantiation of the retriever per user request very cheap, as it's just a configuration object, and will be faster than making a round-trip to a database to fetch a retriever.

For making your agent stateless between user inputs, you can refer to the LangChain documentation, which provides an example of how to create and run a stateless agent:

# Create the agent
from langgraph.prebuilt import create_react_agent
from langchain_anthropic import ChatAnthropic
from langchain_community.tools.tavily_search import TavilySearchResults
from langchain_core.messages import HumanMessage

model = ChatAnthropic(model_name="claude-3-sonnet-20240229")
search = TavilySearchResults(max_results=2)
tools = [search]
agent_executor = create_react_agent(model, tools)

# Run the agent on stateless queries
response = agent_executor.invoke({"messages": [HumanMessage(content="hi!")]})
print(response["messages"])

response = agent_executor.invoke({"messages": [HumanMessage(content="whats the weather in sf?")]})
print(response["messages"])

This code initializes an agent and runs it on a few queries without maintaining any state between interactions ^[1][2].

For checkpointing and resuming agent state, LangChain provides built-in mechanisms. The BaseSingleActionAgent and BaseMultiActionAgent classes include a save method that allows saving the agent's state to a file in either JSON or YAML format. This can be used to checkpoint the agent's state, which can later be loaded to resume from the saved state:

# If working with agent executor
agent.agent.save(file_path="path/agent.yaml")

Additionally, in LangGraph, memory is handled through persistence, also known as checkpointing. You can add a checkpointer to the agent to get chat memory for free ^[3][4].

For integrating Redis with LangChain to store agent state, here are some recommended practices:

1. Installation and Setup:

   pip install redis langchain_community
   docker run --name langchain-redis -d -p 6379:6379 redis redis-server --save 60 1 --loglevel warning

2. Connections:

   redis_url = "redis://:secret-pass@localhost:6379/0"
   redis_client = redis.Redis.from_url(redis_url)

3. Cache:

   from langchain.cache import RedisCache
   from langchain.globals import set_llm_cache
   import redis
   
   redis_client = redis.Redis.from_url(redis_url)
   set_llm_cache(RedisCache(redis_client))

4. Semantic Cache:

   from langchain.cache import RedisSemanticCache
   from langchain.globals import set_llm_cache
   import redis
   from tests.integration_tests.vectorstores.fake_embeddings import FakeEmbeddings
   
   redis_url = "redis://localhost:6379"
   set_llm_cache(RedisSemanticCache(
       embedding=FakeEmbeddings(),
       redis_url=redis_url
   ))

5. VectorStore:

   from langchain_community.vectorstores import Redis

6. Memory:

   • For vector store retriever memory and chat message history memory, refer to the detailed examples in the provided notebooks ^[5][6][7].

These practices ensure efficient and effective integration of Redis with LangChain for storing agent state.

Your manager's concerns about memory growth and node crashes are justified, and using Redis for state management can address these issues effectively. This approach avoids reinventing the wheel and leverages existing solutions within LangChain.

To continue talking to Dosu, mention @dosu.