How to use ConversationSummaryMemory with ollama?

[vladdie]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

llm = ChatOllama(temperature=0.0, model="llama3:8b-instruct-fp16")
memory = ConversationSummaryBufferMemory(llm=llm, max_token_limit=100)
memory.save_context({"input": "Hello"}, {"output": "What's up"})

Description

I tried to use ConversationSummaryBufferMemory with local open source LLM with ollama, but I got the error: " Can't load tokenizer for 'gpt2'." It seems that ConversationSummaryBufferMemory only supports ChatOpenAI()? Is there anyway I can set a customized tokenizer?
Many thanks!

System Info

langchain=0.2.3
Windows 11

[dosubot[bot]]

Hey @vladdie, I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

To use ConversationSummaryBufferMemory with ChatOllama and set a customized tokenizer, you need to ensure that your local LLM object has a method get_num_tokens_from_messages that uses your desired tokenizer. Here is an example of how you can achieve this:

1. Create a Custom LLM Class with a Custom Tokenizer:

from transformers import AutoTokenizer

class CustomLLM:
    def __init__(self, tokenizer):
        self.tokenizer = tokenizer

    def get_num_tokens_from_messages(self, messages):
        # Custom logic to count tokens using the provided tokenizer
        return sum(len(self.tokenizer.tokenize(message.content)) for message in messages)

# Initialize your custom tokenizer
custom_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Create an instance of your custom LLM with the tokenizer
custom_llm = CustomLLM(tokenizer=custom_tokenizer)

2. Integrate the Custom LLM with ConversationSummaryBufferMemory:

from langchain.memory import ConversationSummaryBufferMemory

# Initialize ConversationSummaryBufferMemory with the custom LLM
memory = ConversationSummaryBufferMemory(llm=custom_llm, max_token_limit=100)

# Save context to memory
memory.save_context({"input": "Hello"}, {"output": "What's up"})

3. Use ChatOllama with the Custom Tokenizer:

from langchain_community.chat_models import ChatOllama

# Initialize the ChatOllama model
ollama = ChatOllama(model="llama3:8b-instruct-fp16")

# Example messages
messages = [
    {"role": "user", "content": "Hello"},
    {"role": "assistant", "content": "What's up"}
]

# Convert messages to Ollama format
ollama_messages = ollama._convert_messages_to_ollama_messages(messages)

# Add messages to the memory
for message in ollama_messages:
    memory.add_message(message)

# Now you can use the memory with the ChatOllama model
response = ollama._generate(messages=memory.get_messages())
print(response)

This setup ensures that ConversationSummaryBufferMemory uses your custom tokenizer with ChatOllama. The key is to provide a method in your custom LLM class that handles tokenization appropriately ^[1][2].

To continue talking to Dosu, mention @dosu.

[kurup-007]

@dosu when I run the code which you gave, I get below error.

raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for ConversationSummaryBufferMemory
llm
value is not a valid dict (type=type_error.dict)