"raise NotImplementedError()" while using "with_structured_output" - LLM- "Ollama- llama2"[ "working with Self Rag"]

[Sridevi17j]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from typing import Literal

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_openai import ChatOpenAI


# Data model
class GradeDocuments(BaseModel):
    """Binary score for relevance check on retrieved documents."""

    binary_score: str = Field(
        description="Documents are relevant to the question, 'yes' or 'no'"
    )


# LLM with function call
llm = Ollama(model="gemma:2b")
structured_llm_grader = llm.with_structured_output(GradeDocuments)

# Prompt
system = """You are a grader assessing relevance of a retrieved document to a user question. \n 
    It does not need to be a stringent test. The goal is to filter out erroneous retrievals. \n
    If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \n
    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question."""
grade_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        ("human", "Retrieved document: \n\n {document} \n\n User question: {question}"),
    ]
)

retrieval_grader = grade_prompt | structured_llm_grader
question = "agent memory"
docs = retriever.get_relevant_documents(question)
doc_txt = docs[1].page_content
print(retrieval_grader.invoke({"question": question, "document": doc_txt}))

Description

I am experimenting Self RAG, where i encounter "Not Implemented Error" . I am using Local Ollama model as LLM. How do I extend my LLM to use "with_structured_output" as it is defined only for few.

System Info

System Information

OS: Windows
OS Version: 10.0.22631
Python Version: 3.12.0 (tags/v3.12.0:0fb18b0, Oct 2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)]

Package Information

langchain_core: 0.2.1
langchain: 0.2.1
langchain_community: 0.2.1
langsmith: 0.1.63
langchain_huggingface: 0.0.1
langchain_openai: 0.1.7
langchain_text_splitters: 0.2.0
langchainhub: 0.1.16
langgraph: 0.0.55

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langserve

[dosubot[bot]]

To extend your local Ollama model to use the with_structured_output method in your Self RAG experiment, you can follow these steps:

1. Define Your Schema: Create a Pydantic class that defines the schema for the structured output.

2. Instantiate the Ollama Model: Create an instance of the Ollama class with the appropriate parameters.

3. Use the with_structured_output Method: Call the with_structured_output method on the instance of Ollama with your schema.

4. Invoke the Model: Use the invoke method to get the structured output.

Here is an example using your provided code:

from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_openai import Ollama

# Data model
class GradeDocuments(BaseModel):
    """Binary score for relevance check on retrieved documents."""
    binary_score: str = Field(description="Documents are relevant to the question, 'yes' or 'no'")

# LLM with function call
llm = Ollama(model="gemma:2b")
structured_llm_grader = llm.with_structured_output(GradeDocuments)

# Prompt
system = """You are a grader assessing relevance of a retrieved document to a user question. \n 
    It does not need to be a stringent test. The goal is to filter out erroneous retrievals. \n
    If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \n
    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question."""
grade_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        ("human", "Retrieved document: \n\n {document} \n\n User question: {question}"),
    ]
)

retrieval_grader = grade_prompt | structured_llm_grader
question = "agent memory"
docs = retriever.get_relevant_documents(question)
doc_txt = docs[1].page_content
print(retrieval_grader.invoke({"question": question, "document": doc_txt}))

This should help you avoid the NotImplementedError by properly extending the Ollama model to use the with_structured_output method. For more detailed examples and different strategies to achieve structured output, you can refer to the LangChain documentation on structured output ^{[1][2][3][4][5]}.

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[Sridevi17j]

@dosu

1. from langchain_openai import Ollama - this line fro your code, will not work because - see below
   "ImportError: cannot import name 'Ollama' from 'langchain_openai'"

2. you have given the same code as mine,.... how do I extend my llm to use structured_output function? please give any example

[dosubot[bot]]

To extend your LLM to use the with_structured_output function, you need to ensure you're using the correct import and follow the proper steps. Here is a corrected and detailed example:

1. Define Your Schema: Create a Pydantic class that defines the schema for the structured output.

2. Instantiate the Ollama Model: Use the correct import for the Ollama model.

3. Use the with_structured_output Method: Call the with_structured_output method on the instance of Ollama with your schema.

4. Invoke the Model: Use the invoke method to get the structured output.

Here is an example using your provided code, with the correct import and steps:

from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_community.chat_models import ChatOllama
from langchain_core.prompts import ChatPromptTemplate

# Data model
class GradeDocuments(BaseModel):
    """Binary score for relevance check on retrieved documents."""
    binary_score: str = Field(description="Documents are relevant to the question, 'yes' or 'no'")

# LLM with function call
llm = ChatOllama(model="gemma:2b")
structured_llm_grader = llm.with_structured_output(GradeDocuments)

# Prompt
system = """You are a grader assessing relevance of a retrieved document to a user question. \n 
    It does not need to be a stringent test. The goal is to filter out erroneous retrievals. \n
    If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \n
    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question."""
grade_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        ("human", "Retrieved document: \n\n {document} \n\n User question: {question}"),
    ]
)

retrieval_grader = grade_prompt | structured_llm_grader
question = "agent memory"
docs = retriever.get_relevant_documents(question)
doc_txt = docs[1].page_content
print(retrieval_grader.invoke({"question": question, "document": doc_txt}))

This example demonstrates how to correctly extend your LLM to use the with_structured_output function with the ChatOllama model ^[1][2].

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[Ajay-Sai-Kiran]

from langchain_community.llms import Ollama

[Sridevi17j]

@dosu even the code given by you is throwing the same error...

[dosubot[bot]]

To resolve the NotImplementedError when using the with_structured_output method with the Ollama model, ensure you are importing Ollama from the correct module and using the method correctly. Here is a revised example:

1. Correct Import: Import Ollama from the langchain_community.llms module.
2. Define Your Schema: Create a Pydantic class for the structured output.
3. Instantiate the Ollama Model: Use the correct import for the Ollama model.
4. Use the with_structured_output Method: Call the with_structured_output method on the instance of Ollama with your schema.
5. Invoke the Model: Use the invoke method to get the structured output.

Here is the corrected code:

from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_community.llms import Ollama
from langchain_core.prompts import ChatPromptTemplate

# Data model
class GradeDocuments(BaseModel):
    """Binary score for relevance check on retrieved documents."""
    binary_score: str = Field(description="Documents are relevant to the question, 'yes' or 'no'")

# LLM with function call
llm = Ollama(model="gemma:2b")
structured_llm_grader = llm.with_structured_output(GradeDocuments)

# Prompt
system = """You are a grader assessing relevance of a retrieved document to a user question. \n 
    It does not need to be a stringent test. The goal is to filter out erroneous retrievals. \n
    If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \n
    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question."""
grade_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        ("human", "Retrieved document: \n\n {document} \n\n User question: {question}"),
    ]
)

retrieval_grader = grade_prompt | structured_llm_grader
question = "agent memory"
docs = retriever.get_relevant_documents(question)
doc_txt = docs[1].page_content
print(retrieval_grader.invoke({"question": question, "document": doc_txt}))

Ensure you have the correct dependencies installed and are using the latest versions of the relevant packages. If the issue persists, it might be related to specific configurations or limitations in the Ollama model's implementation of the with_structured_output method ^[1][2].

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[jamal-ansary]

Same here, i get raise NotImplementedError() when using ChatOllama. i tried my code with chat open AI and it works fine.

I also tested the example here: https://python.langchain.com/v0.1/docs/modules/model_io/chat/structured_output/
Open AI works fine, but ChatOllama gives error.

Edit:
I think i know where the issue is: they don't support Structured output with ollama.

https://python.langchain.com/v0.1/docs/integrations/chat/

[JonasBerweiler]

You can use with_structured_output by using langchain-experimental

• pip install langchain-experimental
• import OllamaFunctions from langchain-experimental: from langchain_experimental.llms.ollama_functions import OllamaFunctions
• replace llm = Ollama(model="gemma:2b") with llm = OllamaFunctions(model="gemma:2b")

[kallebl0mquist]

Than You for the explanation.

Ollama very recently added function calling. So it should be theoretically possible to go without OllamaFunctions.

https://ollama.com/blog/tool-support

Also in the ChatOllama() API documentation with_structured_output is listed as working (if i understand correctly.

https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.ollama.ChatOllama.html#langchain_community.chat_models.ollama.ChatOllama

After updating ollama, langchain, and langchain-community i still get notImplementedError.

So i am going back to OllamaFunctions but wonder whether that is as good.

[GuyPaddock]

I was able to get your code to work by switching the import from from langchain_community.chat_models import ChatOllama to from langchain_ollama import ChatOllama.

For example:

from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate


# Data model
class GradeDocuments(BaseModel):
    """Binary score for relevance check on retrieved documents."""
    binary_score: str = Field(description="Documents are relevant to the question, 'yes' or 'no'")

# LLM with function call
llm = ChatOllama(model='mistral-nemo:12b', temperature=0)
structured_llm_grader = llm.with_structured_output(GradeDocuments)

# Prompt
system = """You are a grader assessing relevance of a retrieved document to a user question. \n 
    It does not need to be a stringent test. The goal is to filter out erroneous retrievals. \n
    If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \n
    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question."""
grade_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        ("human", "Retrieved document: \n\n {document} \n\n User question: {question}"),
    ]
)

retrieval_grader = grade_prompt | structured_llm_grader

response = retrieval_grader.invoke({"question": "What is the answer to life, the universe, and everything?", "document": "The answer to life is 42."})

print(type(response))
print(response)

[sevenyd]

i tried all the above answers and this one is working, try it