ChatOllama is only getting/using the last four objects from the JSON

[DirenCA]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_ollama.chat_models import ChatOllama
from langchain_community.agent_toolkits.openapi.toolkit import RequestsToolkit
from langchain_community.utilities.requests import TextRequestsWrapper
from langgraph.prebuilt import create_react_agent


ALLOW_DANGEROUS_REQUEST = True

toolkit = RequestsToolkit(
    requests_wrapper=TextRequestsWrapper(headers={}),
    allow_dangerous_requests=ALLOW_DANGEROUS_REQUEST,
)

# Füge das JsonToolkit zu deinen bestehenden Tools hinzu
tools = toolkit.get_tools()


system_message = """
You are an AI assistant with access to an API to help answer user queries.
Only use the data from the API call  to answer questions.
Given the user's query, you must decide what to do with it based on the
list of tools provided to you. Always use 'http://localhost:8082/FROST-Server/v1.1' as base url.
Use the whole data that requests_get gives you.
"""

#Initialize the LLM
llm = ChatOllama(
    model="llama3.1",
    temperature=0,
    max_tokens=10000,
    timeout=None,
)

query = "How many Datastreams are there?"

agent_executor = create_react_agent(llm, tools, state_modifier=system_message)

events = agent_executor.stream(
     {"messages": [("user", query)]},
     stream_mode="values",
)

for event in events:
    event["messages"][-1].pretty_print()

Description

Hello!

I´m currently trying to build a Chatbot which executes a GET request to gather data from my Database(FROST-Server). The request works fine and llama is smart enough to decide which enpoints to call most of the time. The problem is that it seems that my llm is not using or getting all the objects of the JSON. So asking something like "How many Datastreams are there?" (the objects are called 'Datatstream') will always give me the last four objects as output and not even the number 4.

This is how the last objects look like:
{"@iot.selfLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(102)","@iot.id":102,"name":"Bewässerung des Baumes","description":"Die Menge an Wasser, die für die Bewässerung des Baumes gemessen wurde","observationType":"OM_Measurement","unitOfMeasurement":{"name":"Liter","symbol":"L","definition":"ucum:L"},"observedArea":{"type":"Point","coordinates":[52.48906,13.34923]},"phenomenonTime":"2024-07-23T00:00:00Z/2024-07-23T00:00:00Z","ObservedProperty@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(102)/ObservedProperty","Sensor@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(102)/Sensor","Thing@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(102)/Thing","Observations@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(102)/Observations"},{"@iot.selfLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(103)","@iot.id":103,"name":"Bewässerung des Baumes","description":"Die Menge an Wasser, die für die Bewässerung des Baumes gemessen wurde","observationType":"OM_Measurement","unitOfMeasurement":{"name":"Liter","symbol":"L","definition":"ucum:L"},"observedArea":{"type":"Point","coordinates":[52.50875,13.40275]},"phenomenonTime":"2024-07-19T00:00:00Z/2024-07-19T00:00:00Z","ObservedProperty@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(103)/ObservedProperty","Sensor@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(103)/Sensor","Thing@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(103)/Thing","Observations@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(103)/Observations"},{"@iot.selfLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(104)","@iot.id":104,"name":"Bewässerung des Baumes","description":"Die Menge an Wasser, die für die Bewässerung des Baumes gemessen wurde","observationType":"OM_Measurement","unitOfMeasurement":{"name":"Liter","symbol":"L","definition":"ucum:L"},"observedArea":{"type":"Point","coordinates":[52.50131,13.37639]},"phenomenonTime":"2024-07-24T00:00:00Z/2024-07-24T00:00:00Z","ObservedProperty@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(104)/ObservedProperty","Sensor@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(104)/Sensor","Thing@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(104)/Thing","Observations@iot.navigationLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams(104)/Observations"}],"@iot.nextLink":"http://localhost:8082/FROST-Server/v1.1/Datastreams?$skip=100&$orderby=%40iot.id+asc&$skipFilter=%28%40iot.id+gt+104%29"}

System Info

platform Windows
Python Version 3.12.0

[dosubot[bot]]

Hello, @DirenCA! I'm here to help you with any questions or issues you may have. Let's work together to solve this bug.

The issue you're encountering with the ChatOllama model only using the last four objects from the JSON response might be related to the model's configuration or limitations. Specifically, the request_timeout parameter, which is set to 60 seconds by default, could limit the number of JSON objects processed within that time frame. Additionally, the service_url must be correctly configured; otherwise, the service will not function properly ^[1].

To address this, you can try the following steps:

1. Increase the request_timeout: Ensure that the request_timeout parameter is set to a higher value to allow more time for processing larger datasets.

2. Check the service_url: Verify that the service_url is correctly configured and accessible.

3. Pagination Handling: Ensure that your implementation correctly handles pagination if the API response includes a nextLink for additional data. You may need to implement logic to follow these links and aggregate the results.

Here's an example of how you might adjust the request_timeout:

llm = ChatOllama(
    model="llama3.1",
    temperature=0,
    max_tokens=10000,
    timeout=120,  # Increase timeout to 120 seconds
)

Additionally, ensure that your code handles pagination by following the @iot.nextLink provided in the JSON response to fetch all available data.

To continue talking to Dosu, mention @dosu.

[DirenCA]

@dosu
increasing the timeout didn´t help.
what do you mean with service_url? 'http://localhost:8082/FROST-Server/v1.1' is the correct url.
i don´t think pagination is the problem. even without following '@iot.nextLink' there are more than four objects listed.

[dosubot[bot]]

The issue with the ChatOllama model only using the last four objects from the JSON response is likely due to the way the _stream method processes the response. Specifically, the code snippet:

substring = '"object":"chat.completion.chunk"}'
for line in res.iter_lines():
    chunks = []
    if line:
        json_string = line.decode("utf-8")

        # Find all positions of the substring
        positions = [m.start() for m in re.finditer(substring, json_string)]
        positions = [-1 * len(substring)] + positions

        for i in range(len(positions) - 1):
            chunk = json.loads(
                json_string[
                    positions[i] + len(substring) : positions[i + 1]
                    + len(substring)
                ]
            )
            chunks.append(chunk)

This code is designed to split the JSON response into chunks based on the substring '"object":"chat.completion.chunk"}'. If the response contains multiple such substrings, the code will only process the last four chunks due to the way the positions list is constructed and iterated over. This could lead to only the last four objects being used.

To fix this, ensure that the entire JSON response is correctly parsed and all chunks are processed. You might need to adjust the logic for splitting the JSON response or handle the response differently to ensure all objects are included.

Additionally, the ChatOllama class in the LangChain framework provides methods like _chat_stream_with_aggregation and _create_chat_stream that handle streaming responses. These methods aggregate chunks correctly and ensure that all parts of the response are processed. You might want to use or adapt these methods to handle the JSON response more effectively. Here is an example of how the _chat_stream_with_aggregation method works:

def _chat_stream_with_aggregation(
    self,
    messages: List[BaseMessage],
    stop: Optional[List[str]] = None,
    run_manager: Optional[CallbackManagerForLLMRun] = None,
    verbose: bool = False,
    **kwargs: Any,
) -> ChatGenerationChunk:
    final_chunk: Optional[ChatGenerationChunk] = None
    for stream_resp in self._create_chat_stream(messages, stop, **kwargs):
        if stream_resp:
            chunk = _chat_stream_response_to_chat_generation_chunk(stream_resp)
            if final_chunk is None:
                final_chunk = chunk
            else:
                final_chunk += chunk
            if run_manager:
                run_manager.on_llm_new_token(
                    chunk.text,
                    chunk=chunk,
                    verbose=verbose,
                )
    if final_chunk is None:
        raise ValueError("No data received from Ollama stream.")

    return final_chunk

This method ensures that all chunks from the stream are aggregated into a final result, preventing the issue of only processing the last few objects ^[1][2][3].

To continue talking to Dosu, mention @dosu.