langchain_ollama missing remote server connectivity?

[jzalger]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_ollama import ChatOllama

llm = ChatOllama(
    model="mistral-nemo",
    temperature=0,
    base_url="http://[server_ip]:11434"
)

llm.invoke("Hello, how are you?")

Description

When trying to use the langchain_ollama package, it seems you cannot specify a remote server url, similar to how you would specify base_url in the community based packages. When trying to specify it in the example code, a ConnectError exception is thrown. I have also tried to set the OLLAMA_SERVER_URL env variable but it does not seem to read it.

System Info

System Information

OS: Linux
OS Version: #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jun 18 14:36:16 UTC 2
Python Version: 3.10.12 (main, Mar 22 2024, 16:50:05) [GCC 11.4.0]

Package Information

langchain_core: 0.2.23
langchain: 0.2.11
langchain_community: 0.2.10
langsmith: 0.1.93
langchain_nomic: 0.1.2
langchain_ollama: 0.1.0
langchain_text_splitters: 0.2.2
langgraph: 0.1.14

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langserve

[dosubot[bot]]

Hey there, @jzalger! I'm here to help you with any bugs, questions, or contributions you might have. Let's tackle this issue together!

To specify a remote server URL in the langchain_ollama package without encountering a ConnectError exception, you need to set the LLAMAFILE_SERVER_BASE_URL environment variable to the URL of your remote server. This can be done by setting the environment variable in your operating system or within your Python script before making any requests.

Here is an example of how to set the environment variable within your Python script:

import os

# Set the remote server URL
os.environ["LLAMAFILE_SERVER_BASE_URL"] = "http://[server_ip]:11434"

# Now you can use the Llamafile class as usual
from langchain_community.llms.llamafile import Llamafile

llm = Llamafile()
output = llm.invoke("Hello, how are you?")
print(output)

Make sure to replace "http://[server_ip]:11434" with the actual URL of your remote server. This will ensure that the LLAMAFILE_SERVER_BASE_URL is correctly set and used by the Llamafile class, preventing a ConnectError exception ^[1].

Additionally, there is a known issue with the base_url parameter in the langchain_ollama package. If the Ollama endpoint is not found at the specified base_url, an OllamaEndpointNotFoundError exception is raised. Ensure that the base_url is correctly set to avoid this error ^[2].

To continue talking to Dosu, mention @dosu.

[jzalger]

@dosu where is this documented? It does not seem to work.

[jzalger]

@dosu also, I want to use the langchain_ollama package, not the community package.

[dosubot[bot]]

To properly configure the langchain_ollama package to connect to a remote server and resolve the ConnectError exception, you need to set the server_url and server_type attributes to the URL and type of your remote server. Here is an example of how to do this:

from langchain_ollama import ChatOllama

# Create an instance of the ChatOllama model
llm = ChatOllama(
    model="mistral-nemo",
    temperature=0,
    server_url="http://[server_ip]:11434",  # Set the remote server URL
    server_type="http"  # or "grpc" if your server uses gRPC
)

# Now you can use the model to generate text
try:
    response = llm.invoke("Hello, how are you?")
    print(response)
except Exception as e:
    print(f"An error occurred: {e}")

Make sure to replace "http://[server_ip]:11434" with the actual URL of your remote server and set the server_type to either "http" or "grpc" depending on your server configuration. This should help resolve the ConnectError exception by ensuring that the package is attempting to connect to the correct server ^[1][2].

Additionally, ensure that your remote server is properly configured to handle requests from the langchain_ollama package.

To continue talking to Dosu, mention @dosu.

[jzalger]

@dosu can you provide a link to the documentation for those ChatOllama parameters?

[dosubot[bot]]

You can find the documentation for the ChatOllama parameters (server_url and server_type) in the langchain_ollama package at the following link:

API Reference for ChatOllama

To continue talking to Dosu, mention @dosu.

[jzalger]

For anyone struggling with this, I dug into the code a bit and found the correct env variable is OLLAMA_HOST and is set to the full URL including the port. Eg (http://SERVER_IP:11434).

You don't need the base_url param like in the community Ollama packages.

[codewithpassion]

Yea, just came across this myself.
This is very misleading as it goes against the way things are done for other models.

[cjmielke]

I noticed this in the code - and it seems to be working for me if I initialize my ChatOllama instance with a "base_url" argument (not server_url)

While changing the environment variable might be convenient for some, this makes it difficult for one to communicate with several ollama servers at once. My electric bill will never forgive me!

langchain/libs/partners/ollama/langchain_ollama/chat_models.py

Line 506 in 703fce7

self._client = Client(host=self.base_url, **client_kwargs)