Couldn't initialize an LLM managed by OpenLLM locally from current process

[bloomingseed]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_community.llms import OpenLLM

llm = OpenLLM(
    model_name="dolly-v2",
    model_id="databricks/dolly-v2-3b",
    temperature=0.94,
    repetition_penalty=1.2,
)

Description

I couldn't initialize an LLM managed by OpenLLM locally from current process. Running the example code from the docs page on OpenLLM resulted in the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-5-99412c629874>](https://localhost:8080/#) in <cell line: 9>()
      7 #     repetition_penalty=1.2,
      8 # )
----> 9 llm = OpenLLM(
     10     model_name="dolly-v2",
     11     model_id="databricks/dolly-v2-3b",

[/usr/local/lib/python3.10/dist-packages/langchain_community/llms/openllm.py](https://localhost:8080/#) in __init__(self, model_name, model_id, server_url, timeout, server_type, embedded, **llm_kwargs)
    171             # behaviour to invoke the runners remotely.
    172             # We need to also enable ensure_available to download and setup the model.
--> 173             runner = openllm.Runner(
    174                 model_name=model_name,
    175                 model_id=model_id,

AttributeError: module 'openllm' has no attribute 'Runner'

System Info

System Information

OS: Linux
OS Version: #1 SMP PREEMPT_DYNAMIC Thu Jun 27 21:05:47 UTC 2024
Python Version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0]

Package Information

langchain_core: 0.2.29
langchain: 0.2.12
langchain_community: 0.2.11
langsmith: 0.1.98
langchain_text_splitters: 0.2.2

Optional packages not installed

langgraph
langserve

Other Dependencies

aiohttp: 3.10.1
async-timeout: 4.0.3
dataclasses-json: 0.6.7
jsonpatch: 1.33
numpy: 1.26.4
orjson: 3.10.7
packaging: 24.1
pydantic: 2.8.2
PyYAML: 6.0.1
requests: 2.32.3
SQLAlchemy: 2.0.31
tenacity: 8.5.0
typing-extensions: 4.12.2

openllm -v output

openllm, 0.6.7
Python (CPython) 3.10.12

[dosubot[bot]]

Hey @bloomingseed! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together!

To resolve the AttributeError: module 'openllm' has no attribute 'Runner', you need to ensure that the openllm module is correctly imported and that the Runner class exists within the openllm module. Based on the provided code, the Runner class is used to initialize the LLM instance locally.

Here is the relevant part of the code where Runner is used:

runner = openllm.Runner(
    model_name=model_name,
    model_id=model_id,
    init_local=embedded,
    ensure_available=True,
    **llm_kwargs,
)

If you encounter the AttributeError, it is likely that the Runner class is either not defined in the openllm module or there is an issue with the import. To resolve this, you should:

1. Verify that the openllm module is installed correctly.
2. Check the openllm module documentation or source code to ensure that the Runner class is defined and accessible.

If the Runner class is indeed missing or incorrectly referenced, you may need to update the openllm library or adjust your code to match the correct usage as per the latest documentation.

If the Runner class is present in the openllm module, ensure that you have the correct version of the openllm library installed. You can update the library using:

pip install --upgrade openllm

If the issue persists, you may need to consult the openllm documentation or seek further assistance from the library maintainers ^[1].

To continue talking to Dosu, mention @dosu.