Running vllm on runpod, how can I use it with llms.VLLM module? reverse proxy? What's your suggestion?

[al3rez]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

llm = VLLM(
    model="casperhansen/llama-3-70b-instruct-awq",
    trust_remote_code=True,  # mandatory for hf models
    max_new_tokens=128,
    top_k=10,
    top_p=0.95,
    temperature=0.8,
)

Description

I'm running vllm on runpod 4x GPU using HuggingFace and vllm (casperhansen/llama-3-70b-instruct-awq) how can I use the remote URL for the VLLM() to create a llm? I've done it using chainlit + requests, but I want to be able to do it using

llm = VLLM(
    model="casperhansen/llama-3-70b-instruct-awq",
    trust_remote_code=True,  # mandatory for hf models
    max_new_tokens=128,
    top_k=10,
    top_p=0.95,
    temperature=0.8,
)

Any ideas?

System Info

System Information

OS: Linux
OS Version: #202403110203171518180122.04~aba43ee SMP PREEMPT_DYNAMIC Wed M
Python Version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

Package Information

langchain_core: 0.2.1
langchain: 0.2.1
langchain_community: 0.2.1
langsmith: 0.1.63
langchain_anthropic: 0.1.13
langchain_openai: 0.1.7
langchain_text_splitters: 0.2.0
langgraph: 0.0.56

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langserve

[dosubot[bot]]

Hey @al3rez! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together while we wait for a human maintainer.

To use the remote URL for the VLLM class to create an LLM, you can set the model attribute to the remote URL. Here is an example of how you can instantiate the VLLM class with a remote URL:

from langchain_community.llms.vllm import VLLM

# Example remote URL for the model
remote_model_url = "https://huggingface.co/remote-model-url"

# Create an instance of VLLM with the remote URL
llm = VLLM(
    model=remote_model_url,
    tensor_parallel_size=4,  # Adjust based on your GPU setup
    trust_remote_code=True,  # Set to True if you trust the remote code
    max_new_tokens=128,
    top_k=10,
    top_p=0.95,
    temperature=0.8,
)

Regarding whether to use a reverse proxy or a better approach, using a reverse proxy can be beneficial for load balancing, SSL termination, and other features. If you are deploying the model in a production environment, a reverse proxy like Nginx or Traefik could be a good choice.

Alternatively, you can use the OpenLLM class from the langchain_community library, which supports both in-process model instances and remote OpenLLM servers. Here is an example of how to use it with a remote server:

from langchain_community.llms import OpenLLM

# Example remote server URL
server_url = 'http://localhost:3000'

# Create an instance of OpenLLM with the remote server URL
llm = OpenLLM(server_url=server_url)
llm.invoke("What is the difference between a duck and a goose?")

This approach allows you to leverage the OpenLLM class's built-in support for remote servers, which might simplify your setup and provide additional features ^[1][2].

To continue talking to Dosu, mention @dosu.

[al3rez]

how can i do bind_tools with this?

[dosubot[bot]]

To bind tools with the VLLM class setup, you can use the bind_tools method. Here is an example of how to do this:

from langchain_community.llms.vllm import VLLM
from langchain_core.tools import BaseTool

# Initialize the VLLM model
vllm_model = VLLM(
    model="casperhansen/llama-3-70b-instruct-awq",
    tensor_parallel_size=4,  # Use 4 GPUs for distributed execution
    trust_remote_code=True,  # Trust remote code if necessary
    max_new_tokens=128,
    top_k=10,
    top_p=0.95,
    temperature=0.8,
)

# Define your tool
class MyTool(BaseTool):
    def run(self, input_data):
        # Tool logic here
        return "Tool output"

tools = [MyTool()]

# Bind tools to the VLLM model
vllm_model.bind_tools(tools)

# Example usage
prompts = ["Your prompt here"]
result = vllm_model._generate(prompts)
print(result.generations)

This configuration sets up the VLLM model to use 4 GPUs for tensor parallelism and binds the tools to the model using the bind_tools method. Adjust other parameters as needed for your specific use case ^[1].

To continue talking to Dosu, mention @dosu.

[Dekel-Eli]

@al3rez Hey, did you by any chance get a solution for this?
if I run a vllm serverless where do i insert the url and api key (ex. 'https://api.runpod.ai/v2/[id]/ and 'rpa_...')

[spyros724]

Have you found any solution to this?

[spyros724]

This is useful:

https://python.langchain.com/docs/integrations/providers/runpod/