what is the difference between litellm and vllm?

[YunchaoYang]

Wondering what is the major difference between the vllm and litellm? Is there a way for them to work together?

[krrishdholakia]

Hey @YunchaoYang yes! You can call vllm through the litellm proxy - https://docs.litellm.ai/docs/providers/openai_compatible

VLLM is openai-compatible.

LiteLLM Proxy

1. Create config.yaml with vllm model(s)

model_list:
  - model_name: my-model
    litellm_params:
      model: openai/<your-model-name>  # add openai/ prefix to route as OpenAI provider
      api_base: <model-api-base>       # add api base for OpenAI compatible provider
      api_key: api-key                 # api key to send your model

2. Start the proxy

litellm --config /path/to/config.yaml

3. Send Request to LiteLLM Proxy Server

curl --location 'http://0.0.0.0:4000/chat/completions' \
    --header 'Authorization: Bearer sk-1234' \
    --header 'Content-Type: application/json' \
    --data '{
    "model": "my-model",
    "messages": [
        {
        "role": "user",
        "content": "what llm are you"
        }
    ],
}'

[krrishdholakia]

LiteLLM is just a proxy that handles calling different LLM API's easily