how to use litellm pipelines

[flyfox666]

I have been using litellm to call some of the interfaces I have built, but now I have changed to pipelines. I can't get it to work, so I'm looking for guidance.

[justinh-rahb]

For a basic setup that will support Anthropic and Cohere APIs as an example:

docker run -d \
  --name pipelines \
  -p 9099:9099 \
  -e ANTHROPIC_API_KEY=sk-ant-12345 \
  -e COHERE_API_KEY=12345 \
  -e PIPELINES_URLS="https://raw.githubusercontent.com/open-webui/pipelines/main/examples/pipelines/providers/anthropic_manifold_pipeline.py;https://raw.githubusercontent.com/open-webui/pipelines/main/examples/pipelines/providers/cohere_manifold_pipeline.py \
  --restart always \
  ghcr.io/open-webui/pipelines:main

[flyfox666]

Ok, thanks for the tip, I downloaded the litellm py in open webui as follows:

! Screenshot 2024-06-03 232420

But it keeps telling me that my previous config.yaml file doesn't exist, and I've tried several paths, but none of them are correct.

What I want to know is whether the litellm under pipelines is the same as having a py file of my own. All my previous api's, some of which I built myself, are implemented through litellm, so I want to keep using them~.

Of course, it would be nice to have a guide on the logic and principles of how pipelines work nowadays this is best.

Really thanks

[justinh-rahb]

Here's a guide to help you move your litellm setup out of the open-webui container and into its own separate container. This will allow you to use litellm independently and connect it to open-webui as needed.

1. Find your config.yaml file

Your config.yaml file should be located in the open-webui container. To find it, use the following command:

docker cp open-webui:/app/backend/data/litellm/config.yaml .

This will copy the config.yaml file from the open-webui container to your current directory.

2. Run a standalone litellm container

To run litellm in its own container, use the following docker run command:

docker run -d \
  -p 4000:4000 \
  --name litellm \
  -v ./config.yaml:/app/config.yaml \
  -e LITELLM_MASTER_KEY=sk-12345 \
  --restart always \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml --port 4000

Replace ./config.yaml with the path to the config.yaml file you copied from the open-webui container. Set the LITELLM_MASTER_KEY environment variable to your API key value.

3. Connect litellm to open-webui

Once you have the litellm container running, you can connect it to the open-webui container. In the open-webui settings, add an "OpenAI" base URL under "Connections" with the value http://host.docker.internal:4000/v1 and set the API key to your LITELLM_MASTER_KEY value (e.g., sk-12345).

By following these steps, you'll have moved your litellm setup out of the open-webui container and into its own separate container. You can now use litellm independently and connect it to open-webui as needed.

[flyfox666]

So that‘s it. That’s why I‘m stuck! Thank

[josephbkfung]

Thanks for the instruction . I used https://raw.githubusercontent.com/open-webui/pipelines/main/examples/pipelines/providers/anthropic_manifold_pipeline.py and my Anthropic API to setup Claude 3. Works for text message. But if I upload a file, I got a Error 422 Unprocessable Entity error. Is this the limitation of LiteLLM or the example python code?

[justinh-rahb]

If you're using Pipelines and the Anthropic provider pipeline script, then you're not using LiteLLM at all. And no, it currently does not support input of files or images, only text.