Dipping my toes into local LLM - recommended container image for ollama?

[psyciknz]

I've been following this project for a few weeks and wanted to try it out now.

If there a recommended container for ollama? I was intending to run it on a 2nd machine and more than likely behind a reverse proxy, but initially just want a container that paperless-gpt can utilise for it's processing.

[icereed]

Hey 👋
Welcome aboard 🫶

On with operating system do you plan to run ollama? Sounds like Linux to me, but I don’t want to merely assume.

[icereed]

I tried LLMs once without GPU. It’s almost unusable 😅
LLMs need GPUs. That’s why chip makers are getting so rich with GPUs right now 😂

If you want to tune your setup even more and hide ollama behind an API key you can try https://docs.litellm.ai/docs/simple_proxy.

It adds a little complexity to your setup, but you can plug a lot of different LLMs into it.

[psyciknz]

So that would be proxying the public llms. Like OpenAI etc?

[icereed]

It supports local LLMs, too 😊

[icereed]

https://docs.litellm.ai/docs/providers/ollama#using-ollama-apichat

[icereed]

I’m not sure yet, if LiteLLM supports the endpoints for Visual LLMs like discussed here #20