Support for OpenAI API Compatible VLMs

[yibudak]

Although Local VLM is recommended for those who cares about privacy but qwen2.5vl:3b model requires approximately 28 GB of RAM. Most people's computers are not yet powerful enough to run local VLM. The only remaining option is Gemini, but many people don't trust Google either. Therefore, people need to be able to use their own custom endpoints for other models. This would also allow us to use providers such as OpenAI and OpenRouter.

[JerryZLiu]

Hm, qwen2.5vl:3b only takes up about 3-4 GB of ram. I picked it because most people with 16GB ram should be able to run it. Is it taking up 28GB of ram on your machine?

[grunsab]

That's the size on the disk, not the amount of RAM it's taking.

[JerryZLiu]

Actually that is the ram, basically what's happening here is you've somehow set the context window to the max, 131k tokens. The amount of memory LLMs consume is roughly quadratic relative to the context window. This is what mine consumes with a smaller context window.

[JerryZLiu]

Did you do something weird to set the context window? You can adjust the default context window in ollama settings. I recommend 4k, Dayflow doesn't need that much since it chunks the prompts.

[yibudak]

Ah yes, context... Now it takes the same amount RAM as like your screenshot.

But this feature request still relevant. 😊

[JerryZLiu]

yep will try to add that ability soon, for what it's worth LMStudio running the same model is only 3GB, which seems pretty manageable for most people.