Feature request: support for SGLang and Ollama

[rachittshah]

Hi folks!

Since the current project only supports vLLM, I'd love to propose support for SGLang and Ollama.

Thinking behind proposal:

Many teams use SGLang over vLLM and vice versa. Adding support will allow for greater adoption.

Ollama is one of the quickest ways to get started with on prem LLMs, reduces friction and increases the DX.

Happy to shot for a PR for this!

[init27]

Hi @rachittshah!

Thanks so much for checking out the project and the proposal!

I completely agree and it would be really great to have a PR from you-thanks for the offer!

IMO, it would be awesome to have Ollama support first if you're interested in taking it up. llm_client.py is a great place to start looking.

Please let me know if you have any questions!

[rachittshah]

Sounds good @init27 !

Thank you for the pointer, let me shoot a PR as soon as I'm done with this.

Will use this thread for questions.

[rachittshah]

Raised #4 which solves for Ollama.