OpenAI compatible backends

[arty-hlr]

Hi,

I see you support vLLM and are planning on supporting ollama and cerebras. Wouldn't it make more sense to support openAI compatible backends instead? This would include all of the above and lmstudio, but also remote providers like openAI, together.ai, openrouter, etc. Or what would the issue be with that?

Looking forward to using this project, seems really promising!

Best,
Florian

[arty-hlr]

Looking through the existing PRs, it seems that #4 and #5 would be covering both ollama directly and specifying the base URL for an openAI compatible API endpoint.

[init27]

This is now merged to main and fixed, please change your API endpoint and API key and everything should work OOBX!

[arty-hlr]

@init27 I was able to make it work with lmstudio, with a few caveats:

• configs/config.yaml is not read by default, it doesn't change the config until I set explicitly -c configs/config.yaml
  
  
• the documentation (README.md) should be updated to show that you can use other openai compatible endpoints, this was not part of the PR
• something more robust should be used for ingesting (compare with the output of docling on the right for example, no unnecessary line breaks, markdown formatting, etc, maybe even image descriptions with a vision model could be added)
  

Let me know if I should make separate issues for those, I don't have time tonight but I can make some time tomorrow, also if the problems are unclear. This is simikar to something we've been doing at work, so let me know how I can help.

[init27]

@arty-hlr first, thank you for being so active in the repo. We are very grateful for your contributions!

Next, thanks for pointing out the gaps:

1. I will fix and clarify this.
2. Dcoumentation-this would be the best place to have your contribution. I'd like to fix the remaining bugs so it's a low lift for you at the moment.
3. For ingesting, I would be open to any thoughts you'd have on this. Right now it's quite basic indeed.
4. For captioning images, we are planning to add support in next 2-3 weeks. Please let me know if you have special asks

[arty-hlr]

@init27

1. Thanks!
2. PR for the README is there, I added a comment about api-endpoint not being referenced as openAI compatible which I think would be necessary to change back.
3. I don't really know the internals of docling, but I'd suggest taking a look at how it's generating markdown
4. I would say either using a vision model or basic OCR as options

Another piece of feedback, I noticed that there are a lot of PRs into main, I believe that creating at least a dev branch and potentially feature branches and releases would benefit this repo a lot

[init27]

@arty-hlr thanks so much, looking forward to your PR and once again thank you SO MUCh for being so pro active on this repo!

I wasn't aware of docling, will check it out today!

Thanks for the feedback, I agree-will add branches.