openai api compatibility

[iplayfast]

I know that llama-cpp-python has openai compatibility https://abetlen.github.io/llama-cpp-python/ but it seems to me this functionality would be better in llama.cpp's server.

The server currently has an api, but I'm not aware of any documentation for it, or how compatible it is to openai.
I'm using autogen with text-generation-webui and it works. I'd rather keep things simpler and just use the llama.cpp server.

[ianscrivener]

Agree!! It would be fantastic for llama.cpp to have an OpenAI compatible API.

[ianscrivener]

Yes, good point re prompt formatting
Prompt formats are data - and should be separate from code, like models. LM-Studio has handled models and inference config (including prompts) very elegantly

Default and custom prompt formats would be easy to do with a well designed API

For an API I'd be suggesting full llama.cpp funcitonality.. in addition to OpenAI compatible endpoints for LLM inference and embeddings (not all OpenAI stuff). Eg;

/llama_cpp/xxxx .. _essentially ./main_ 
/llama_cpp/prompts/
/llama_cpp/prompts/{prompt}
/llama_cpp/llava
/llama_cpp/models
/llama_cpp/models/{model}
/llama_cpp/perplexity

openai/chat/completions
openai/completions
openai/embeddings
openai/models
openai/models/{model}

[shibe2]

Lm-Studio has handled models and prompts very elegantly - https://github.com/lmstudio-ai/model-catalog/tree/main/models

I'm not familiar with it, and I don't see anything related to prompts at that URL.

[ianscrivener]

Sorry, here's an exampe of LM Studio inference config including prompt:
https://gist.github.com/ianscrivener/9c57078a6d2e8891f83c26fc34e19d4f
(I've updated my post)

[shibe2]

Does it handle special tokens like in ChatML?

[ianscrivener]

/llama_cpp/prompts                       ... baked in prompts & custom prompts
/llama_cpp/prompts/{prompt}
/llama_cpp/presets                        ... inference param presets, including referece to prompt
/llama_cpp/presets/{preset}

[shibe2]

Have you tried api_like_OAI.py?

[shibe2]

Server API documentation. It's not comprehensive and may get out of date sometimes, but it's enough to get you started.

[theoparis]

This doesn't seem to have functions/tools support 😕

[iplayfast]

@shibe2 thanks for those links. Was not aware of either of them.