api confusion

[zooteld]

I apologize for starting this question string in the issues. now that I am in the correct place... I am attempting to connect https://github.com/zooteld/llama-cpp-python-streamlit to my llama.cpp server running locally. I can see that something is actually hitting the server via the console, but I am not recieving anything back from the server to the app. The author of the app informed me its an endpoint issue, (uses differend json structure). it is wrote to use the llama-cpp-python bindings. any pointers on how to tackle this?

[KerfuffleV2]

Whatever sends requests to the server example would have to use the format that example expects. I took a very quick look at the repo you link. It seems like it may be using the OpenAI-style format. If that's the case, this might help you: https://github.com/ggerganov/llama.cpp/tree/master/examples/server#api-like-oai

I've never used it personally, so I probably won't be able to help with follow up questions on its use or anything like that.

[zooteld]

ya that is not working either... I noticed that to get it to connect I am having to change the /v1/chat/"/completion to just completion to match what is in my llama.cpp/example/server/public folder

[KerfuffleV2]

You're running it like this:

1. Start the server example.
2. Start the api_like_oai script and point it at the running server.
3. Run your streamlit thing and point that at whatever port api_like_oai is listening on.

Correct? I.E. api_like_oai is meant to work like a proxy and translate OpenAI-type requests into something server can understand.

[zooteld]

well i was starting it with the server and in my start command i have --path pointing to the examples/server/public folder so its launching the index.html

[KerfuffleV2]

It's not clear what "it" here is referring to.

To make this work you should have the server running on port 8080 for example. Then you start api_like_OAI.py - it will connect to 8080 by default, and listen for requests on port 8081 by default.

Therefore, your streamlit thing should be connecting to port 8081 and the paths, etc should be the same as if you were querying an OpenAI endpoint.

[zooteld]

I was referring to the api_like_OAI.py as it. lol

[KerfuffleV2]

It's not clear if you're saying it worked or not. You only showed output from the server example, and that shows requests with 200 status which indicates success.

[zooteld]

ah no Im still recieving nothing back to the app. an its still not showing a task in the console when the app requests from the api
but something is hitting the server tho

[KerfuffleV2]

You can try some other OpenAI-compatible endpoint or client to make sure that the api_like_OAI.py and server side isn't where the problem exists. If that works, then you can eliminate those as potential issues.

[S-Ali-Zaidi]

I had this same issue using api_like_OAI.py with the ChatBox program on my Mac. The endpoint at port 8081 using api_like_OAI.py is able to stream text to SOME UI programs like MacGPT, but it’s unable to do so with Chatbox.

But Chatbox is able to receive and display text that is sent using a server created by Studio LM (which is built on llama.cpp) as well as the OpenAI servers.. So not just why this is happening. I tried to Wireshark the outgoing chunks from both api_like_OAI.py and StudioLM and couldn’t find any difference that would break streaming functionality for some apps.

[KerfuffleV2]

You could possibly try #3972, it says it's fixing some api_like_OAI.py issues.

[zooteld]

for my use in the app I was beating my head against the desk trying to get working, the author of the origional version pushed an update enabling endpoint swapping. So now after launching the server and ai_like_OAI.py, the app connects to 8081 and uses /v1/completions as an endpoint. thank you all and especially KerfuffleV2 for putting up with me.

[KerfuffleV2]

No problem. Glad you managed to solve your problem!