Support llama.cpp

[ParetoOptimalDev]

I can't get ollama to work with gpu accelleration, so I'm using llama.cpp which has a Nix flake that worked perfectly (once I understood "cuda" was the cuda version and not the cuda library) 😍

It looks like llama.cpp has a different api so I can't just use (gptel-make-ollama. This sound correct?

Then again I see something about llama-cpp-python having an "OpenAI-like API". The downside of this being I'll have to package llama-cpp-python for nix

Maybe I can use that and gptel somehow? Just looking for a bit of guidance, but will tinker around when I get time and try things. If I find anything useful I'll report back here.

[ParetoOptimalDev]

Good news, looks like llama-cpp-python is packaged by this awesome repo:

https://github.com/nixified-ai/flake

and I'll soon find out if it can be run with:

nix shell github:nixified-ai/flake -c llama-cpp-python

Edit: hmm doesn't expose that, will dig in more later to tryout compatibility of llama-cpp-python and gptel.

[karthink]

I couldn't find any info on the llama-cpp-python's web API except for what's in the Github README but if what it says is correct, support for it in gptel should be trivial:

(defvar gptel--llama-cpp-python 
  (gptel-make-openai
   "llama-cpp-python"
   :stream t                   ;If llama-cpp-python supports streaming responses
   :protocol "ws"
   :host "localhost:8000"
   :endpoint "/api/v1/chat-stream"
   :models '("list" "of" "available" "model" "names"))
  "GPTel backend for llama-cpp-python.")

;; Make it the default
(setq-default gptel-backend gptel--llama-cpp-python
              gptel-model   "name")

Unfortunately I can't test this -- no GPU, and I'm also on Nix so it's not easy to install.

[karthink]

It looks like llama.cpp has a different api so I can't just use (gptel-make-ollama. This sound correct?

Do you have a link to llama.cpp's (not -python) API documentation?

EDIT:

I can't get ollama to work with gpu accelleration

Incidentally, I couldn't get it to run on NixOS at all, and couldn't get the package to build when I tried the latest version. The latest binary release from Ollama worked perfectly (including GPU support) on Arch on a different machine.

[ParetoOptimalDev]

There isn't any, I found this related issue:

ggml-org/llama.cpp#1742

That's where I learned about llama-cpp-python.

[ParetoOptimalDev]

I think that I'm going to be able to use what you linked above after this finishes (but it's 17GB):

nix run github:nixified-ai/flake#packages.x86_64-linux.textgen-nvidia 

[karthink]

I think that I'm going to be able to use what you linked above after this finishes (but it's 16GB or more):

Cool, please let me know if it works as expected -- including the streaming responses bit.

[ParetoOptimalDev]

It doesn't seem to work. I noticed there are examples in the text-generation-webui repo though:

https://github.com/oobabooga/text-generation-webui/blob/main/api-examples/api-example-chat-stream.py

So I modified the above to use 5005:

(defvar gptel--llama-cpp-python 
      (gptel-make-openai
       "llama-cpp-python"
       :stream t               ;If llama-cpp-python supports streaming responses
       :protocol "http"
       :host "localhost:5005"
       :models '("nous-hermes-llama2-13b.Q4_0.gguf"))
      "GPTel backend for llama-cpp-python.")

It still didn't work and gave a 404 though.

[karthink]

It still didn't work and gave a 404 though.

I edited the snippet (added an :endpoint field), any luck?

EDIT: Also it looks like the protocol is not http, it's ws. I'm checking if Curl handles that...

[ParetoOptimalDev]

More details on their endpoint support: https://github.com/oobabooga/text-generation-webui/blob/262f8ae5bb49b2fb1d9aac9af01e3e5cd98765db/extensions/openai/README.md?plain=1#L190

[ParetoOptimalDev]

It still didn't work and gave a 404 though.

I edited the snippet (added an :endpoint field), any luck?

EDIT: Also it looks like the protocol is not http, it's ws. I'm checking if Curl handles that...

Ah, you are right. It didn't work. curl should support ws.

[karthink]

Did you try it with the :protocol set to "ws"?

[karthink]

Ah, I just realized it's going to fail anyway because gptel expects a HTTP 200/OK message. But it will help to check if the API works as expected with the following Curl command:

curl --location --silent --compressed --disable -XPOST -w(abcdefgh . %{size_header}) -m60 -D- -d'{"model":"nous-hermes-llama2-13b.Q4_0.gguf","messages":[{"role":"system","content":"You are a large language model living in Emacs and a helpful assistant. Respond concisely."},{"role":"user","content":"Hello"}],"stream":true,"temperature":1.0}' -H"Content-Type: application/json" "ws://localhost:5005/api/v1/chat-stream"

The output will help me add support for it as well.

[ParetoOptimalDev]

$ curl --location --silent --compressed --disable -XPOST -w(abcdefgh . %{size_header}) -m60 -D- -d'{"model":"nous-hermes-llama2-13b.Q4_0.gguf","messages":[{"role":"system","content":"You are a large language model living in Emacs and a helpful assistant. Respond concisely."},{"role":"user","content":"Hello"}],"stream":true,"temperature":1.0}' -H"Content-Type: application/json" "ws://localhost:5005/api/v1/chat-stream"
Malformed access time modifier ‘a’
$ curl --location --silent --compressed --disable -XPOST -w "(abcdefgh . %{size_header})" -m60 -D- -d'{"model":"nous-hermes-llama2-13b.Q4_0.gguf","messages":[{"role":"system","content":"You are a large language model living in Emacs and a helpful assistant. Respond concisely."},{"role":"user","content":"Hello"}],"stream":true,"temperature":1.0}' -H"Content-Type: application/json" "ws://localhost:5005/api/v1/chat-stream"
(abcdefgh . 0)

[ParetoOptimalDev]

I'm actually unable to get textgen from the nixified-ai flake working anyway, or just lllama-cpp-python. I might look at interoperating purely with llama.cpp again.

Reason being it's hard to tell which versions of llama-cpp-python will even work with llama-cpp and I don't understand how to debug them well.

[karthink]

Hmm, I'm guessing I need to look into Curl's websocket support. I don't think there's a quick fix to support llama-cpp-python in gptel after all.

Reason being it's hard to tell which versions of llama-cpp-python will even work with llama-cpp and I don't understand how to debug them well.

Local LLM support is a bit of a mess across the board right now.