Build with llama support does not build llama support

[mooleshacat]

LocalAI version:

Master branch

root@AI0:~/localai# ./LocalAI --version
LocalAI version  ()

Environment, CPU architecture, OS, and Version:

Linux AI0 6.1.0-37-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.140-1 (2025-05-22) x86_64 GNU/Linux
AMD 5950x 32GB ram 2x rtx 3070 1x rtx 2070 super 20tb hdd (for now)

Describe the bug

• Follow build instructions for llama support
• Try and load local ai and give it test command
• get error about no llama support
• frustrated

To Reproduce

• Follow build instructions for llama support
• Try and load local ai and give it test command
• get error about no llama support
• frustrated

Expected behavior

build with llama support

Logs

Additional context

I have been informed by an AI that the build is supposed to have another binary that runs the llama backend but when I compile it the binary is missing, giving the failure for the llama support.

The AI suggested it may need to be built embedded llama because llama binary is not included (even though required to build llama support)

I am trying embedded but I have been compiling and recompiling for the past 48 hours without success.

The LLAMA build commands and instructions should have EVERYTHING to build llama support including the separate binary that is not included. This embedded binary llama support instructions should be right beside it, but it should not even be needed if you give the requirements for the install (the llama server binary) and give instructions to install it.

Then you wouldn't even need to mention embedded unless someone really wanted it to simplify the build and installation.

Still unable to get llama support enabled. Documentations need updating and required binary source needs to be included and built and instructions for that should be included.

{"error":{"code":500,"message":"could not load model - all backends returned error: 10 errors occurred:\n\t* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/llama-cpp. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n\t* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/llama-ggml. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n\t* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/llama-cpp. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n\t* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/gpt4all. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n\t* could not load model: rpc error: code = Unknown desc = unable to load model\n\t* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/stablediffusion. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n\t* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/tinydream. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n\t* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/piper. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS\n\n","type":""}}root@AI0:~/localai#

P.S. the docker build fails to connect to the GPU's which is why I am forced to manual compile and install.

[mooleshacat]

Update on separate post to not make OP too long. I have found out your documentation is lacking and the GPU support is terrible.

I have the following desired setup:

1. FirstGPU (3070) - Codellama-13b-instruct

model:
  name: codellama-13b
  parameters:
    model: codellama-13b-instruct.Q4_K_M.gguf
    gpu: 0
    context_size: 4096
    threads: 4

2. Second GPU (3070) - Wizard-Coder-13B

model:
  name: wizard-coder
  parameters:
    model: wizardcoder-13b-v1.1.Q4_K_M.gguf
    gpu: 1
    context_size: 4096
    threads: 4

3. Third GPU (2070) - phi-2 & TinyLlama

model:
  name: phi-2
  parameters:
    model: phi-2.Q4_K_M.gguf
    gpu: 2
    context_size: 2048
    threads: 2

model:
  name: tinyllama
  parameters:
    model: tinyllama-1.1b.Q4_K_M.gguf
    gpu: 2
    context_size: 2048
    threads: 2

Unfortunately I am missing backend: llama and also the GPU can't be configured this way... You have to build entire separate binary (4 hours of compile or more) and run separate binaries that were built targetting the specific GPU.

What is this? Where is the ease of use and installation? Why are we rebuilding entire binary 3 times targetting different GPU instead of putting a yaml configuration ?

Where is the documentation detailing the correct build instructions (ALL of them) for Llama support? What about instructions for multiple GPU like my desired setup above?