Using the a model from huggingface with the server and Docker.

[espoirMur]

Hello

I am running the llama-cpp server using docker and docker-compose. I have my gguf on hugging face and I want to use them with docker. However, my setup is not working.

Here is what I am trying.

services:
  llamacpp-server:
    image: ghcr.io/ggerganov/llama.cpp:server
    environment:
      LLAMA_ARG_HF_REPO: espoir/congo_news_summarizer_qwen_models
      LLAMA_ARG_HF_FILE: Qwen_1.5_8Q.gguf
      LLAMA_ARG_CTX_SIZE: 4096
      LLAMA_ARG_N_PARALLEL: 1
      LLAMA_ARG_ENDPOINT_METRICS: 1
      LLAMA_ARG_PORT: 8000
      LLAMA_ARG_N_PREDICT: 512
    expose:
      - 8000

With that setup, the server is not pulling my model.

I am getting this error:

llamacpp-server-1  | 
llamacpp-server-1  | llama_load_model_from_file: failed to load model
llamacpp-server-1  | llama_init_from_gpt_params: error: failed to load model 'models/7B/ggml-model-f16.gguf'
llamacpp-server-1  |  ERR [              load_model] unable to load model | tid="281473073727328" timestamp=1734634007 model="models/7B/ggml-model-f16.gguf"
llamacpp-server-1  | free(): invalid pointer

Am I running the right version of the docker image?

The equivalent command from the command line works directly without any problem.

../llama.cpp/build/bin/llama-server --hf-repo "espoir/congo_news_summarizer_qwen_models" --hf-file Qwen_1.5_8Q.gguf

[espoirMur]

Found the answer, I was using the wrong image. My local image was old. I removed it and pull the new image and it seems to work.