How to add local unimplemented gguf models?

[yhyu13]

Hi, I am just getting strating using this project.

In this tutorial

https://localai.io/basics/getting_started/index.html

git clone https://github.com/go-skynet/LocalAI

cd LocalAI

# (optional) Checkout a specific LocalAI tag
# git checkout -b build <TAG>

# copy your models to models/
cp your-model.bin models/

# (optional) Edit the .env file to set things like context size and threads
# vim .env

# start with docker compose
docker compose up -d --pull always
# or you can build the images with:
# docker compose up -d --build

# Now API is accessible at localhost:8080
curl http://localhost:8080/v1/models
# {"object":"list","data":[{"id":"your-model.bin","object":"model"}]}

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "your-model.bin",
     "prompt": "A long time ago in a galaxy far, far away",
     "temperature": 0.7
   }'

It only copy the model into a model dir which later mount to docker. But it doesn't seem to work for unimplemented models.

I was trying to load one of the mode from this repo https://huggingface.co/TheBloke/Xwin-LM-7B-V0.1-GGUF. Which is a fresh new llama2 fine-tuned model just released yesterday.

I suppose all I need it to put a single .gguf file into ./model, but localAI complains about this new model is "unimplemented" in the curl msg returned.

Maybe a stupid question, could any one please point me to a more detailed tutorail on how to add brand new gguf models (which llama.cpp supports, but somehow LocalAI is unimplemented)?

[yhyu13]

I saw several config files that need to setup for each new model？ But I have no idea where to put them. And the procedure in adapting a new gguf model for LocalAI.

Thanks!

@mudler

[lunamidori5]

You put the .yaml and model files in the models folder

[lunamidori5]

https://localai.io/howtos/easy-model-import-downloaded/