gpt4-x-alpaca - only bin file - how do I interact with this, using llama.cpp?

[djbritt]

I have found 2 different repos for this, and i only see bin files from 7 months ago.

1- https://huggingface.co/chavinlo/gpt4-x-alpaca/tree/main
2- https://huggingface.co/anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g/tree/main/gpt4-x-alpaca-13b-ggml-q4_1-from-gptq-4bit-128g

They only provide a bin file, and I can't get llama.cpp to open it so I can't interact with this model.

Any help is appreciated.

[ianscrivener]

you need a GGUF v2 file format to run with the latest llama.cpp.

There have been a LOT of LLM model updates in the past 7 months... perhaps there is a better and more recent model in GGUFv2?

TheBloke has many already converted and quantized: https://huggingface.co/TheBloke?search_models=GGUF

[KerfuffleV2]

I have found 2 different repos for this, and i only see bin files from 7 months ago.

Although they're both named .bin, they're actually different types of binary files. The second repo is the old GGML format (like ian said you need to use GGUF now). There actually is a script to convert from GGML to GGUF but generally if you can find/convert a GGUF model that's going to work better.

The first repo is the original full unquantized model in PyTorch format. If you have the time/disk space you can download that and use the convert.py script in this repo to convert it to GGUF.