unable to find llama-export-lora - has it been renamed?

[markat1]

So I'm trying to convert my safetensors LoRA into gguf.

Right now I have converted safetensors LoRA into a ggml-adapter-model.bin. (using a different repository to make the .bin file than llama.cpp)

Next I need the llama-export-lora, but I can't find it in llama.cpp repository - It did exist, because I managed to get it some time ago, but now I don't have it anymore so that's why I need to redownload it again.

[compilade]

To convert an adapter_model.safetensors to a gguf LoRA adapter, you can use the recently-updated convert_lora_to_gguf.py. (See
#8332)

Then you should be able to use it with the --lora option of llama-cli and llama-server and others.

[markat1]

hi @compilade ,
can you give a full example? can I also merge lora into a base gguf model ?

Let me reference how I'm trying to do it right now:

1. I used convert-lora-ggml.py, on adapter_config.json

2. ran python convert-lora-to-ggml.py . llama

3. then downloaded a f16 of the base model.

4. then I wanted to run llama-export-lora -l ggml-adapter-model.bin -o merged_model.f16.gguf -m <base model gguf> in order to merge lora into a base model (can't because I don't have llama-export-lora)

Thank you for your reply!

[compilade]

can you give a full example?

You need the base model as well as the LoRA adapter directories.

For example, I'm going to convert https://huggingface.co/grimjim/Llama-3-Instruct-abliteration-LoRA-8B, which uses https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct as a base model.

To make this easier to follow, here are the file listings of the model directories:

$ cd models
$ tree Llama-3-Instruct-abliteration-LoRA-8B/
Llama-3-Instruct-abliteration-LoRA-8B/
├── adapter_config.json
├── adapter_model.safetensors
└── README.md

1 directory, 3 files

$ tree Meta-Llama-3-8B-Instruct/
Meta-Llama-3-8B-Instruct/
├── config.json
├── generation_config.json
├── LICENSE
├── model-00001-of-00004.safetensors
├── model-00002-of-00004.safetensors
├── model-00003-of-00004.safetensors
├── model-00004-of-00004.safetensors
├── model.safetensors.index.json
├── README.md
├── special_tokens_map.json
├── tokenizer_config.json
└── tokenizer.json

1 directory, 12 files

$ cd ..

Okay, so assume the current working directory is the llama.cpp repository, and that the model directories are under models/ (this isn't important, but it's clearer for explanations).

To convert Llama-3-Instruct-abliteration-LoRA-8B to a GGUF LoRA adapter, I would use the following:

$ python3 convert_lora_to_gguf.py --base models/Meta-Llama-3-8B-Instruct --outfile models/llama-3-abliterated-f16-lora.gguf --outtype f16 models/Llama-3-Instruct-abliteration-LoRA-8B

Then, to use it (assuming llama-cli is already compiled, and that models/llama-3-8B-instruct-Q4_K_M.gguf is the base model converted to GGUF), I would use:

$ ./build/bin/llama-cli -m models/llama-3-8B-instruct-Q4_K_M.gguf --lora models/llama-3-abliterated-f16-lora.gguf -p "How to make the most irresponsible pizza?"

(obviously it's better if you can use a higher precision base model GGUF, since the LoRA isn't really quantization-aware, but it should still work when quantized)

can I also merge lora into a base gguf model ?

I'm not sure, but #8607 might be extremely relevant for you.

I hope this helps!

[markat1]

btw thanks again for helping me out converting lora adapter to gguf!
Seems like I managed to make that work!

btw I figured out that convert_lora_to_gguf appeared after running make -j on llama.cpp folder.

python3 convert_lora_to_gguf.py --base models/Meta-Llama-3-8B-Instruct --outfile models/llama-3-abliterated-f16-lora.gguf --outtype f16 models/Llama-3-Instruct-abliteration-LoRA-8B
INFO:lora-to-gguf:Loading base model: Meta-Llama-3-8B-Instruct
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:lora-to-gguf:Exporting model...
INFO:hf-to-gguf:blk.0.ffn_down.weight.lora_a, torch.bfloat16 --> F16, shape = {14336, 32}

.....

INFO:hf-to-gguf:Set model quantization version
INFO:gguf.gguf_writer:Writing the following files:
INFO:gguf.gguf_writer:models/llama-3-abliterated-f16-lora.gguf: n_tensors = 448, total_size = 167.8M
Writing: 100%|████████████████████████████████████| 168M/168M [00:00<00:00, 307Mbyte/s]
INFO:lora-to-gguf:Model successfully exported to models/llama-3-abliterated-f16-lora.gguf