llama.cpp and lora - forcing 16 bit or a merge is kind of defeating the purpose of a separate lora file ?

[cmp-nct]

" the simultaneous use of LoRAs and GPU acceleration is only supported for f16 models"
Given LORA already forces mmap to be disabled, we have full access on the memory.

Why not during load: dequantize any LORA layer to FP16 -> apply lora -> quantize again
For best quality it would be possible to point to a FP16 model to load the raw layer.

Like a on-the-fly combination, so the GPU kernels will not even know it was a LORA.

Having to merge LORAs kind of defeats the purpose of it, that's just a preprocessed finetune ?:)

[samikrc]

Got hit by the same issue as well. I ran a qlora finetuning of OpenHermes 2.5 (mistral) using axolotl, converted the adapter file to ggml using convert-lora-to-ggml.py, and then tried loading the model through llama.cpp server like so:

llama.cpp$ ./build/bin/server -m ./models/7B/openhermes-2.5-mistral-7b.Q4_K_M.gguf --lora ./models/7B/qlora-out/ggml-adapter-model.bin -t 10 -a "openhermes-2.5" -c 4352 -ngl 99 -n 512 -np 2 --host 0.0.0.0 --port 8585

Got the error:

llama_model_apply_lora_from_file: failed to apply lora adapter: llama_apply_lora_from_file_internal: error: the simultaneous use of LoRAs and GPU acceleration is only supported for f16 models. dest_t->type: 12
llama_init_from_gpt_params: error: failed to apply lora adapter

Do I need to change the convert-lora-to-ggml.py to write out in f16?

[samikrc]

Hmm ... this seem to have got resolved by downloading the latest of llama.cpp, compiling and using it.