Does qlora adapter only support BitsAndBytes quantization model ?

[tarukumar]

What i have observed is when I try to deploy the model using qlora_adapter_name_or_path for qlora adapter it fails to deploy with the error mentioned in the line https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py#L899-L911. The question is to deploy qlora adpater should i use --lora-modules or adapter-cache parameter for qlora adapter? What is the best approach here ?

[jeejeelee]

See: https://github.com/vllm-project/vllm/blob/main/examples/lora_with_quantization_inference.py

[chenqianfzh]

@jeejeelee sure, my pleasure.

QLoRA and 4-bit quantization were originally introduced together in the paper "QLoRA: Efficient Finetuning of Quantized LLMs" (https://arxiv.org/abs/2305.14314). In practice, most QLoRA models utilize bitsandbytes for 4-bit quantization.

However, as @tarukumar observed from the code, QLoRA adapters are not restricted to supporting bitsandbytes quantization exclusively. We are prepared to modify the code if other quantization models require such functionality.

Hope this helps. :-)

[tarukumar]

@chenqianfzh That would be good if we can expand that. Thank you!

[tarukumar]

@chenqianfzh should i open the issue for this support ?

[chenqianfzh]

@chenqianfzh should i open the issue for this support ?

Do you see many such use cases? I would add the feature if it is a popular demand.

[tarukumar]

It would be definitely useful may be we can add this as a part of bytedance-iaas#2

[chenqianfzh]

@tarukumar Feel free to open an issue at https://github.com/bd-iaas-us/vllm/issues and assign it to me.

In this issue, please let us know why existing Quantization + LoRA solution in vLLM does not suffice, as well as some models in need of this feature.

Thanks!