KeyError: 'transformer.h.10.attn.c_attn.SCB' to run jais-chat-30b-8bit

[aalinazar]

I'm trying to run jais-chat-30b-8bit on vllm.
But I got this error:

File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/jais.py", line 366, in load_weights
    param = params_dict[name]
KeyError: 'transformer.h.10.attn.c_attn.SCB'

I got the error after running this command line
vllm serve "brainiac-origin/jais-chat-30b-8bit"

Version of the package:

vllm==0.5.5+cu118
transformers==4.44.2

Anyone can help to solve the issue?
Thank you very much for your help.

[jeejeelee]

plz append --quantization bitsandbytes --load-format bitsandbytes

[aalinazar]

Thanks for your quick reply. I really appreciate it.

I tried that:
vllm serve "brainiac-origin/jais-chat-30b-8bit" --quantization bitsandbytes --load-format bitsandbytes

and I got error:

  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 912, in _load_weights
    raise AttributeError(
AttributeError: Model JAISLMHeadModel does not support BitsAndBytes quantization yet.
ERROR 02-13 13:24:16 api_server.py:186] RPCServer process died before responding to readiness probe

Then I tried to upgrade vllm to version vllm == 0.7.2+cu118

I still got similar error:

  File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 1067, in _load_weights
    raise AttributeError(
AttributeError: Model JAISLMHeadModel does not support BitsAndBytes quantization yet. No 'packed_modules_mapping' found.

Would you like to guide me what to do next?