Exception trying to convert TheBloke_Wizard-Vicuna-30B-Uncensored-fp16

[markrmiller]

I get through creating the measurment file fine, but then right away it hits this:

-- Quantizing...
-- Layer: model.layers.0 (Attention)
-- Linear: model.layers.0.self_attn.q_proj -> 0.05:3b_32g/0.95:2b_32g s4, 2.18 bpw
Traceback (most recent call last):
File "/home/user/exllamav2/convert.py", line 300, in
quant(job, save_job, model)
File "/home/user/exllamav2/conversion/quantize.py", line 648, in quant
do_quant(module.q_proj, quantizers["q_proj"], qparams[module.q_proj.key], job)
File "/home/user/exllamav2/conversion/quantize.py", line 480, in do_quant
recons_linear.load(recons_dict)
File "/home/user/exllamav2/exllamav2/linear.py", line 51, in load
self.q_handle = ext.make_q_matrix(w, self.temp_dq)
File "/home/user/exllamav2/exllamav2/ext.py", line 183, in make_q_matrix
return ext_c.make_q_matrix(w["q_weight"],
TypeError: make_q_matrix(): incompatible function arguments. The following argument types are supported:
1. (arg0: torch.Tensor, arg1: torch.Tensor, arg2: torch.Tensor, arg3: torch.Tensor, arg4: torch.Tensor, arg5: torch.Tensor, arg6: torch.Tensor, arg7: torch.Tensor, arg8: torch.Tensor, arg9: torch.Tensor) -> int

Invoked with: tensor([[ 811566591, 543131135, -1281677962, ..., 1820945554,
601242413, 340768092],
[ -924552998, 1223003238, -684363442, ..., -1226673607,
380491495, 1833806393],
[-1362844202, -1354312274, -2045465697, ..., -808748147,
-699863094, 1960528721],
...,
[ -152307997, -1449199898, -151607630, ..., -1427442130,
-1368743938, -1515766243],
[-1696959899, -1431670171, -306480774, ..., -23761478,
-1683300514, 2041489845],
[ 1241343673, -1452430678, 100335270, ..., 1862188703,
-1633060503, 1785458667]], device='cuda:0', dtype=torch.int32), tensor([2736, 686, 6276, ..., 5780, 5922, 5463], device='cuda:0',
dtype=torch.int16), tensor([ 163, 925, 3123, ..., 2416, 2074, 4299], device='cuda:0',
dtype=torch.int16), tensor([[ 1414878309, 1447380294, 1398166869, ..., 1452697190,
1752663656, 1433823077],
[ 1952793717, 1448432983, 1952736342, ..., 1233483623,
1734829673, 1704355957],
[ 1129530485, 1146438981, 1667457861, ..., 912549447,
1195791703, 1433753429],
...,
[ 1448432980, 1433682996, 1415861605, ..., 1718969687,
1733780841, 2022143829],
[ 1700025940, 1415856964, 1146377316, ..., 1751611238,
2020046970, -2004317832],
[ 1682199925, 1147422038, 1414878550, ..., 1735886455,
1751672441, -2004317799]], device='cuda:0', dtype=torch.int32), tensor([0.0007, 0.0005, 0.0007, 0.0006, 0.0006, 0.0004, 0.0003, 0.0003, 0.0003,
0.0005, 0.0002, 0.0006, 0.0005, 0.0005, 0.0006, 0.0005, 0.0005, 0.0005,
0.0005, 0.0005, 0.0005, 0.0005, 0.0004, 0.0005, 0.0005, 0.0005, 0.0005,
0.0004, 0.0005, 0.0004, 0.0005, 0.0005, 0.0004, 0.0004, 0.0004, 0.0004,
0.0005, 0.0005, 0.0004, 0.0004, 0.0004, 0.0006, 0.0004, 0.0005, 0.0004,
0.0004, 0.0004, 0.0005, 0.0005, 0.0004, 0.0004, 0.0004, 0.0004, 0.0005,
0.0004, 0.0004, 0.0004, 0.0005, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004,
0.0004, 0.0004, 0.0004, 0.0005, 0.0004, 0.0004, 0.0005, 0.0005, 0.0004,
0.0004, 0.0005, 0.0004, 0.0004, 0.0004, 0.0004, 0.0005, 0.0004, 0.0004,
0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0005, 0.0004,
0.0004, 0.0004, 0.0005, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004,
0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0005,
0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0005, 0.0004,
0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0005, 0.0004, 0.0004, 0.0004,
0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0003,
0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0005, 0.0004, 0.0004, 0.0004,
0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004,
0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0005, 0.0004, 0.0004, 0.0004,
0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0003, 0.0004, 0.0004,
0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004,
0.0005, 0.0004, 0.0004, 0.0004, 0.0004, 0.0004, 0.0003, 0.0004, 0.0003,
0.0003, 0.0004, 0.0003, 0.0004, 0.0005, 0.0004, 0.0004, 0.0003, 0.0004,
0.0004, 0.0004, 0.0003, 0.0003, 0.0003, 0.0003, 0.0003, 0.0004, 0.0003,
0.0003], device='cuda:0', dtype=torch.float16), tensor([ 3, 0, 3, 3, 3, 6, 3, 9, 3, 12, 3, 15, 3, 18,
3, 21, 3, 24, 3, 27, 3, 30, 2, 33, 2, 35, 2, 37,
2, 39, 2, 41, 2, 43, 2, 45, 2, 47, 2, 49, 2, 51,
2, 53, 2, 55, 2, 57, 2, 59, 2, 61, 2, 63, 2, 65,
2, 67, 2, 69, 2, 71, 2, 73, 2, 75, 2, 77, 2, 79,
2, 81, 2, 83, 2, 85, 2, 87, 2, 89, 2, 91, 2, 93,
2, 95, 2, 97, 2, 99, 2, 101, 2, 103, 2, 105, 2, 107,
2, 109, 2, 111, 2, 113, 2, 115, 2, 117, 2, 119, 2, 121,
2, 123, 2, 125, 2, 127, 2, 129, 2, 131, 2, 133, 2, 135,
2, 137, 2, 139, 2, 141, 2, 143, 2, 145, 2, 147, 2, 149,
2, 151, 2, 153, 2, 155, 2, 157, 2, 159, 2, 161, 2, 163,
2, 165, 2, 167, 2, 169, 2, 171, 2, 173, 2, 175, 2, 177,
2, 179, 2, 181, 2, 183, 2, 185, 2, 187, 2, 189, 2, 191,
2, 193, 2, 195, 2, 197, 2, 199, 2, 201, 2, 203, 2, 205,
2, 207, 2, 209, 2, 211, 2, 213, 2, 215, 2, 217, 2, 219,
2, 221, 2, 223, 2, 225, 2, 227, 2, 229, 2, 231, 2, 233,
2, 235, 2, 237, 2, 239, 2, 241, 2, 243, 2, 245, 2, 247,
2, 249, 2, 251, 2, 253, 2, 255, 2, 257, 2, 259, 2, 261,
2, 263, 2, 265, 2, 267, 2, 269, 2, 271, 2, 273, 2, 275,
2, 277, 2, 279, 2, 281, 2, 283, 2, 285, 2, 287, 2, 289,
2, 291, 2, 293, 2, 295, 2, 297, 2, 299, 2, 301, 2, 303,
2, 305, 2, 307, 2, 309, 2, 311, 2, 313, 2, 315, 2, 317,
2, 319, 2, 321, 2, 323, 2, 325, 2, 327, 2, 329, 2, 331,
2, 333, 2, 335, 2, 337, 2, 339, 2, 341, 2, 343, 2, 345,
2, 347, 2, 349, 2, 351, 2, 353, 2, 355, 2, 357, 2, 359,
2, 361, 2, 363, 2, 365, 2, 367, 2, 369, 2, 371, 2, 373,
2, 375, 2, 377, 2, 379, 2, 381, 2, 383, 2, 385, 2, 387,
2, 389, 2, 391, 2, 393, 2, 395, 2, 397, 2, 399, 2, 401,
2, 403, 2, 405, 2, 407, 2, 409, 2, 411, 2, 413, 2, 415,
2, 417, 2, 419, 2, 421, 2, 423, 2, 425], device='cuda:0',
dtype=torch.int16), tensor([ 0, 32, 0, ..., 2, 207, 1], device='cuda:0',
dtype=torch.int16), tensor(..., device='meta', size=(1, 1)), tensor(..., device='meta', size=(1, 1)), tensor(..., device='meta', size=(1, 1)), tensor([0., 0., 0., ..., 0., 0., 0.], device='cuda:0', dtype=torch.float16)

[turboderp]

I assume you've updated the repo, but you still have the previous version of the package installed?

[markrmiller]

Thanks, it’s actually a fresh checkout from yesterday, but I think I reused a conda environment, I’ll try a new one.