BitNet 1.58b models reproduced successfully

[kalomaze]

https://huggingface.co/1bitLLM/bitnet_b1_58-3B

[Dampfinchen]

great results. But why is it so big? 13 GB seems way too much for a 1.58 Bit 3B model.

[netrunnereve]

AFAIK the weights are stored as {-1, 0, 1} but in f32.

[Dampfinchen]

I see. Would it be possible to store it in FP16 or even 4 Bit without a noticeable dip in quality? If not, it would take a lot of disk space for 7B models and up. But if that's the compromise of having 1.58 Bit models I think most people will gladly take it.

[netrunnereve]

Yeah you could store it however you like, even in ternary if you pack it properly. They likely use f32 as all the ML libraries support it and they don't have to write special code for inference. Optimizing that into something that's faster and more compact is our job 😉

[EriIaz]

great results. But why is it so big? 13 GB seems way too much for a 1.58 Bit 3B model.

Isn't it FP32?

[hrstoyanov]

Are there any plans for efficient non-gpu (AVX) implementation as part of llama.cpp?