Q4_0 repacking suggestion

[bartowski1182]

Mentioned this over on Reddit awhile back, about enabling repacking for Q4_0 and IQ4_NL

Just wanted to let you know that apparently if you use mmap (a common feature to speed up model loading) it'll actually some the the un-repacked weights in memory, increasing the "on paper" memory consumption by like 50%

In practice this memory SHOULD get cleared out by the system if it's needed, but I have a feeling it sometimes may not. So you may want to check when using a repackable quant type that you disable the mmap to avoid excess RAM usage

[a-ghorbani]

hey, appreciate pointing this out! Yep, mmap is set to true by default. Since repacking feature is set at compile time, it will always be available. but, I assume this only affects Q4_0 and IQ4_NL? So if I only disable mmap for Q4_0 and IQ4_NL ones, that should prevent the issue, right? I'll test it with a couple of settings to see the impact as well.

[bartowski1182]

Yes that would work! Until it's available for other sizes I would just disable it for those :)

[bartowski1182]

though looks like even more quant types are coming :O

ggml-org/llama.cpp#12361

[bartowski1182]

And also here: ggml-org/llama.cpp#11227

Looks like Q4_K is getting huge speed bumps !

[a-ghorbani]

Just synced this version: ggml-org/llama.cpp#12361
I'm not seeing any performance improvement on my Pixel 9 with SVE2 on vs. off.

I'm no expert on this stuff, but I wonder if it's related to topics like this: https://gist.github.com/zingaburga/805669eb891c820bd220418ee3f0d6bd.
For example the fact that SVE implementations on phones has vec length of 128 bits (compared to 256/512 on servers) might mean there's no extra boost on mobile for SVE vs NEON.

I might need to dig deeper into this. just did a quick test, and noticed matmul gets a ~1.5-2x boost on my phone (Pixel 9) with SVE2 over NEON, but dotprod is actually ~0.5x slower.

BTW, I tested llama3.2-1b_q4_k_m