LIM/ZD Score Computation Implementation?

[ubergarm]

Greetings Razvan,

Thank you and your colleagues for the paper Layer-wise Quantization: A pragmatic and Effictive Method for Quantizing LLMs Beyond Integer Bit-levels.

I have two questions:

1

I'm curious if you still planned on releasing any example implementations of computing Layer Input Modification (LIM) Score and Z-score Distribution (ZD) on LLMs such as the presented dense model Llama-2-13B? Also wondering if there was any more recent exploration around newer MoE models such as DeepSeek-R1/-V3-0324?

Bonus if the code works on existing q8_0 GGUF quants haha...

No pressure if you've already moved on, I have some interest in evaluating these methods given it is becoming easier to specify per layer quantization schemes e.g. ik_llama.cpp's new llama-quantize --custom-q feature.

2

Our first score, named layer input modification (LIM), is based on how much a layer changes its input representations into the output ones.

Just confirming that I understand correctly that both the LIM and ZD scores are calculated taking all tensors of a given layer into account (e.g. attn_(v|q|k|output) and ffn_(up|down|gate)) and not just individual tensors?

Thanks!

[RazvanDu]

Hey, sorry for the late reply!

1. Unfortunately we have not explored those areas yet but I'd be glad to help you if you need any help and are planning to explore them!

As for the code implementation, I'm not allowed to share the full code because of ServiceNow's policies, but we've just been accepted at ACL 2025 Findings so we might be allowed to release it soon. The LIM implementation is similar to the one from ShortGPT and you can find an unofficial implementation at https://github.com/sramshetty/ShortGPT while for the Z-Score I can provide a snippet of the code that calculates the score:

def count_values(tensor):

# Step 1: Compute the mean and standard deviation
mean = tensor.mean()
std = tensor.std()

# Step 2: Compute z-scores
z_scores = (tensor - mean) / std

# Step 3: Count elements with z-score > 1
count = torch.sum(z_scores > 1).item()
return count

all_layers = []
idx = 0

for layer in model.model.layers:

tensor = layer.self_attn.q_proj.weight

values = [count_values(layer.self_attn.q_proj.weight), count_values(layer.self_attn.k_proj.weight), count_values(layer.self_attn.v_proj.weight), count_values(layer.self_attn.o_proj.weight),
          count_values(layer.mlp.gate_proj.weight), count_values(layer.mlp.up_proj.weight), count_values(layer.mlp.down_proj.weight)]

average = sum(values) / len(values)

all_layers.append((average, idx))
idx += 1

2. Yes! We take all of the tensors and then average out the score for a final per layer score.

Let me know if you have any more questions, and sorry again for the long delay!

[ubergarm]

Thanks much for the thoughtful reply, that helps me understand better the details!

There are now two similar implementations available for GGUF format LLMs:

1. ik_llama.cpp fork has implemented llama-imatrix --layer-similarity to print out the cosine similarity scores across each layer. Anecdotally the 1st layer often tends to be most important by the widest margin as reported by this implementation.
2. @EAddario's llama.cpp implementation of imatrix statistics just added in per layer calculations as well.

Just dropping those links here in case other adventurous quant optimizers are curious.

Congrats on ACL 2025 and thanks again for sharing what you can! Cheers!

[RazvanDu]

Oh that's great, I'm really glad to see that the paper actually ended up implemented in practical scenarios even though we couldn't release our code publicly!

Thanks a lot for the links and let me know if you end up having any other questions. Cheers!

[EAddario]

Hi @RazvanDu, echoing @ubergarm sentiment, also congrats on your paper being included in ACL 2025 and thanks for sharing!

Assuming I understood correctly, yours and ShortGPT's approach is to compute the individual tensors scores and then average to get the layer's. My question is, shouldn't the weighted average (i.e. values * #elements / sum(#elements)) be used instead?

I ask because that's how I implemented it in llama-imatrix --show-statistics but happy to change if wrong.

[RazvanDu]

Hey @EAddario,

Your assumption is correct, we haven't experimented with a weighted approach but intuitively it makes sense!

I think it would be ideal to try out your approach as well as simple average and see which one behaves better experimentally, we didn't think about that when working on the paper so I cannot really give you any insights unfortunately.

Thanks so much for the kind words, really appreciate it!