Revisiting LLM Evaluation through Mechanism Interpretability: a New Metric and Model Utility Law

---

This repository contains the implementation code and associated data for our newly proposed metric, the Model Utilization Index (MUI). This metric is currently derived using Neuron-Based Interpretation and Sparse Activation Encoder (SAE)-based Interpretation methodologies.

🎉 News

• [2025-04-11] 🎉 Exciting Release! We’ve just published our paper titled "Revisiting LLM Evaluation through Mechanism Interpretability: A New Metric and Model Utility Law". Check out our repository for the corresponding analysis code!

Brief Introduction

The Model Utilization Index (MUI) quantifies the extent to which a model leverages its capabilities to complete tasks. The core concept behind MUI is that evaluating an LLM’s overall ability involves more than just measuring its task performance; it also requires analyzing the effort expended to achieve those outcomes. Our extensive experiments reveal an inverse relationship between MUI and performance, suggesting a common trend observed in popular LLMs. We have termed this phenomenon the "Utility Law."

Prerequisites 🛠️

Code Structure

.
├── README.md
├── data				# Directory for the selected seven tasks
├── figures
├── model				# Directory to store model and SAE checkpoints
├── neuron_and_sae				# Directory for model inference and Interpretation 
│   ├── result				# Directory to store the response results
│   ├── get_performance
│   │   ├── depict_neuron.py				# Script for calculation MUI using neurons
│   │   ├── get_Neuron_result				# Directory to store neuron result
│   │   ├── get_neuron.py				# Code implementing Equations 5 and 13
│   │   ├── get_neuron_grad.py				# Code implementing Equations 14
│   │   ├── get_neuron_sum.py				# Code implementing Equations 15
│   │   ├── get_performance_arc.py
│   │   ├── get_performance_bbh.py
│   │   ├── get_performance_gsm8k.py
│   │   ├── get_performance_humaneval.py
│   │   ├── get_performance_math.py
│   │   ├── get_performance_mbpp.py
│   │   ├── get_performance_mmlu.py
│   │   ├── human_eval_data.py
│   │   ├── human_eval_evaluation.py
│   │   ├── human_eval_execution.py
│   │   └── utils_neuron.py
│   └── get_sae				# Code for SAE-based Interpretation
│       ├── depict_sae.py				# Script for calculation MUI using features
│       ├── get_SAE_result
│       ├── get_sae_gemma.py
│       ├── get_sae_llama3.py
│       ├── lm_saes.py
│       └── utils_neuron.py
└── requirements.txt				

Step 1: Clone the Repository and Create Environment

git clone https://github.com/ALEX-nlp/MUI-Eval
cd MUI-Eval
# create environment, make sure to use python 3.10 or later:
conda create --name mui python=3.10.16 -y
conda activate mui
pip install -r requirements.txt

Step 2: Install Target Tested Models (e.g., Llama-3.1, Qwen-2.5)

# Download the Llama-3.1 model from Hugging Face and store it locally
huggingface-cli download --resume-download meta-llama/Llama-3.1-8B-Instruct --local-dir model/Llama-3.1-8B-Instruct

# Download the Qwen-2.5 model from Hugging Face and store it locally
huggingface-cli download --resume-download Qwen/Qwen2.5-7B-Instruct --local-dir model/Qwen2.5-7B-Instruct

If you have already downloaded the models, you may need to update the corresponding paths in neuron_and_sae/get_performance/utils_neuron.py to ensure they point to the correct locations:

# Ensure this path is correct or update as necessary
MODEL_PATHS = {
    "llama3.1": "../../model/Llama-3.1-8B-Instruct",
    "llama3.1-base": "../../model/Llama-3.1-8B-Instruct",  
    "qwen2.5": "../../model/Qwen2.5-7B-Instruct",
}

Step 3: (Optional): Install SAE from Llama Scopes and Gemma Scopes for SAE-based Interpretation

# Download the Llama3_1-8B-Base-LXM-32x model for SAE interpretation and store it locally
huggingface-cli download --resume-download fnlp/Llama3_1-8B-Base-LXM-32x --local-dir model/SAE/Llama3_1-8B-Base-LXM-32x

# Download the Gemma-Scope 9B model for SAE interpretation and store it locally
huggingface-cli download --resume-download google/gemma-scope-9b-pt-res --local-dir model/SAE/gemma-scope-9b-pt-res

Quick Start 🎮

Get started quickly with Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Gemma-2-9b models. This guide will help you use Neuron-Based Interpretation and SAE-based Interpretation to compute the MUI. Below are the steps for conducting Neuron-Based Interpretation for the Llama-3.1-8B model on the ARC task:

# Navigate to the performance directory
cd neuron_and_sae/get_performance

# Run Neuron-Based Interpretation for Llama-3.1 on the ARC task
CUDA_VISIBLE_DEVICES=0 python get_neuron.py --task arc --model llama3.1 --batch_size 2
python depict_neuron.py --model llama3.1 --task arc

# Run Neuron-Based Interpretation for Qwen-2.5 on the ARC task
CUDA_VISIBLE_DEVICES=0 python get_neuron.py --task arc --model qwen2.5 --batch_size 2
python depict_neuron.py --model qwen2.5 --task arc

For SAE-based interpretation using the Gemma model as a demonstration:

# Navigate to the SAE directory
cd neuron_and_sae/get_sae

# Use multiple GPUs to run SAE interpretation for the Gemma model on the GSM8K task
CUDA_VISIBLE_DEVICES=0,1,2,3,5,6 python get_sae_gemma.py --task gsm8k
python depict_sae.py --model gemma2-base --task gsm8k

This quick start guide assumes you have already installed all necessary dependencies and models as per the earlier setup steps. Make sure the CUDA device IDs specified are available in your setup.

Usage 💡

Model Setup: Download the appropriate model and verify the correct paths in the dictionary MODEL_PATHS located in the file neuron_and_sae/get_performance/utils_neuron.py.

Configure Inference: Adjust the inference template and parameters, which can be modified in the functions def format_tokens and def get_model_output within neuron_and_sae/get_performance/utils_neuron.py. We have pre-configured several model variants, which you can find in the MODEL_PATHS dictionary. Ensure the correct model identifier is included in the MODEL_PATHS .

Run Inference: Execute the command below to retrieve responses for any of the seven supported tasks. Replace {$task} with your specific task such as gsm8k, arc, math, humaneval, mbpp, bbh, or mmlu.

cd neuron_and_sae/get_performance
# Run the performance script for a specified task
# For example: CUDA_VISIBLE_DEVICES=0 python get_performance_mmlu.py --model_name llama3.1
CUDA_VISIBLE_DEVICES=0 python get_performance_{$task}.py --model_name model_name 

Note, if you are using a base model that has not undergone instruction tuning, it is advisable to append -base to the model name (e.g., llama3.1-base). We have incorporated a one-shot inference setting in the inference scripts for these base models.

Neuron-based Interpretation: Modify the inference template in the function def get_formatted_instruction located in the file neuron_and_sae/get_performance/get_neuron.py. Use the command below to get neurons for a specific model on the selected task:

cd neuron_and_sae/get_performance
# Get the neurons for the task 
# For example: CUDA_VISIBLE_DEVICES=0 python get_neuron.py --task arc --model llama3.1 --batch_size 2
CUDA_VISIBLE_DEVICES=0 python get_neuron.py --task {$task} --model model_name 

Here we also include additional definitions for neuron contribution score:

get_neuron_sum.py: Calculates response-level contribution scores using Equations 5 and 13.

get_neuron_grad.py: Implements Equation 14.

The usage is identical to the aforementioned commands.

SAE-based Interpretation: Ensure that the downloaded SAE paths are correctly specified in the script. Check the path settings in neuron_and_sae/get_sae/get_sae_gemma.py at line 162 or in neuron_and_sae/get_sae/get_sae_llama.py at line 44. Follow the steps below to extract features for the selected task using the SAE model:

cd neuron_and_sae/get_sae

# Run the SAE feature extraction script for the specified task
CUDA_VISIBLE_DEVICES=0,1,2,3,5,6 python get_sae_gemma.py --task {$task}
CUDA_VISIBLE_DEVICES=0,1,2,3,5,6 python get_sae_llama3.py --task {$task}

MUI Calculation: Before computing the MUI, ensure that the tested model’s neuron counts per layer are correctly specified in neuron_and_sae/get_performance/depict_neuron.py, checking the NEURON_DICTION, NEURON_NUM_DICTION, and NEURON_LAYER_DICTION dictionaries. Execute the following command to obtain the MUI calculated by get_neuron.py:

python depict_neuron.py --model model_name --task {$task}

To further analyze response-level neuron activation and gradients, adjust the NEURON_FILE variable to include options like score_sum, activate_sum, and grad. These settings control the output from get_neuron_sum.py and get_neuron_grad.py:

# Specify options for neuron analysis files
NEURON_FILE = ["activate", "score_sum", "activate_sum", "score", "grad"]

Modify mode_value to apply different threshold settings for top scores and global top-k selections.

# Configuration options for neuron selection thresholds
mode_value = {
    "topscore": [0.8, 0.85, 0.9, 0.95],  # Thresholds for top scores
    "global_topk": [1, 5, 10, 50, 100, 200, 500],  # Global top-k neuron counts
    "topk": [1, 5, 10, 15, 20, 25, 100, 200],  # top-k neuron counts
}

Similar adjustments can be made for SAE results by configuring the threshold selections in depict_sae.py.

Contact

For any questions or feedback, feel free to contact Yixin Cao and Jiahao Ying.

Citation

If you find this work helpful, please consider citing:

@misc{cao2025revisitingllmevaluationmechanism,
      title={Revisiting LLM Evaluation through Mechanism Interpretability: a New Metric and Model Utility Law}, 
      author={Yixin Cao and Jiahao Ying and Yaoning Wang and Xipeng Qiu and Xuanjing Huang and Yugang Jiang},
      year={2025},
      eprint={2504.07440},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2504.07440}, 
}