[Bug]: CPU memory OOM at save_pretrained for GLM-4.5 despite 1.5TB of RAM

[josephrocca]

⚙️ Your current environment

The output of python collect_env.py

### Environment Information ###
Operating System: `Linux-6.8.0-64-generic-x86_64-with-glibc2.35`
Python Version: `3.11.13 | packaged by conda-forge | (main, Jun  4 2025, 14:48:23) [GCC 13.3.0]`
llm-compressor Version: `0.6.1.dev61+gb88221ba`
compressed-tensors Version: `0.10.3a20250806`
transformers Version: `4.55.0`
torch Version: `2.7.1+cu128`
CUDA Devices: `['NVIDIA H200', 'NVIDIA H200', 'NVIDIA H200', 'NVIDIA H200']`
AMD Devices: `None`

🐛 Describe the bug

I'm attempting to use llm-compressor on GLM-4.5, and have been running into host crashes within model.save_pretrained that seem to be due to CPU memory OOM.

I've tried up to 4xH200 on Runpod in order to get a machine that has 1.5TB of RAM, and it still crashes. I'm running the quant on a 8xH200 machine now which has 3TB of RAM (the largest RAM machines that Runpod offers I think).

It's a bit hard for me to iterate and debug here because the quant takes 15+ hours and it crashes only at the final step.

🛠️ Steps to reproduce

Using this Docker image: pytorch/pytorch:2.7.1-cuda12.8-cudnn9-devel with 4xH200 machine on Runpod, I SSH in and run this script:

export PATH="/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:$PATH"
git clone https://github.com/vllm-project/llm-compressor
cd llm-compressor
git reset --hard b88221ba41c6ca31406e9885031b2c84cdaac442 # pinned to latest commit as of writing for reproducibility
pip install -e .
wget -O "quantize-glm-4.5-ram-oom.py" "https://gist.githubusercontent.com/josephrocca/137da84687f01cc081080366d9f611c2/raw/24833720295d50adae39a84867355149f6aecb05/quantize-glm-4.5-ram-oom.py"
python ./quantize-glm-4.5-ram-oom.py

For convenience, here's that python file:

quantize-glm-4.5-ram-oom.py

from datasets import load_dataset
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

from llmcompressor.modeling import replace_modules_for_calibration
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.transformers import oneshot

model_id = "unsloth/GLM-4.5"
config = AutoConfig.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    config=config,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = replace_modules_for_calibration(model)

# Select calibration dataset.
DATASET_ID = "HuggingFaceH4/ultrachat_200k"
DATASET_SPLIT = "train_sft"

# Select number of samples. 512 samples is a good place to start.
# Increasing the number of samples can improve accuracy.
NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 2048

# Load dataset and preprocess.
ds = load_dataset(DATASET_ID, split=f"{DATASET_SPLIT}[:{NUM_CALIBRATION_SAMPLES}]")
ds = ds.shuffle(seed=42)


def preprocess(example):
    return {
        "text": tokenizer.apply_chat_template(
            example["messages"],
            tokenize=False,
        )
    }


ds = ds.map(preprocess)


# Tokenize inputs.
def tokenize(sample):
    return tokenizer(
        sample["text"],
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
        add_special_tokens=False,
    )


ds = ds.map(tokenize, remove_columns=ds.column_names)

# Configure the quantization algorithm to run.
# Ignore list copied from: https://huggingface.co/zai-org/GLM-4.5-FP8/blob/main/config.json
recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["model.layers.12.input_layernorm","model.layers.48.input_layernorm","model.layers.74.input_layernorm","model.layers.5.self_attn.q_norm","model.layers.11.self_attn.k_proj.bias","model.layers.3.self_attn.k_proj.bias","model.layers.53.self_attn.q_norm","model.layers.52.mlp.gate","model.layers.29.mlp.gate.e_score_correction_bias","model.layers.34.post_attention_layernorm","model.layers.88.post_attention_layernorm","model.layers.17.input_layernorm","model.layers.37.self_attn.k_norm","model.layers.9.mlp.gate.e_score_correction_bias","model.layers.17.self_attn.v_proj.bias","model.layers.82.self_attn.q_norm","model.layers.41.mlp.gate","model.layers.73.input_layernorm","model.layers.72.input_layernorm","model.layers.61.post_attention_layernorm","model.layers.30.input_layernorm","model.layers.80.mlp.gate","model.layers.14.input_layernorm","model.layers.45.self_attn.k_norm","model.layers.55.mlp.gate","model.layers.9.input_layernorm","model.layers.13.input_layernorm","model.layers.47.self_attn.q_proj.bias","model.layers.84.post_attention_layernorm","model.layers.85.self_attn.q_norm","model.layers.53.self_attn.v_proj.bias","model.layers.83.self_attn.v_proj.bias","model.layers.35.self_attn.v_proj.bias","model.layers.76.mlp.gate.e_score_correction_bias","model.layers.36.input_layernorm","model.layers.13.post_attention_layernorm","model.layers.71.self_attn.k_proj.bias","model.layers.84.mlp.gate.e_score_correction_bias","model.layers.46.self_attn.k_norm","model.layers.50.self_attn.q_norm","model.layers.59.self_attn.q_proj.bias","model.layers.5.input_layernorm","model.layers.29.mlp.gate","model.layers.73.self_attn.q_norm","model.layers.27.mlp.gate","model.layers.49.post_attention_layernorm","model.layers.63.self_attn.q_proj.bias","model.layers.7.self_attn.q_proj.bias","model.layers.5.self_attn.k_proj.bias","model.layers.70.self_attn.q_proj.bias","model.layers.70.self_attn.k_norm","model.layers.15.self_attn.q_proj.bias","model.layers.59.self_attn.q_norm","model.layers.89.self_attn.k_proj.bias","model.layers.89.self_attn.q_norm","model.layers.42.mlp.gate.e_score_correction_bias","model.layers.62.self_attn.q_norm","model.layers.18.input_layernorm","model.layers.79.post_attention_layernorm","model.layers.63.self_attn.k_norm","model.layers.58.input_layernorm","model.layers.57.mlp.gate","model.layers.32.self_attn.q_norm","model.layers.34.self_attn.v_proj.bias","model.layers.83.self_attn.k_proj.bias","model.layers.7.input_layernorm","model.layers.72.self_attn.k_norm","model.layers.55.self_attn.v_proj.bias","model.layers.2.self_attn.k_proj.bias","model.layers.23.post_attention_layernorm","model.layers.60.post_attention_layernorm","model.layers.33.self_attn.k_proj.bias","model.layers.85.self_attn.k_proj.bias","model.layers.83.mlp.gate.e_score_correction_bias","model.layers.20.self_attn.k_proj.bias","model.layers.11.mlp.gate","model.layers.82.input_layernorm","model.layers.83.self_attn.q_norm","model.layers.9.post_attention_layernorm","model.layers.55.mlp.gate.e_score_correction_bias","model.layers.21.mlp.gate","model.layers.41.mlp.gate.e_score_correction_bias","model.layers.10.mlp.gate.e_score_correction_bias","model.layers.49.mlp.gate","model.layers.56.self_attn.k_norm","model.layers.86.self_attn.q_norm","model.layers.89.input_layernorm","model.layers.16.input_layernorm","model.layers.25.mlp.gate.e_score_correction_bias","model.layers.8.self_attn.k_norm","model.layers.72.post_attention_layernorm","model.layers.92.self_attn.q_norm","model.layers.73.post_attention_layernorm","model.layers.71.mlp.gate.e_score_correction_bias","model.layers.42.input_layernorm","model.layers.25.self_attn.k_norm","model.layers.50.self_attn.q_proj.bias","model.layers.13.self_attn.k_norm","model.layers.65.mlp.gate.e_score_correction_bias","model.layers.9.self_attn.k_norm","model.layers.3.input_layernorm","model.layers.90.self_attn.q_proj.bias","model.layers.35.self_attn.q_norm","model.layers.74.mlp.gate","model.layers.21.self_attn.k_norm","model.layers.92.mlp.gate","model.layers.15.mlp.gate.e_score_correction_bias","model.layers.61.self_attn.k_proj.bias","model.layers.63.self_attn.q_norm","model.layers.73.self_attn.q_proj.bias","model.layers.18.self_attn.q_norm","model.layers.45.mlp.gate","model.layers.52.self_attn.v_proj.bias","model.layers.22.post_attention_layernorm","model.layers.85.self_attn.q_proj.bias","model.layers.3.self_attn.k_norm","model.layers.54.self_attn.v_proj.bias","model.layers.79.self_attn.v_proj.bias","model.layers.31.self_attn.q_proj.bias","model.layers.2.input_layernorm","model.layers.57.self_attn.q_norm","model.layers.65.self_attn.k_norm","model.layers.60.input_layernorm","model.layers.70.self_attn.v_proj.bias","model.layers.87.post_attention_layernorm","model.layers.83.post_attention_layernorm","model.layers.51.self_attn.k_proj.bias","model.layers.23.self_attn.k_norm","model.layers.49.self_attn.k_norm","model.layers.76.self_attn.k_proj.bias","model.layers.7.self_attn.q_norm","model.layers.54.mlp.gate","model.layers.63.post_attention_layernorm","model.layers.27.self_attn.k_proj.bias","model.layers.81.input_layernorm","model.layers.66.post_attention_layernorm","model.layers.84.self_attn.q_proj.bias","model.layers.36.self_attn.k_proj.bias","model.layers.30.self_attn.v_proj.bias","model.layers.48.self_attn.k_norm","model.layers.62.mlp.gate","model.layers.8.mlp.gate","model.layers.11.mlp.gate.e_score_correction_bias","model.layers.28.mlp.gate","model.layers.30.mlp.gate.e_score_correction_bias","model.layers.43.input_layernorm","model.layers.3.self_attn.q_proj.bias","model.layers.88.input_layernorm","model.layers.5.post_attention_layernorm","model.layers.35.self_attn.k_norm","model.layers.56.post_attention_layernorm","model.layers.30.mlp.gate","model.layers.67.self_attn.q_proj.bias","model.layers.58.post_attention_layernorm","model.layers.80.post_attention_layernorm","model.layers.88.self_attn.v_proj.bias","model.layers.90.self_attn.v_proj.bias","model.layers.2.self_attn.k_norm","model.layers.67.mlp.gate.e_score_correction_bias","model.layers.31.input_layernorm","model.layers.26.post_attention_layernorm","model.layers.81.self_attn.k_proj.bias","model.layers.51.self_attn.q_norm","model.layers.7.self_attn.k_proj.bias","model.layers.76.self_attn.q_proj.bias","model.layers.1.self_attn.k_norm","model.layers.34.mlp.gate.e_score_correction_bias","model.layers.80.self_attn.k_proj.bias","model.layers.0.self_attn.k_norm","model.layers.44.self_attn.q_norm","model.layers.29.input_layernorm","model.layers.77.self_attn.v_proj.bias","model.layers.73.mlp.gate","model.layers.31.self_attn.v_proj.bias","model.layers.46.self_attn.q_proj.bias","model.layers.62.self_attn.k_norm","model.layers.87.self_attn.k_proj.bias","model.layers.58.self_attn.k_norm","model.layers.66.mlp.gate.e_score_correction_bias","model.layers.32.mlp.gate.e_score_correction_bias","model.layers.73.self_attn.k_proj.bias","model.layers.85.mlp.gate.e_score_correction_bias","model.layers.23.mlp.gate","model.layers.72.self_attn.q_proj.bias","model.layers.25.mlp.gate","model.layers.43.mlp.gate.e_score_correction_bias","model.layers.50.mlp.gate.e_score_correction_bias","model.layers.6.mlp.gate.e_score_correction_bias","model.layers.14.mlp.gate","model.layers.50.self_attn.v_proj.bias","model.layers.33.self_attn.k_norm","model.layers.92.self_attn.q_proj.bias","model.layers.6.self_attn.k_proj.bias","model.layers.43.mlp.gate","model.layers.68.self_attn.q_proj.bias","model.layers.33.self_attn.v_proj.bias","model.layers.41.self_attn.k_norm","model.layers.32.self_attn.k_norm","model.layers.28.post_attention_layernorm","model.layers.61.self_attn.v_proj.bias","model.layers.16.post_attention_layernorm","model.layers.48.mlp.gate.e_score_correction_bias","model.layers.4.self_attn.v_proj.bias","model.layers.88.self_attn.k_norm","model.layers.68.self_attn.k_norm","model.layers.77.self_attn.q_proj.bias","model.layers.8.self_attn.v_proj.bias","model.layers.20.mlp.gate.e_score_correction_bias","model.layers.86.self_attn.k_norm","model.layers.36.self_attn.v_proj.bias","model.layers.71.input_layernorm","model.layers.72.mlp.gate","model.layers.24.self_attn.k_norm","model.layers.20.self_attn.q_norm","model.layers.62.post_attention_layernorm","model.layers.3.self_attn.v_proj.bias","model.layers.25.input_layernorm","model.layers.20.self_attn.v_proj.bias","model.layers.64.self_attn.v_proj.bias","model.layers.19.self_attn.k_proj.bias","model.layers.63.mlp.gate.e_score_correction_bias","model.layers.92.embed_tokens","model.layers.62.self_attn.q_proj.bias","model.layers.69.self_attn.q_norm","model.layers.9.mlp.gate","model.layers.62.mlp.gate.e_score_correction_bias","model.layers.35.self_attn.k_proj.bias","model.layers.1.self_attn.q_proj.bias","model.layers.40.self_attn.q_norm","model.layers.26.input_layernorm","model.layers.50.self_attn.k_proj.bias","model.layers.39.input_layernorm","model.layers.28.self_attn.k_proj.bias","model.layers.39.self_attn.q_proj.bias","model.layers.5.mlp.gate","model.layers.56.input_layernorm","model.layers.60.self_attn.k_norm","model.layers.76.self_attn.q_norm","lm_head","model.layers.69.self_attn.k_norm","model.layers.23.self_attn.q_proj.bias","model.layers.58.self_attn.q_proj.bias","model.norm","model.layers.53.self_attn.k_proj.bias","model.layers.73.self_attn.v_proj.bias","model.layers.33.self_attn.q_proj.bias","model.layers.64.input_layernorm","model.layers.15.mlp.gate","model.layers.55.self_attn.q_proj.bias","model.layers.55.self_attn.k_norm","model.layers.27.mlp.gate.e_score_correction_bias","model.layers.68.input_layernorm","model.layers.30.self_attn.q_norm","model.layers.82.post_attention_layernorm","model.layers.29.self_attn.q_proj.bias","model.layers.79.self_attn.k_proj.bias","model.layers.16.self_attn.k_proj.bias","model.layers.71.self_attn.k_norm","model.layers.16.self_attn.k_norm","model.layers.75.self_attn.k_proj.bias","model.layers.52.self_attn.q_norm","model.layers.87.self_attn.v_proj.bias","model.layers.23.self_attn.k_proj.bias","model.layers.24.post_attention_layernorm","model.layers.40.self_attn.q_proj.bias","model.layers.46.mlp.gate.e_score_correction_bias","model.layers.54.input_layernorm","model.layers.75.post_attention_layernorm","model.layers.72.self_attn.k_proj.bias","model.layers.82.self_attn.q_proj.bias","model.layers.75.mlp.gate.e_score_correction_bias","model.layers.27.self_attn.q_norm","model.layers.39.self_attn.q_norm","model.layers.45.self_attn.k_proj.bias","model.layers.90.self_attn.k_proj.bias","model.layers.65.self_attn.q_proj.bias","model.layers.5.self_attn.k_norm","model.layers.89.self_attn.q_proj.bias","model.layers.10.self_attn.k_proj.bias","model.layers.86.self_attn.v_proj.bias","model.layers.89.mlp.gate.e_score_correction_bias","model.layers.42.self_attn.k_norm","model.layers.57.self_attn.k_proj.bias","model.layers.68.post_attention_layernorm","model.layers.1.self_attn.v_proj.bias","model.layers.38.self_attn.k_proj.bias","model.layers.75.self_attn.v_proj.bias","model.layers.44.self_attn.q_proj.bias","model.layers.8.mlp.gate.e_score_correction_bias","model.layers.55.input_layernorm","model.layers.7.self_attn.v_proj.bias","model.layers.51.self_attn.k_norm","model.layers.19.mlp.gate","model.layers.12.self_attn.q_norm","model.layers.78.self_attn.k_norm","model.layers.61.self_attn.q_norm","model.layers.18.mlp.gate","model.layers.8.self_attn.q_norm","model.layers.7.self_attn.k_norm","model.layers.84.self_attn.q_norm","model.layers.44.self_attn.k_proj.bias","model.layers.38.self_attn.q_proj.bias","model.layers.37.mlp.gate.e_score_correction_bias","model.layers.34.mlp.gate","model.layers.92.eh_proj","model.layers.75.self_attn.q_proj.bias","model.layers.61.mlp.gate.e_score_correction_bias","model.layers.33.mlp.gate.e_score_correction_bias","model.layers.81.mlp.gate.e_score_correction_bias","model.layers.47.self_attn.k_proj.bias","model.layers.72.self_attn.q_norm","model.layers.62.self_attn.k_proj.bias","model.layers.33.self_attn.q_norm","model.layers.24.mlp.gate","model.layers.51.input_layernorm","model.layers.42.self_attn.k_proj.bias","model.layers.43.self_attn.k_norm","model.layers.46.self_attn.k_proj.bias","model.layers.66.input_layernorm","model.layers.31.mlp.gate.e_score_correction_bias","model.layers.77.post_attention_layernorm","model.layers.52.self_attn.k_norm","model.layers.4.mlp.gate","model.layers.51.post_attention_layernorm","model.layers.15.self_attn.k_norm","model.layers.60.self_attn.k_proj.bias","model.layers.6.self_attn.q_norm","model.layers.85.mlp.gate","model.layers.66.self_attn.k_norm","model.layers.22.self_attn.v_proj.bias","model.layers.36.self_attn.k_norm","model.layers.8.input_layernorm","model.layers.19.mlp.gate.e_score_correction_bias","model.layers.25.self_attn.k_proj.bias","model.layers.31.self_attn.q_norm","model.layers.16.mlp.gate.e_score_correction_bias","model.layers.57.self_attn.v_proj.bias","model.layers.35.mlp.gate","model.layers.75.input_layernorm","model.layers.2.self_attn.v_proj.bias","model.layers.9.self_attn.v_proj.bias","model.layers.80.self_attn.q_proj.bias","model.layers.19.self_attn.v_proj.bias","model.layers.4.post_attention_layernorm","model.layers.65.input_layernorm","model.layers.82.mlp.gate.e_score_correction_bias","model.layers.69.post_attention_layernorm","model.layers.21.self_attn.q_norm","model.layers.18.post_attention_layernorm","model.layers.81.self_attn.q_proj.bias","model.layers.77.input_layernorm","model.layers.17.self_attn.q_norm","model.layers.80.input_layernorm","model.layers.13.mlp.gate.e_score_correction_bias","model.layers.0.post_attention_layernorm","model.layers.19.post_attention_layernorm","model.layers.91.self_attn.v_proj.bias","model.layers.77.mlp.gate","model.layers.11.self_attn.q_norm","model.layers.92.hnorm","model.layers.11.self_attn.q_proj.bias","model.layers.47.self_attn.q_norm","model.layers.42.self_attn.v_proj.bias","model.layers.90.input_layernorm","model.layers.78.input_layernorm","model.layers.11.self_attn.v_proj.bias","model.layers.80.self_attn.q_norm","model.layers.83.input_layernorm","model.layers.43.self_attn.q_norm","model.layers.91.post_attention_layernorm","model.layers.50.mlp.gate","model.layers.48.self_attn.k_proj.bias","model.layers.70.mlp.gate.e_score_correction_bias","model.layers.14.self_attn.q_proj.bias","model.layers.31.self_attn.k_proj.bias","model.layers.47.self_attn.v_proj.bias","model.layers.12.self_attn.k_proj.bias","model.layers.12.mlp.gate","model.layers.34.self_attn.q_norm","model.layers.62.self_attn.v_proj.bias","model.layers.26.mlp.gate.e_score_correction_bias","model.layers.45.mlp.gate.e_score_correction_bias","model.layers.77.self_attn.q_norm","model.layers.47.self_attn.k_norm","model.layers.53.post_attention_layernorm","model.layers.7.post_attention_layernorm","model.layers.90.post_attention_layernorm","model.layers.40.mlp.gate.e_score_correction_bias","model.layers.63.input_layernorm","model.layers.52.mlp.gate.e_score_correction_bias","model.layers.92.input_layernorm","model.layers.44.self_attn.v_proj.bias","model.layers.46.mlp.gate","model.layers.18.self_attn.k_proj.bias","model.layers.62.input_layernorm","model.layers.24.self_attn.v_proj.bias","model.layers.40.post_attention_layernorm","model.layers.8.post_attention_layernorm","model.layers.82.mlp.gate","model.layers.17.post_attention_layernorm","model.layers.45.self_attn.v_proj.bias","model.layers.84.self_attn.k_proj.bias","model.layers.15.self_attn.k_proj.bias","model.layers.0.self_attn.q_proj.bias","model.layers.92.shared_head.norm","model.layers.51.self_attn.q_proj.bias","model.layers.22.mlp.gate","model.layers.4.self_attn.k_norm","model.layers.84.input_layernorm","model.layers.30.self_attn.k_proj.bias","model.layers.12.self_attn.v_proj.bias","model.layers.37.input_layernorm","model.layers.37.self_attn.q_norm","model.layers.45.self_attn.q_norm","model.layers.72.self_attn.v_proj.bias","model.layers.64.self_attn.k_norm","model.layers.43.self_attn.k_proj.bias","model.layers.38.mlp.gate.e_score_correction_bias","model.layers.79.input_layernorm","model.layers.48.mlp.gate","model.layers.52.self_attn.q_proj.bias","model.layers.46.self_attn.v_proj.bias","model.layers.81.mlp.gate","model.layers.71.self_attn.q_proj.bias","model.layers.50.self_attn.k_norm","model.layers.18.self_attn.v_proj.bias","model.layers.53.input_layernorm","model.layers.64.self_attn.q_proj.bias","model.layers.12.self_attn.k_norm","model.embed_tokens","model.layers.78.self_attn.k_proj.bias","model.layers.40.self_attn.k_norm","model.layers.11.input_layernorm","model.layers.22.input_layernorm","model.layers.39.mlp.gate","model.layers.30.self_attn.k_norm","model.layers.71.mlp.gate","model.layers.71.post_attention_layernorm","model.layers.78.self_attn.q_norm","model.layers.87.mlp.gate","model.layers.14.self_attn.q_norm","model.layers.42.post_attention_layernorm","model.layers.3.post_attention_layernorm","model.layers.92.self_attn.k_proj.bias","model.layers.3.mlp.gate","model.layers.89.mlp.gate","model.layers.86.self_attn.k_proj.bias","model.layers.87.self_attn.q_proj.bias","model.layers.63.self_attn.k_proj.bias","model.layers.87.input_layernorm","model.layers.10.self_attn.q_proj.bias","model.layers.81.post_attention_layernorm","model.layers.39.self_attn.v_proj.bias","model.layers.43.post_attention_layernorm","model.layers.76.mlp.gate","model.layers.28.self_attn.v_proj.bias","model.layers.57.self_attn.q_proj.bias","model.layers.21.mlp.gate.e_score_correction_bias","model.layers.75.self_attn.q_norm","model.layers.58.self_attn.k_proj.bias","model.layers.52.input_layernorm","model.layers.0.self_attn.k_proj.bias","model.layers.78.mlp.gate.e_score_correction_bias","model.layers.85.self_attn.k_norm","model.layers.42.self_attn.q_norm","model.layers.25.self_attn.v_proj.bias","model.layers.36.post_attention_layernorm","model.layers.4.self_attn.q_norm","model.layers.16.self_attn.q_norm","model.layers.41.self_attn.k_proj.bias","model.layers.5.self_attn.q_proj.bias","model.layers.28.self_attn.q_norm","model.layers.41.self_attn.q_proj.bias","model.layers.68.mlp.gate.e_score_correction_bias","model.layers.26.self_attn.k_proj.bias","model.layers.86.input_layernorm","model.layers.35.mlp.gate.e_score_correction_bias","model.layers.69.input_layernorm","model.layers.77.mlp.gate.e_score_correction_bias","model.layers.10.self_attn.q_norm","model.layers.32.post_attention_layernorm","model.layers.54.self_attn.q_norm","model.layers.45.post_attention_layernorm","model.layers.45.input_layernorm","model.layers.15.post_attention_layernorm","model.layers.64.post_attention_layernorm","model.layers.15.self_attn.v_proj.bias","model.layers.24.self_attn.k_proj.bias","model.layers.64.self_attn.k_proj.bias","model.layers.14.post_attention_layernorm","model.layers.40.mlp.gate","model.layers.50.input_layernorm","model.layers.21.self_attn.v_proj.bias","model.layers.66.self_attn.q_proj.bias","model.layers.78.mlp.gate","model.layers.2.self_attn.q_proj.bias","model.layers.12.self_attn.q_proj.bias","model.layers.7.mlp.gate","model.layers.35.post_attention_layernorm","model.layers.52.post_attention_layernorm","model.layers.30.post_attention_layernorm","model.layers.59.input_layernorm","model.layers.86.post_attention_layernorm","model.layers.0.self_attn.v_proj.bias","model.layers.24.mlp.gate.e_score_correction_bias","model.layers.34.self_attn.k_norm","model.layers.4.self_attn.k_proj.bias","model.layers.74.self_attn.q_proj.bias","model.layers.78.self_attn.v_proj.bias","model.layers.49.self_attn.q_proj.bias","model.layers.53.mlp.gate.e_score_correction_bias","model.layers.60.self_attn.v_proj.bias","model.layers.59.mlp.gate","model.layers.61.mlp.gate","model.layers.26.self_attn.q_norm","model.layers.1.input_layernorm","model.layers.33.mlp.gate","model.layers.91.mlp.gate","model.layers.26.self_attn.q_proj.bias","model.layers.83.mlp.gate","model.layers.12.mlp.gate.e_score_correction_bias","model.layers.36.self_attn.q_norm","model.layers.91.mlp.gate.e_score_correction_bias","model.layers.25.self_attn.q_norm","model.layers.91.self_attn.k_norm","model.layers.20.self_attn.k_norm","model.layers.8.self_attn.k_proj.bias","model.layers.29.self_attn.v_proj.bias","model.layers.82.self_attn.k_proj.bias","model.layers.89.post_attention_layernorm","model.layers.29.self_attn.k_norm","model.layers.88.self_attn.q_norm","model.layers.48.self_attn.q_proj.bias","model.layers.22.self_attn.k_norm","model.layers.27.self_attn.q_proj.bias","model.layers.23.mlp.gate.e_score_correction_bias","model.layers.1.post_attention_layernorm","model.layers.29.post_attention_layernorm","model.layers.58.mlp.gate","model.layers.27.input_layernorm","model.layers.66.self_attn.k_proj.bias","model.layers.87.mlp.gate.e_score_correction_bias","model.layers.22.self_attn.q_norm","model.layers.4.mlp.gate.e_score_correction_bias","model.layers.51.mlp.gate","model.layers.90.mlp.gate.e_score_correction_bias","model.layers.3.self_attn.q_norm","model.layers.33.input_layernorm","model.layers.91.self_attn.k_proj.bias","model.layers.8.self_attn.q_proj.bias","model.layers.12.post_attention_layernorm","model.layers.3.mlp.gate.e_score_correction_bias","model.layers.69.mlp.gate.e_score_correction_bias","model.layers.17.self_attn.k_proj.bias","model.layers.82.self_attn.k_norm","model.layers.10.post_attention_layernorm","model.layers.84.mlp.gate","model.layers.6.input_layernorm","model.layers.70.mlp.gate","model.layers.33.post_attention_layernorm","model.layers.31.self_attn.k_norm","model.layers.88.self_attn.q_proj.bias","model.layers.80.self_attn.k_norm","model.layers.36.mlp.gate.e_score_correction_bias","model.layers.56.self_attn.v_proj.bias","model.layers.1.self_attn.k_proj.bias","model.layers.44.self_attn.k_norm","model.layers.60.mlp.gate.e_score_correction_bias","model.layers.32.self_attn.k_proj.bias","model.layers.69.self_attn.k_proj.bias","model.layers.4.input_layernorm","model.layers.15.self_attn.q_norm","model.layers.79.self_attn.k_norm","model.layers.4.self_attn.q_proj.bias","model.layers.51.mlp.gate.e_score_correction_bias","model.layers.57.self_attn.k_norm","model.layers.90.self_attn.k_norm","model.layers.27.self_attn.k_norm","model.layers.50.post_attention_layernorm","model.layers.21.self_attn.q_proj.bias","model.layers.10.input_layernorm","model.layers.64.self_attn.q_norm","model.layers.67.self_attn.k_proj.bias","model.layers.85.self_attn.v_proj.bias","model.layers.14.mlp.gate.e_score_correction_bias","model.layers.32.mlp.gate","model.layers.67.self_attn.v_proj.bias","model.layers.26.self_attn.k_norm","model.layers.19.self_attn.q_norm","model.layers.22.mlp.gate.e_score_correction_bias","model.layers.68.self_attn.q_norm","model.layers.86.mlp.gate.e_score_correction_bias","model.layers.20.post_attention_layernorm","model.layers.21.post_attention_layernorm","model.layers.49.self_attn.v_proj.bias","model.layers.38.mlp.gate","model.layers.82.self_attn.v_proj.bias","model.layers.42.mlp.gate","model.layers.21.input_layernorm","model.layers.22.self_attn.k_proj.bias","model.layers.54.self_attn.q_proj.bias","model.layers.76.post_attention_layernorm","model.layers.67.input_layernorm","model.layers.65.self_attn.v_proj.bias","model.layers.67.self_attn.q_norm","model.layers.79.self_attn.q_norm","model.layers.47.input_layernorm","model.layers.79.mlp.gate.e_score_correction_bias","model.layers.76.self_attn.k_norm","model.layers.60.mlp.gate","model.layers.20.self_attn.q_proj.bias","model.layers.11.post_attention_layernorm","model.layers.24.self_attn.q_norm","model.layers.53.self_attn.k_norm","model.layers.53.self_attn.q_proj.bias","model.layers.75.self_attn.k_norm","model.layers.28.self_attn.q_proj.bias","model.layers.65.self_attn.q_norm","model.layers.39.self_attn.k_proj.bias","model.layers.85.input_layernorm","model.layers.17.self_attn.q_proj.bias","model.layers.65.post_attention_layernorm","model.layers.68.mlp.gate","model.layers.13.self_attn.q_norm","model.layers.81.self_attn.v_proj.bias","model.layers.58.self_attn.q_norm","model.layers.34.input_layernorm","model.layers.25.self_attn.q_proj.bias","model.layers.77.self_attn.k_proj.bias","model.layers.39.mlp.gate.e_score_correction_bias","model.layers.0.input_layernorm","model.layers.76.self_attn.v_proj.bias","model.layers.6.self_attn.k_norm","model.layers.74.self_attn.k_proj.bias","model.layers.92.self_attn.k_norm","model.layers.37.self_attn.k_proj.bias","model.layers.18.self_attn.k_norm","model.layers.48.self_attn.q_norm","model.layers.84.self_attn.v_proj.bias","model.layers.47.mlp.gate.e_score_correction_bias","model.layers.49.self_attn.q_norm","model.layers.51.self_attn.v_proj.bias","model.layers.61.input_layernorm","model.layers.28.self_attn.k_norm","model.layers.32.self_attn.v_proj.bias","model.layers.55.self_attn.q_norm","model.layers.55.self_attn.k_proj.bias","model.layers.47.post_attention_layernorm","model.layers.69.self_attn.q_proj.bias","model.layers.28.input_layernorm","model.layers.40.self_attn.v_proj.bias","model.layers.19.input_layernorm","model.layers.83.self_attn.q_proj.bias","model.layers.66.self_attn.v_proj.bias","model.layers.59.mlp.gate.e_score_correction_bias","model.layers.23.self_attn.v_proj.bias","model.layers.0.self_attn.q_norm","model.layers.5.mlp.gate.e_score_correction_bias","model.layers.37.self_attn.v_proj.bias","model.layers.2.post_attention_layernorm","model.layers.5.self_attn.v_proj.bias","model.layers.34.self_attn.k_proj.bias","model.layers.38.self_attn.v_proj.bias","model.layers.81.self_attn.k_norm","model.layers.32.input_layernorm","model.layers.10.self_attn.k_norm","model.layers.54.self_attn.k_norm","model.layers.70.input_layernorm","model.layers.41.self_attn.v_proj.bias","model.layers.74.post_attention_layernorm","model.layers.35.input_layernorm","model.layers.7.mlp.gate.e_score_correction_bias","model.layers.6.post_attention_layernorm","model.layers.86.mlp.gate","model.layers.20.mlp.gate","model.layers.31.mlp.gate","model.layers.2.self_attn.q_norm","model.layers.23.self_attn.q_norm","model.layers.6.self_attn.q_proj.bias","model.layers.83.self_attn.k_norm","model.layers.38.input_layernorm","model.layers.60.self_attn.q_norm","model.layers.91.self_attn.q_norm","model.layers.44.input_layernorm","model.layers.67.post_attention_layernorm","model.layers.27.self_attn.v_proj.bias","model.layers.15.input_layernorm","model.layers.26.self_attn.v_proj.bias","model.layers.44.post_attention_layernorm","model.layers.46.self_attn.q_norm","model.layers.58.self_attn.v_proj.bias","model.layers.67.self_attn.k_norm","model.layers.38.self_attn.k_norm","model.layers.56.mlp.gate.e_score_correction_bias","model.layers.73.self_attn.k_norm","model.layers.80.mlp.gate.e_score_correction_bias","model.layers.37.self_attn.q_proj.bias","model.layers.17.mlp.gate","model.layers.19.self_attn.k_norm","model.layers.53.mlp.gate","model.layers.21.self_attn.k_proj.bias","model.layers.56.self_attn.q_norm","model.layers.59.self_attn.v_proj.bias","model.layers.64.mlp.gate.e_score_correction_bias","model.layers.86.self_attn.q_proj.bias","model.layers.46.input_layernorm","model.layers.6.mlp.gate","model.layers.40.self_attn.k_proj.bias","model.layers.60.self_attn.q_proj.bias","model.layers.6.self_attn.v_proj.bias","model.layers.47.mlp.gate","model.layers.32.self_attn.q_proj.bias","model.layers.17.mlp.gate.e_score_correction_bias","model.layers.29.self_attn.k_proj.bias","model.layers.44.mlp.gate.e_score_correction_bias","model.layers.57.mlp.gate.e_score_correction_bias","model.layers.87.self_attn.q_norm","model.layers.71.self_attn.q_norm","model.layers.52.self_attn.k_proj.bias","model.layers.88.mlp.gate.e_score_correction_bias","model.layers.68.self_attn.k_proj.bias","model.layers.16.self_attn.q_proj.bias","model.layers.48.self_attn.v_proj.bias","model.layers.74.self_attn.q_norm","model.layers.34.self_attn.q_proj.bias","model.layers.1.self_attn.q_norm","model.layers.63.mlp.gate","model.layers.92.enorm","model.layers.14.self_attn.v_proj.bias","model.layers.54.post_attention_layernorm","model.layers.41.self_attn.q_norm","model.layers.37.post_attention_layernorm","model.layers.61.self_attn.q_proj.bias","model.layers.91.input_layernorm","model.layers.11.self_attn.k_norm","model.layers.43.self_attn.q_proj.bias","model.layers.18.mlp.gate.e_score_correction_bias","model.layers.59.post_attention_layernorm","model.layers.9.self_attn.q_norm","model.layers.74.self_attn.k_norm","model.layers.43.self_attn.v_proj.bias","model.layers.49.self_attn.k_proj.bias","model.layers.56.self_attn.k_proj.bias","model.layers.14.self_attn.k_proj.bias","model.layers.65.mlp.gate","model.layers.56.self_attn.q_proj.bias","model.layers.24.self_attn.q_proj.bias","model.layers.71.self_attn.v_proj.bias","model.layers.54.mlp.gate.e_score_correction_bias","model.layers.39.post_attention_layernorm","model.layers.91.self_attn.q_proj.bias","model.layers.92.mlp.gate.e_score_correction_bias","model.layers.46.post_attention_layernorm","model.layers.92.self_attn.v_proj.bias","model.layers.63.self_attn.v_proj.bias","model.layers.25.post_attention_layernorm","model.layers.45.self_attn.q_proj.bias","model.layers.38.self_attn.q_norm","model.layers.68.self_attn.v_proj.bias","model.layers.69.mlp.gate","model.layers.84.self_attn.k_norm","model.layers.19.self_attn.q_proj.bias","model.layers.38.post_attention_layernorm","model.layers.44.mlp.gate","model.layers.70.post_attention_layernorm","model.layers.58.mlp.gate.e_score_correction_bias","model.layers.28.mlp.gate.e_score_correction_bias","model.layers.48.post_attention_layernorm","model.layers.36.self_attn.q_proj.bias","model.layers.66.self_attn.q_norm","model.layers.70.self_attn.k_proj.bias","model.layers.57.input_layernorm","model.layers.42.self_attn.q_proj.bias","model.layers.26.mlp.gate","model.layers.74.self_attn.v_proj.bias","model.layers.59.self_attn.k_proj.bias","model.layers.88.mlp.gate","model.layers.14.self_attn.k_norm","model.layers.88.self_attn.k_proj.bias","model.layers.35.self_attn.q_proj.bias","model.layers.65.self_attn.k_proj.bias","model.layers.18.self_attn.q_proj.bias","model.layers.30.self_attn.q_proj.bias","model.layers.29.self_attn.q_norm","model.layers.78.post_attention_layernorm","model.layers.79.mlp.gate","model.layers.27.post_attention_layernorm","model.layers.80.self_attn.v_proj.bias","model.layers.16.mlp.gate","model.layers.10.self_attn.v_proj.bias","model.layers.13.self_attn.v_proj.bias","model.layers.70.self_attn.q_norm","model.layers.73.mlp.gate.e_score_correction_bias","model.layers.41.post_attention_layernorm","model.layers.39.self_attn.k_norm","model.layers.10.mlp.gate","model.layers.36.mlp.gate","model.layers.92.post_attention_layernorm","model.layers.64.mlp.gate","model.layers.40.input_layernorm","model.layers.77.self_attn.k_norm","model.layers.31.post_attention_layernorm","model.layers.56.mlp.gate","model.layers.16.self_attn.v_proj.bias","model.layers.9.self_attn.q_proj.bias","model.layers.13.mlp.gate","model.layers.17.self_attn.k_norm","model.layers.22.self_attn.q_proj.bias","model.layers.49.mlp.gate.e_score_correction_bias","model.layers.9.self_attn.k_proj.bias","model.layers.37.mlp.gate","model.layers.72.mlp.gate.e_score_correction_bias","model.layers.90.self_attn.q_norm","model.layers.79.self_attn.q_proj.bias","model.layers.75.mlp.gate","model.layers.74.mlp.gate.e_score_correction_bias","model.layers.89.self_attn.v_proj.bias","model.layers.59.self_attn.k_norm","model.layers.76.input_layernorm","model.layers.66.mlp.gate","model.layers.55.post_attention_layernorm","model.layers.24.input_layernorm","model.layers.57.post_attention_layernorm","model.layers.23.input_layernorm","model.layers.67.mlp.gate","model.layers.13.self_attn.k_proj.bias","model.layers.69.self_attn.v_proj.bias","model.layers.85.post_attention_layernorm","model.layers.13.self_attn.q_proj.bias","model.layers.87.self_attn.k_norm","model.layers.89.self_attn.k_norm","model.layers.54.self_attn.k_proj.bias","model.layers.49.input_layernorm","model.layers.78.self_attn.q_proj.bias","model.layers.81.self_attn.q_norm","model.layers.41.input_layernorm","model.layers.90.mlp.gate","model.layers.61.self_attn.k_norm","model.layers.20.input_layernorm"],
)

# Apply algorithms.
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

# Save to disk compressed.
model.save_pretrained("/workspace/quantized-W4A16-G128", save_compressed=True)
tokenizer.save_pretrained("/workspace/quantized-W4A16-G128")

Thanks!

[kylesayrs]

Hi @josephrocca,

Sorry to hear about the memory failure. In previous work vllm-project/compressed-tensors#301, I found that CPU memory usage does not increase over time. I agree that I would expect a 358B model to require a little more than 700 Gigbytes of CPU memory. Indeed, I've been able to quantize larger models like deepseekv3 on machines with about 1.3T of CPU memory.

Your script and usage looks correct to me.

Do you have the last log statement printed before the process ran out of memory? Have you confirmed that the process was killed because of a memory issue (and not for any other reason?).

Format	Model compression
Quantized

[josephrocca]

Hi @kylesayrs, thanks for your reply. Yes I did notice some other github issues posted this, and the PR you linked. I should have added them as related links/context in my original post.

The final logs of my last 4xH200 run were:

2025-08-09T15:47:24.661130+0000 | compress | METRIC - GPU 2 | usage: 0.49% | total memory: 150 GB
2025-08-09T15:47:24.661167+0000 | compress | METRIC - GPU 3 | usage: 0.49% | total memory: 150 GB
2025-08-09T15:47:24.661238+0000 | compress | METRIC - Compressed module size: 15.91296 MB
2025-08-09T15:47:24.661389+0000 | compress_modules | INFO - Quantizing model.layers.91.mlp.experts.104.up_proj using 428 samples
2025-08-09T15:47:25.963622+0000 | compress | METRIC - time 1.30s
2025-08-09T15:47:25.965281+0000 | compress | METRIC - error 136.73
2025-08-09T15:47:25.965626+0000 | compress | METRIC - GPU 0 | usage: 32.60% | total memory: 150 GB
2025-08-09T15:47:25.965681+0000 | compress | METRIC - GPU 1 | usage: 0.49% | total memory: 150 GB
2025-08-09T15:47:25.965719+0000 | compress | METRIC - GPU 2 | usage: 0.49% | total memory: 150 GB
2025-08-09T15:47:25.965754+0000 | compress | METRIC - GPU 3 | usage: 0.49% | total memory: 150 GB
2025-08-09T15:47:25.965818+0000 | compress | METRIC - Compressed module size: 15.91296 MB
2025-08-09T15:47:25.965963+0000 | compress_modules | INFO - Quantizing model.layers.91.mlp.experts.104.down_proj using 428 samples
2025-08-09T15:47:26.415472+0000 | compress | METRIC - time 0.45s
2025-08-09T15:47:26.416167+0000 | compress | METRIC - error 96.25
2025-08-09T15:47:26.416314+0000 | compress | METRIC - GPU 0 | usage: 32.60% | total memory: 150 GB
2025-08-09T15:47:26.416377+0000 | compress | METRIC - GPU 1 | usage: 0.49% | total memory: 150 GB
2025-08-09T15:47:26.416422+0000 | compress | METRIC - GPU 2 | usage: 0.49% | total memory: 150 GB
2025-08-09T15:47:26.416457+0000 | compress | METRIC - GPU 3 | usage: 0.49% | total memory: 150 GB
2025-08-09T15:47:26.416520+0000 | compress | METRIC - Compressed module size: 15.91296 MB
(92/93): Propagating: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 512/512 [00:19<00:00, 26.18it/s]
(93/93): Calibrating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 512/512 [00:02<00:00, 217.10it/s]
(93/93): Propagating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 512/512 [00:01<00:00, 293.47it/s]
2025-08-09T15:48:00.341459+0000 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
2025-08-09T15:48:04.718948+0000 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
2025-08-09T15:48:04.748838+0000 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
Compressing model: 48490it [1:54:02,  6.06it/s]client_loop: send disconnect: Broken pipe

Again, I should have included them in my original post, sorry.

Have you confirmed that the process was killed because of a memory issue (and not for any other reason?).

I'm pretty sure it is - the machine becomes completely unresponsive (I cannot SSH in), and Runpod's system/host logs in their web UI show repeated messages saying something like system unhealthy: oom error.

[josephrocca]

3TB of RAM did indeed solve the issue:

• https://huggingface.co/rocca/GLM-4.5-compressed-tensors-w4a16-g128-v1

If this is considered acceptable, or if there's already a tracking issue for improving memory efficiency, I'll close this issue.

[koush]

I ran into this issue and simply enabled an extremely large swap file on nvme. I only have 128GB of memory assigned on this host and it now seems to continue much further (the run is still in progress). From my understanding, after a layer is quantized, the weights are offloaded back to cpu/system memory and never read or written again until the final quantized model is saved to disk anyways.

Enabling swap allows those cpu device tensors to get swapped to disk. There is no noticeable performance loss. I'll update once my run completes: qwen 3 coder 480b (awq on 96gb vram, 128gb ram, 1tb swap).

Update:
Qwen 3 Coder 480b finished up in around 3 hours, but the final sample generation failed because the model was too large for a single gpu:

llm-compressor/examples/awq/qwen3_coder_moe_example.py

Line 69 in bcad892

# Confirm generations of the quantized model look sane.

It did complete successfully. The sample scripts should probably save the model before running sample generation, because the generation crashed and nothing was saved.

[brian-dellabetta]

Thanks @koush for the info, since the processed parts of the model are just being cached on CPU before saving, swap should work well. It doesn't need to swap it back onto memory frequently or anything.

We will looking into enhancing this soon, to circumvent the large host memory or large swap requirements.