Skip to content

Commit 5d711f1

Browse files
authored
Add fallback for sparsegpt inversion failure (#1578)
## Purpose ## * Add fallback for spgt inversion failure, which may occur randomly --------- Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
1 parent af8f44b commit 5d711f1

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

src/llmcompressor/modifiers/obcq/sgpt_sparsify.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33

44
import torch
55
import transformers
6+
from loguru import logger
67

78
SGPT_PRECISION = torch.float32
89

@@ -108,11 +109,12 @@ def sparsify_weight(
108109
H = torch.linalg.cholesky(H, upper=True)
109110
Hinv = H
110111
except torch._C._LinAlgError:
111-
raise torch._C._LinAlgError(
112+
logger.warning(
112113
"Failed to invert hessian due to numerical instability. Consider "
113114
"increasing SparseGPTModifier.dampening_frac, increasing the number "
114115
"of calibration samples, or shuffling the calibration dataset"
115116
)
117+
Hinv = H = torch.eye(num_columns, dtype=H.dtype, device=H.device)
116118

117119
# sparsity mask
118120
# TODO: consider computing sparsity mask in the same way and place as gptq

0 commit comments

Comments
 (0)