trainable_activation_sparsity_allmodules Error

[YantaiYang-05]

When I run trainable_activation_sparsity_allmodules, it throws an error. _, s2, _ = torch.svd(weight_matrix2)
RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling cusolverDnSgesvdj(handle, jobz, econ, m, n, A, lda, S, U, ldu, V, ldv, work, lwork, info, params). If you keep seeing this error, you may use torch.backends.cuda.preferred_linalg_library() to try linear algebra operators with other supported backends. See https://pytorch.org/docs/stable/backends.html#torch.backends.cuda.preferred_linalg_library
/pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion invalid multinomial distribution (sum of probabilities <= 0) failed.

[victor7246]

Please share the details for reproducibilty.

[YantaiYang-05]

Thanks for your response. I simply cloned the repository, set up the environment according to the README.md, and then ran this script, but it threw the error above. I didn’t modify any other code:CUDA_VISIBLE_DEVICES=1 python experiments/trainable_activation_sparsity_allmodules.py \
--log DEBUG
--use_gpu
--model_name /root/autodl-tmp/model/llama-2-7b-hf
--num_episodes 15
--learning-rate-action 0.000001
--sparsity_level 0.15
--ppl-eval-dataset wikitext2
--finetune-dataset wikitext2
--finetune-train-nsamples 8000
--finetune-train-seqlen 1024
--finetune-train-batch-size 3
--lora-alpha 10
--lora-r 32
--lora-dropout 0.05
--lora-target-option attn_head_and_mlp
--eval-steps 16
--save-steps 16
--epochs 1
--model_save_path "models/"
--sparsity_technique bernoulli

[YantaiYang-05]

I have run the code with different sparsity levels multiple times, and the loss tends to become NaN at Episode 2/3. I have pinpointed the issue to line 1039 in trainable_activation_sparsity_allmodules.py, where o = torch.nan_to_num(o).to(o.device) is executed, and after printing, it shows all zeros.

[codetalker7]

Hi @YantaiYang-05! Thanks for filing the issue; we're currently in the process of re-structuring the code a bit, and this will be done in a few days. We'll get back to this issue once that has been completed. Thank you!