RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)

[acerhp]

使用的环境是由作者提供的docker镜像
使用的显卡是 Tesla P100-PCIE 16GB
在运行./scripts/text2image.sh --debug报错
报错代码如下：
`Generate Samples
WARNING: No training data specified
using world size: 1 and model-parallel size: 1

using dynamic loss scaling
initializing model parallel with size 1
initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234
padded vocab (size: 58219) with 21 dummy tokens (new size: 58240)
prepare tokenizer done
building CogView2 model ...
number of parameters on model parallel rank 0: 3928849920
current device: 1
Load model file pretrained/cogview/cogview-base/142000/mp_rank_00_model_states.pt
Working on No. 0 on 0...
show raw text: 一只可爱的小猫。
Traceback (most recent call last):
File "generate_samples.py", line 329, in
main()
File "generate_samples.py", line 326, in main
generate_images_continually(model, args)
File "generate_samples.py", line 221, in generate_images_continually
generate_images_once(model, args, raw_text, seq, num=args.batch_size, output_path=output_path)
File "generate_samples.py", line 166, in generate_images_once
output_tokens_list.append(filling_sequence(model, seq.clone(), args))
File "/root/cogview/generation/sampling.py", line 128, in filling_sequence
logits, *mems = model(tokens, position_ids, attention_mask, txt_indices_bool, img_indices_bool, is_sparse=args.is_sparse, *mems)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(input, **kwargs)
File "/root/cogview/fp16/fp16.py", line 65, in forward
return fp16_to_fp32(self.module((fp32_to_fp16(inputs)), **kwargs))
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/cogview/model/gpt2_modeling.py", line 112, in forward
transformer_output = self.transformer(embeddings, position_ids, attention_mask, txt_indices_bool, img_indices_bool, is_sparse, *mems)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/cogview/mpu/sparse_transformer.py", line 604, in forward
hidden_states = layer(*args, mem=mem_i)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/cogview/mpu/sparse_transformer.py", line 322, in forward
attention_output = self.attention(layernorm_output1, ltor_mask, pivot_idx, is_sparse, mem)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/cogview/mpu/sparse_transformer.py", line 166, in forward
output = self.dense(context_layer)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/root/cogview/mpu/layers.py", line 319, in forward
output_parallel = F.linear(input_parallel, self.weight)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 1753, in linear
return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)
`
希望有人能为我解答这个问题，谢谢