RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED #11
Description
01/11/2020 12:16:13 Loading data...
Load data from tmspan_cached_roberta_train.pkl.
Load data size 75943.
Load data from tmspan_cached_roberta_dev.pkl.
Load data size 9536.
01/11/2020 12:18:07 Num update steps 23732!
01/11/2020 12:18:07 Build bert model.
01/11/2020 12:18:18 Build Drop model.
gcn iteration_steps=3
01/11/2020 12:18:18 Build optimizer etc...
01/11/2020 12:18:23 At epoch 1
Traceback (most recent call last):
File "./roberta_gcn_cli.py", line 103, in
main()
File "./roberta_gcn_cli.py", line 82, in main
model.update(batch)
File "/home/srmishr1/numnet_plus/tools/model.py", line 47, in update
output_dict = self.mnetwork(**tasks)
File "/home/srmishr1/.conda/envs/numnet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/srmishr1/numnet_plus/tag_mspan_robert_gcn/tag_mspan_roberta_gcn.py", line 232, in forward
sequence_output_list[2] = self._gcn_enc(self._proj_ln(sequence_output_list[2] + gcn_info_vec))
File "/home/srmishr1/.conda/envs/numnet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/srmishr1/numnet_plus/mspan_roberta_gcn/util.py", line 22, in forward
output, _ = self.enc_layer(input)
File "/home/srmishr1/.conda/envs/numnet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/srmishr1/.conda/envs/numnet/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 179, in forward
self.dropout, self.training, self.bidirectional, self.batch_first)
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED