test_gpu fails on RTX 3090 with CUDA 11.1 #89
Description
Thank you for the warp-transducer, I was successfully using it with previous CUDA versions and on older card (RTX 2080) and with previous Ubuntu 18.04. I am currently trying to change the setup and move to the following configuration:
HW: RTX 3090Ti
OS: Ubuntu 20.04
CUDA: 11.1
The first change I needed to make was to remove the compute_30, since it is not supported by Cuda 11.1. The code can be compiled and I am able to run test_gpu. However, the option_test in test_gpu fails.
Here is the output:
Running gpu tests
finish small_test 0
grad mismatch in options_test expected grad: -0.186844 calculated score: 0 !(-0.186944 < 0 < -0.186744)
grad mismatch in options_test expected grad: -0.062555 calculated score: 0 !(-0.062655 < 0 < -0.062455)
grad mismatch in options_test expected grad: 0.249399 calculated score: 0 !(0.249299 < 0 < 0.249499)
grad mismatch in options_test expected grad: -0.203377 calculated score: 0 !(-0.203477 < 0 < -0.203277)
grad mismatch in options_test expected grad: 0.202399 calculated score: 0 !(0.202299 < 0 < 0.202499)
grad mismatch in options_test expected grad: 0.000977 calculated score: 0 !(0.000877 < 0 < 0.001077)
grad mismatch in options_test expected grad: -0.141016 calculated score: 0 !(-0.141116 < 0 < -0.140916)
grad mismatch in options_test expected grad: 0.079123 calculated score: 0 !(0.079023 < 0 < 0.079223)
grad mismatch in options_test expected grad: 0.061893 calculated score: 0 !(0.061793 < 0 < 0.061993)
grad mismatch in options_test expected grad: -0.011552 calculated score: 0 !(-0.011652 < 0 < -0.011452)
grad mismatch in options_test expected grad: -0.08128 calculated score: 0 !(-0.08138 < 0 < -0.08118)
grad mismatch in options_test expected grad: 0.092832 calculated score: 0 !(0.092732 < 0 < 0.092932)
grad mismatch in options_test expected grad: -0.154257 calculated score: 0 !(-0.154357 < 0 < -0.154157)
grad mismatch in options_test expected grad: 0.229433 calculated score: 0 !(0.229333 < 0 < 0.229533)
grad mismatch in options_test expected grad: -0.075176 calculated score: 0 !(-0.075276 < 0 < -0.075076)
....
score mismatch in options_test expected score: 4.28065 calculated score: -0 !(4.28055 < -0 < 4.28075)
score mismatch in options_test expected score: 3.93844 calculated score: -0 !(3.93834 < -0 < 3.93854)
finish options_test 0
finish inf_test 0
finished 0
Some or all tests fail
All the grads calculated by option_test are 0.
Is this known issue? How can it be debugged and solved?
Thank you!