nvrtc compilation failed: Error using GATConv

[gchhablani]

Hi,

I face this error while trying to run forward in a GATConv layer:

Traceback (most recent call last):
  File "custom_train.py", line 76, in <module>
    trainer.train(model, train_data, val_data)
  File "/home/base_trainer.py", line 133, in train
    outputs = model(inputs)
  File "/home/.env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/models/simple_gat.py", line 51, in forward
    out = gatconv_layer(out, edge_index)
  File "/home/.env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/.env/lib/python3.8/site-packages/torch_geometric/nn/conv/gat_conv.py", line 153, in forward
    out = self.propagate(edge_index, x=(x_l, x_r),
  File "/home/.env/lib/python3.8/site-packages/torch_geometric/nn/conv/message_passing.py", line 237, in propagate
    out = self.message(**msg_kwargs)
  File "/home/.env/lib/python3.8/site-packages/torch_geometric/nn/conv/gat_conv.py", line 181, in message
    alpha = softmax(alpha, index, ptr, size_i)
RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1.
  Make sure that libnvrtc-builtins.so.11.1 is installed correctly.
nvrtc compilation failed: 

#define NAN __int_as_float(0x7fffffff)
#define POS_INFINITY __int_as_float(0x7f800000)
#define NEG_INFINITY __int_as_float(0xff800000)


template<typename T>
__device__ T maximum(T a, T b) {
  return isnan(a) ? a : (a > b ? a : b);
}

template<typename T>
__device__ T minimum(T a, T b) {
  return isnan(a) ? a : (a < b ? a : b);
}

extern "C" __global__
void fused_add_div(float* t0, float* t1, float* aten_div, float* aten_add) {
{
  if (512 * blockIdx.x + threadIdx.x<47348 ? 1 : 0) {
    float t1_1 = __ldg(t1 + 512 * blockIdx.x + threadIdx.x);
    aten_add[512 * blockIdx.x + threadIdx.x] = t1_1 + 1.000000016862384e-16f;
    float v = __ldg(t0 + 512 * blockIdx.x + threadIdx.x);
    aten_div[512 * blockIdx.x + threadIdx.x] = v / (t1_1 + 1.000000016862384e-16f);
  }
}
}

Please help me solve this issue. I'm unable to find a solution that works. The code works fine on a CPU.

Python version: 3.8.3
NVCC Version: 11.2

torch==1.8.0+cu111
torch-cluster==1.5.9
torch-geometric==1.6.3
torch-scatter==2.0.6
torch-sparse==0.6.9
torch-spline-conv==1.2.1

Thanks,
Gunjan

[rusty1s]

Might be helpful: chainer/chainer#4813

[rusty1s]

Any chance you can try to install PyTorch via conda?

[gchhablani]

I can try, but I am using Colab, and hence I am using Python Envs to maintain versions locally. I can install using pip on Colab, but not using conda.

[tirtharajdash]

@gchhablani while I was installing PyG, I followed section 3 of https://pytorch-geometric.readthedocs.io/en/latest/notes/installation.html This doesn't mess the Cuda installations. Have we gone through the same route?

[gchhablani]

@tirtharajdash Yes, I did the same thing indirectly. I used these requirements and passed the links to check using the -f argument.

The issue here (atleast on my local machine) is probably that my CUDA toolkit version (11.2) is different form PyTorch's cuda version (11.1).

[gchhablani]

@rusty1s You were right. Installing the correct version of torch (with CUDA 10.1) on Colab worked. Thanks a lot.