I found out that you need something like this to have Pytorch working correctly with your GPU:
pip install torch==1.12.0+cu116 torchvision==0.13.0+cu116 torchaudio==0.12.0 -f https://download.pytorch.org/whl/torch_stable.html
Adapt it to yout Cuda version