Haystack Install downgrade Pytorch Causes CUDA Incompatibility

[eboraks]

When running Haystack with GPU I am getting the following error. After digging into it, I realize that Haystack is downgrading Pytorch to a version that isn't compatible with my CUDA. Is there away to prevent Haystack from doing it?

NVIDIA GeForce RTX 3060 with CUDA capability sm_86 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70. If you want to use the NVIDIA GeForce RTX 3060 GPU with PyTorch, please check the instructions at

Steps to reproduce Haystack downgrading Pytorch

1. Create a fresh environment with conda
2. Install Pytorch 1.13 (pip3 install torch torchvision torchaudio)
3. Check Torch and Cuda versions

>>> torch.__version__
'1.13.0'
>>> torch.version.cuda
'11.7'

5. Install Haystack (pip3 install 'farm-haystack[docstores-gpu,faiss-gpu]'). The installer using an old version of Torch
   Collecting torch<1.13,>1.9 Using cached torch-1.12.1-cp38-cp38-manylinux1_x86_64.whl (776.3 MB)
6. Checking Torch and Cuda versions

>>> torch.__version__
'1.12.1+cu102'
>>> torch.version.cuda
'10.2'

Is there a way to prevent Haystack from using an old Pytorch?

Stuff that I tried

1.. tried specifying to haystack which Torch wheel to use

pip3 install farm-haystack -f https://download.pytorch.org/whl/cu117/torch-1.13.0%2Bcu117-cp311-cp311-linux_x86_64.whl
But that didn't work, I got the following warning

pip3 install farm-haystack -f https://download.pytorch.org/whl/cu117/torch-1.13.0%2Bcu117-cp311-cp311-linux_x86_64.whl

2. I also tried specifying the following haystack to use the cu117 (Cuda 11.7), but no luck :(

pip3 install farm-haystack -f https://download.pytorch.org/whl/cu117

import torch
torch.version
'1.12.1+cu102'
torch.version.cuda
'10.2'

[eboraks]

I tried specifying to haystack which Torch wheel to use

pip3 install farm-haystack -f https://download.pytorch.org/whl/cu117/torch-1.13.0%2Bcu117-cp311-cp311-linux_x86_64.whl

But that didn't work, I got the following warning

pip3 install farm-haystack -f https://download.pytorch.org/whl/cu117/torch-1.13.0%2Bcu117-cp311-cp311-linux_x86_64.whl

[eboraks]

I also tried specifying the following haystack to use the cu117 (Cuda 11.7), but no luck :(

pip3 install farm-haystack -f https://download.pytorch.org/whl/cu117

>>> import torch
>>> torch.__version__
'1.12.1+cu102'
>>> torch.version.cuda
'10.2'

[vblagoje]

I know what you are talking about. The downgrade happens because we have a strict PyTorch version range specified. So when the new torch version comes out, it uninstalls yours and then installs even a wrong Cuda version (see light-the-torch readme). I'll see if we can somehow use https://github.com/pmeier/light-the-torch For now, install your desired version of torch, and it should most likely work.

[eboraks]

@vblagoje light-the-torch sound like a good solution, but I think a simpler solution is to let the implementer specify the Torch version or not install Pytorch and make it a dependency.

[eboraks]

Here is the solution I come up with installing from the source and configuring the Pytorch version.

Installing FARM and specifying Pytorch version

git clone https://github.com/deepset-ai/FARM.git
cd FARM

Edit requirement.txt to point to the right PyTorch torch>1.12.2,<1.14

pip install -r requirements.txt
pip install --editable .
cd ...

Installing Haystack forces it to use a specific Pytorch version

git clone https://github.com/deepset-ai/haystack.git
cd haystack

Edit pyproject.toml to point to the right PyTorch torch>1.12.2,<1.14

pip install --upgrade pip
pip install -e '.[docstores, docstores-gpu, faiss, faiss-gpu]'

Checking Torch version

>>> import torch
>>> torch.__version__
'1.13.0+cu117'
>>> torch.version.cuda
'11.7'

[vblagoje]

Thanks for this @eboraks We can't at the moment migrate to 1.13 due to pytorch/pytorch#88869, so for the time being, the approach I would take is to use light-the-torch and install 1.12.1 in a fresh virtual environment. Please note that the transformers version we use excludes torch version 1.12.0, so if you have that version installed, it will be uninstalled. Then install farm-haystack. I just tried this, and it worked well in both CPU and GPU machines.

[eboraks]

@vblagoje can you outline the installation steps you performed to go around the issue? BTW, so far my workaround to install FARM and Haystack from the source code works fine.

[vblagoje]

@eboraks sure thing, all I did was in the new virtual/conda env I installed torch and then everything else. Don't use version 1.12.0 as the transformer's requirements will revert it and install another. The exact step for that part was installing light-the-torch and then using ltt install torch==1.12.1 worked as planned to install the proper Cuda variant of that torch version. HTH