Some compatible questions when I use ROCm on AMD GPU. #6370

Looong01 · 2023-01-09T01:17:12Z

Looong01
Jan 9, 2023

I have an AMD GPU. I have installed the PyTorch ROCm version and PYG CPU version.
Can I use both of them to build, train and test my GNN model at the same time?
If it will give GPU errors when I use them?
Will the PyTorch ROCm version and PYG CPU version be compatible?

Answered by jeffdaily

Jan 26, 2023

inside docker container based on image rocm/pytorch:latest

git clone https://github.com/pyg-team/pytorch_geometric
git clone --recursive https://github.com/rusty1s/pytorch_sparse
git clone https://github.com/rusty1s/pytorch_scatter
cd pytorch_sparse
pip install . -vvv |& tee ~/build_sparse.log
cd ../pytorch_scatter
pip install . -vvv |& tee ~/build_scatter.log
cd ../pytorch_geometric
pip install . -vvv |& tee ~/build_geometric.log

This was ubuntu 20, rocm 5.4.0.

View full answer

rusty1s · 2023-01-09T09:14:13Z

rusty1s
Jan 9, 2023
Maintainer

We added ROCm support to torch-scatter and torch-sparse lately, so you should find some success installing them via

pip install --verbose torch-scatter torch-sparse

Let me know if this works.

14 replies

Looong01 Jan 26, 2023
Author

Hopefully @jeffdaily has some insights.

I have solved the previous problems by installing these two packages (https://github.com/ROCmSoftwarePlatform/rocThrust) and (https://github.com/ROCmSoftwarePlatform/hipSPARSE).

However, it gives another error when it's during the compiling process of "pip install torch-scatter torch-sparse torch-geometric":

      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:455:23: note: candidate: ‘c10::RegisterOperators&& c10::RegisterOperators::op(c10::RegisterOperators::Options&&) &&’
        455 |   RegisterOperators&& op(Options&& options) && {
            |                       ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:455:23: note:   candidate expects 1 argument, 2 provided
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:461:22: note: candidate: ‘c10::RegisterOperators& c10::RegisterOperators::op(c10::RegisterOperators::Options&&) &’
        461 |   RegisterOperators& op(Options&& options) & {
            |                      ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:461:22: note:   candidate expects 1 argument, 2 provided
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:471:23: note: candidate: ‘c10::RegisterOperators&& c10::RegisterOperators::op(const string&, c10::RegisterOperators::Options&&) &&’
        471 |   RegisterOperators&& op(const std::string& schemaOrName, Options&& options = RegisterOperators::options()) && {
            |                       ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:471:69: note:   no known conversion for argument 2 from ‘int64_t (*)() noexcept’ {aka ‘long int (*)() noexcept’} to ‘c10::RegisterOperators::Options&&’
        471 |   RegisterOperators&& op(const std::string& schemaOrName, Options&& options = RegisterOperators::options()) && {
            |                                                           ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:476:23: note: candidate: ‘c10::RegisterOperators&& c10::RegisterOperators::op(c10::FunctionSchema, c10::RegisterOperators::Options&&) &&’
        476 |   RegisterOperators&& op(FunctionSchema schema, Options&& options) && {
            |                       ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:476:41: note:   no known conversion for argument 1 from ‘const char [28]’ to ‘c10::FunctionSchema’
        476 |   RegisterOperators&& op(FunctionSchema schema, Options&& options) && {
            |                          ~~~~~~~~~~~~~~~^~~~~~
      csrc/version.cpp:41:13: warning: ‘registry’ defined but not used [-Wunused-variable]
         41 | static auto registry = torch::RegisterOperators().op(
            |             ^~~~~~~~
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for torch-scatter
  Running setup.py clean for torch-scatter

The screenshot:

businessabstraction May 24, 2023

ROCm

Will you be providing ROCm builds in the future? I am building a box specifically for PyG, andf AMD offers 24Gb for half the price of NVidia.

Looong01 May 24, 2023
Author

ROCm

Will you be providing ROCm builds in the future? I am building a box specifically for PyG, andf AMD offers 24Gb for half the price of NVidia.

here, https://github.com/Looong01/pyg-rocm-build

Andy3117006664 Nov 3, 2023

You are my hero

yyou1996 Apr 18, 2025

You are the true hero sir!

jeffdaily · 2023-01-26T16:51:14Z

jeffdaily
Jan 26, 2023

We don't control these packages and how the wheels are prepared and delivered -- there are no ROCm versions of these wheels that can be pip installed. You must build all of the prereqs from source. Be careful with using pip in case it installs a CUDA version of the wheel instead.

The missing thrust header error needs some triage. The file csrc/hip/segment_csr_hip.hip appears to have been hipified, but didn't convert the thrust include to a hipthrust include. I'll need to reproduce these steps to confirm what's going wrong.

4 replies

Looong01 Jan 26, 2023
Author

We don't control these packages and how the wheels are prepared and delivered -- there are no ROCm versions of these wheels that can be pip installed. You must build all of the prereqs from source. Be careful with using pip in case it installs a CUDA version of the wheel instead.

The missing thrust header error needs some triage. The file csrc/hip/segment_csr_hip.hip appears to have been hipified, but didn't convert the thrust include to a hipthrust include. I'll need to reproduce these steps to confirm what's going wrong.

Thank you! I have solved the previous problems by installing these two packages (https://github.com/ROCmSoftwarePlatform/rocThrust) and (https://github.com/ROCmSoftwarePlatform/hipSPARSE).

However, it gives another error when it's during the compiling process of "pip install torch-scatter torch-sparse torch-geometric":

      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:455:23: note: candidate: ‘c10::RegisterOperators&& c10::RegisterOperators::op(c10::RegisterOperators::Options&&) &&’
        455 |   RegisterOperators&& op(Options&& options) && {
            |                       ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:455:23: note:   candidate expects 1 argument, 2 provided
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:461:22: note: candidate: ‘c10::RegisterOperators& c10::RegisterOperators::op(c10::RegisterOperators::Options&&) &’
        461 |   RegisterOperators& op(Options&& options) & {
            |                      ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:461:22: note:   candidate expects 1 argument, 2 provided
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:471:23: note: candidate: ‘c10::RegisterOperators&& c10::RegisterOperators::op(const string&, c10::RegisterOperators::Options&&) &&’
        471 |   RegisterOperators&& op(const std::string& schemaOrName, Options&& options = RegisterOperators::options()) && {
            |                       ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:471:69: note:   no known conversion for argument 2 from ‘int64_t (*)() noexcept’ {aka ‘long int (*)() noexcept’} to ‘c10::RegisterOperators::Options&&’
        471 |   RegisterOperators&& op(const std::string& schemaOrName, Options&& options = RegisterOperators::options()) && {
            |                                                           ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:476:23: note: candidate: ‘c10::RegisterOperators&& c10::RegisterOperators::op(c10::FunctionSchema, c10::RegisterOperators::Options&&) &&’
        476 |   RegisterOperators&& op(FunctionSchema schema, Options&& options) && {
            |                       ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:476:41: note:   no known conversion for argument 1 from ‘const char [28]’ to ‘c10::FunctionSchema’
        476 |   RegisterOperators&& op(FunctionSchema schema, Options&& options) && {
            |                          ~~~~~~~~~~~~~~~^~~~~~
      csrc/version.cpp:41:13: warning: ‘registry’ defined but not used [-Wunused-variable]
         41 | static auto registry = torch::RegisterOperators().op(
            |             ^~~~~~~~
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for torch-scatter
  Running setup.py clean for torch-scatter

The screenshot:

Looong01 Jan 26, 2023
Author

It seems that it can include the "thrust/complex.h" now. But it gives another error during compiling.

Looong01 Jan 26, 2023
Author

And this:

      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:455:23: note:   candidate expects 1 argument, 2 provided
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:461:22: note: candidate: ‘c10::RegisterOperators& c10::RegisterOperators::op(c10::RegisterOperators::Options&&) &’
        461 |   RegisterOperators& op(Options&& options) & {
            |                      ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:461:22: note:   candidate expects 1 argument, 2 provided
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:471:23: note: candidate: ‘c10::RegisterOperators&& c10::RegisterOperators::op(const string&, c10::RegisterOperators::Options&&) &&’
        471 |   RegisterOperators&& op(const std::string& schemaOrName, Options&& options = RegisterOperators::options()) && {
            |                       ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:471:69: note:   no known conversion for argument 2 from ‘int64_t (*)() noexcept’ {aka ‘long int (*)() noexcept’} to ‘c10::RegisterOperators::Options&&’
        471 |   RegisterOperators&& op(const std::string& schemaOrName, Options&& options = RegisterOperators::options()) && {
            |                                                           ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:476:23: note: candidate: ‘c10::RegisterOperators&& c10::RegisterOperators::op(c10::FunctionSchema, c10::RegisterOperators::Options&&) &&’
        476 |   RegisterOperators&& op(FunctionSchema schema, Options&& options) && {
            |                       ^~
      /home/loong/anaconda3/envs/PyTorch/lib/python3.10/site-packages/torch/include/ATen/core/op_registration/op_registration.h:476:41: note:   no known conversion for argument 1 from ‘const char [28]’ to ‘c10::FunctionSchema’
        476 |   RegisterOperators&& op(FunctionSchema schema, Options&& options) && {
            |                          ~~~~~~~~~~~~~~~^~~~~~
      csrc/version.cpp:41:13: warning: ‘registry’ defined but not used [-Wunused-variable]
         41 | static auto registry = torch::RegisterOperators().op(
            |             ^~~~~~~~
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> torch-scatter

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Looong01 Jan 26, 2023
Author

We don't control these packages and how the wheels are prepared and delivered -- there are no ROCm versions of these wheels that can be pip installed. You must build all of the prereqs from source. Be careful with using pip in case it installs a CUDA version of the wheel instead.

The missing thrust header error needs some triage. The file csrc/hip/segment_csr_hip.hip appears to have been hipified, but didn't convert the thrust include to a hipthrust include. I'll need to reproduce these steps to confirm what's going wrong.

My environment is:
OS: Ubuntu 22.04.1 Linux home 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Python: 3.10
ROCm: 5.4.2
PyTorch: 2.0.0.dev20230125+rocm5.3
GPU: 6600xt (I use HSA_OVERRIDE_GFX_VERSION=10.3.0 to override the GFX name.)

jeffdaily · 2023-01-26T20:43:23Z

jeffdaily
Jan 26, 2023

inside docker container based on image rocm/pytorch:latest

git clone https://github.com/pyg-team/pytorch_geometric
git clone --recursive https://github.com/rusty1s/pytorch_sparse
git clone https://github.com/rusty1s/pytorch_scatter
cd pytorch_sparse
pip install . -vvv |& tee ~/build_sparse.log
cd ../pytorch_scatter
pip install . -vvv |& tee ~/build_scatter.log
cd ../pytorch_geometric
pip install . -vvv |& tee ~/build_geometric.log

This was ubuntu 20, rocm 5.4.0.

3 replies

jeffdaily Jan 26, 2023

The sparse build takes a long time. It was single-threaded for me. But I didn't encounter any build errors in any package. I can provide log files if needed. This was building for gfx906 (MI50).

Looong01 Jan 27, 2023
Author

The sparse build takes a long time. It was single-threaded for me. But I didn't encounter any build errors in any package. I can provide log files if needed. This was building for gfx906 (MI50).

Could you help to provide the wheels you have already built? I can try to install .whl directly without compiling. Maybe that will work. Thank you very much!

Looong01 Jan 27, 2023
Author

Thank you, I have already install these packages by using your commands above.
I think it's the problem of the tar packages on PyPi.

If I use pip install torch-scatter torch-sparse torch-geometric, it will returns error. But if I git clone them from github repos, it will be successful.

@rusty1s Hey, bro. Maybe some mistakes in the tar packages on PyPi. The codes of Github repos have no problem.
And I have compiled the wheels of ROCm version of pytorch_cluster pytorch_geometric pytorch_scatter pytorch_sparse pytorch_spline_conv. Do you think it's good idea to release them on your website (https://data.pyg.org/whl/) ?

Looong01 · 2023-01-27T13:16:11Z

Looong01
Jan 27, 2023
Author

@rusty1s These are the wheels I built for PyTorch 1.13 from py3.7 to py3.10.
https://github.com/Looong01/pyg-rocm-wheels

11 replies

Looong01 Feb 7, 2023
Author

I'm using ubuntu 20.04, ROCm 5.4.0, Python 3.8.13, torch 1.13.0. But I'm facing the below error

OSError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/conda/lib/python3.8/site-packages/torch_scatter/_version_cuda.so)

What operation did you do before this error?

PavanKumarMiriyala Feb 7, 2023

I did

wget https://github.com/Looong01/pyg-rocm-build/releases/download/1/torch-1.13-rocm-5.4.2-py38-linux_x86_64.zip
unzip torch-1.13-rocm-5.4.2-py38-linux_x86_64.zip
pip3 install *.whl

Installation is successful. I'm facing the above error when I run an example gcn code.

Looong01 Feb 7, 2023
Author

I'm using ubuntu 20.04, ROCm 5.4.0, Python 3.8.13, torch 1.13.0. But I'm facing the below error

OSError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/conda/lib/python3.8/site-packages/torch_scatter/_version_cuda.so)

You can make an issue here. https://github.com/Looong01/pyg-rocm-build/issues

Looong01 Feb 7, 2023
Author

I did

wget https://github.com/Looong01/pyg-rocm-build/releases/download/1/torch-1.13-rocm-5.4.2-py38-linux_x86_64.zip

unzip torch-1.13-rocm-5.4.2-py38-linux_x86_64.zip

pip3 install *.whl

Installation is successful. I'm facing the above error when I run an example gcn code.

I think the libc version in your os is too old.

Solution:

sudo strings /lib/x86_64-linux-gnu/libc.so.6 |grep GLIBC, make sure there are no strings include GLIBC_2.32
If 1 is true, download th newest or required version of glibc (http://ftp.gnu.org/gnu/glibc/)
Take glibc-2.37 as an example.

tar xvf glibc-2.37.tar.gz
cd glibc-2.37
mkdir build
cd ./build
../configure --prefix=<the path you want to install>
make -j<the number of your CPU cores> #if your cpu has 8 cores, then "make -j8"
sudo make install
sudo rm -rf /lib/x86_64-linux-gnu/libc.so.6
sudo ln -s <the path you installed glibc>/lib/libc-2.37.so /lib/x86_64-linux-gnu/libc.so.6

PavanKumarMiriyala Feb 7, 2023

Yes, it is due to old libc version in my OS. I resolved that issue. Thank you so much.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some compatible questions when I use ROCm on AMD GPU. #6370

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 32 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Some compatible questions when I use ROCm on AMD GPU. #6370

Replies: 4 comments · 32 replies

rusty1s Jan 9, 2023 Maintainer

Looong01 Jan 26, 2023 Author

Looong01 May 24, 2023 Author

Looong01 Jan 26, 2023 Author

Looong01 Jan 26, 2023 Author

Looong01 Jan 26, 2023 Author

Looong01 Jan 26, 2023 Author

Looong01 Jan 27, 2023 Author

Looong01 Jan 27, 2023 Author

Looong01 Jan 27, 2023 Author

Looong01 Feb 7, 2023 Author

Looong01 Feb 7, 2023 Author

Looong01 Feb 7, 2023 Author

Replies: 4 comments 32 replies

rusty1s
Jan 9, 2023
Maintainer

Looong01 Jan 26, 2023
Author

Looong01 May 24, 2023
Author

Looong01 Jan 26, 2023
Author

Looong01 Jan 26, 2023
Author

Looong01 Jan 26, 2023
Author

Looong01 Jan 26, 2023
Author

Looong01 Jan 27, 2023
Author

Looong01 Jan 27, 2023
Author

Looong01
Jan 27, 2023
Author

Looong01 Feb 7, 2023
Author

Looong01 Feb 7, 2023
Author

Looong01 Feb 7, 2023
Author