Skip to content

Commit 9ac73b7

Browse files
rodburnsrodburns
authored
Adding setup script credit to Michoumichmich (#19)
1 parent 1020d88 commit 9ac73b7

File tree

11 files changed

+979
-0
lines changed

11 files changed

+979
-0
lines changed

‎setup-script/README.md

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# oneAPI on CUDA setup script
2+
3+
This script allows you to build and setup the DPC++ compiler, oneMKL and oneDNN with Nvidia GPUs support.
4+
5+
### Use
6+
7+
1. If needed, set `$DPCPP_HOME` and `$CUDA_ROOT` in the script.
8+
2. Run the script with ```./build.sh```.
9+
10+
Everything will be installed to `$DPCPP_HOME/deploy`.
11+
12+
To build with testing support and run the tests for DPC++, oneMKL, oneTBB, oneDNN and Lapack,
13+
run: ```DPCPP_TESTS=ON ./build.sh```
14+
15+
If you want to build libc++, use `CC=clang-X CXX=clang++-X` with another version of clang, gcc won't compile the libc as
16+
the `asm` syntax is not the same. Using this clang/dpc++ won't work either, there's a bug. Then link
17+
with `-stdlib=libc++`.
18+
19+
### Environment variables
20+
21+
Once everything was built, add the first four exports of the script in your environment or add them in your shell's
22+
config file.
23+
24+
### Using the CUDA Backend
25+
26+
1. Use the following selector:
27+
28+
```C++
29+
class CUDADeviceSelector : public sycl::device_selector {
30+
public:
31+
int operator()(const sycl::device &device) const override {
32+
   return device.get_platform().get_backend() == sycl::backend::cuda ? 1 : -1;
33+
}
34+
};
35+
```
36+
37+
2. Build with `-fsycl -fsycl-targets=nvptx64-nvidia-cuda-sycldevice`
38+
3. For oneMKL, link with: `-lonemkl_blas_cublas -lonemkl`
39+
40+
### Intel's openCL
41+
42+
You can also install Intel's openCL driver (`intel-oneapi-runtime-opencl`) so you can target your CPU
43+
with `-fsycl-targets=spir64_x86_64-unknown-unknown-sycldevice`. For the setup
44+
see [here](https://software.intel.com/content/www/us/en/develop/documentation/installation-guide-for-intel-oneapi-toolkits-linux/top/installation/install-using-package-managers.html)
45+
.
46+
47+
### Dependencies
48+
49+
* Even though the latest version of CUDA seems to work with DPC++, we need to use CUDA 10.2 as oneMKL uses cuBLAS 10.2.
50+
On RHEL: `cuda-10-2 libcublas-devel-10-2`. For the setup
51+
see: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pre-installation-actions.
52+
* `libatomic`
53+
* to be able to target Intel's CPUs with openCL.
54+
* `ninja-build cmake libxml2-devel ...` and other "build essentials" required for building a compiler.
55+
* `libcudnn8 libcudnn8-devel` for oneDNN.
56+
* `gcc-gfortran` or another Fortran compiler for NETLIB Lapack.
57+
* `hwloc-devel` for oneTBB
58+
* `mpfr-devel` when building libcxx
59+
60+
### Caveats
61+
62+
* CLion integration: you need to add `cidr.compiler.clang.fsycl=true` in the registry of CLion [see](https://www.jetbrains.com/help/clion/tuning-the-ide.html?keymap=secondary_macos#configure-platform-properties)
63+
* oneMKL does not support testing when building for both cuBLAS and cuRAND
64+
* for oneMKL with cuda you should use the namespace `oneapi::mkl::blas::column_major::` as cuBLAS is column_major.
65+
66+
### Repositories used
67+
68+
- OpenCL Headers+Loaders: https://github.com/KhronosGroup/OpenCL-Headers.git
69+
, https://github.com/KhronosGroup/OpenCL-ICD-Loader.git
70+
- DPC++ Compiler: https://github.com/intel/llvm.git
71+
- NETLIB Lapack: https://github.com/Reference-LAPACK/lapack.git
72+
- oneTBB: https://github.com/oneapi-src/oneTBB.git
73+
- oneMKL: https://github.com/oneapi-src/oneMKL.git
74+
- oneDNN: https://github.com/oneapi-src/oneDNN.git

‎setup-script/build.sh

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
#!/bin/sh
2+
export DPCPP_HOME=~/sycl_workspace
3+
export CUDA_ROOT=/usr/local/cuda-10.2
4+
export LD_LIBRARY_PATH=$DPCPP_HOME/deploy/lib/:$DPCPP_HOME/deploy/lib64/:$DPCPP_HOME/lapack/install/lib64/:$DPCPP_HOME/OpenCL-ICD-Loader/install/lib64:$CUDA_ROOT/lib:$CUDA_ROOT/lib64:$LD_LIBRARY_PATH
5+
export PATH=$DPCPP_HOME/deploy/bin/:$CUDA_ROOT/bin:$PATH
6+
7+
mkdir -p $DPCPP_HOME
8+
cd $DPCPP_HOME
9+
mkdir -p deploy
10+
11+
#export LD_PRELOAD=/opt/intel/opencl/libOpenCL.so.1
12+
13+
run_test=false
14+
cmake_test="OFF"
15+
16+
if [[ -z "$DPCPP_TESTS" ]]; then
17+
echo "Not testing"
18+
else
19+
echo "testing"
20+
run_test=true
21+
cmake_test="ON"
22+
fi
23+
24+
export CXXFLAGS="${CXXFLAGS} -D_GLIBCXX_USE_CXX11_ABI=1"
25+
26+
# OpenCL headers+ICD
27+
cd $DPCPP_HOME
28+
(if cd OpenCL-Headers; then git pull; else git clone https://github.com/KhronosGroup/OpenCL-Headers.git; fi)
29+
(if cd OpenCL-ICD-Loader; then git pull; else git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader.git; fi)
30+
cd OpenCL-ICD-Loader
31+
mkdir -p build
32+
cd build
33+
cmake \
34+
-DOPENCL_ICD_LOADER_HEADERS_DIR=$DPCPP_HOME/OpenCL-Headers \
35+
-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/OpenCL-ICD-Loader/install \
36+
..
37+
make install -j $(nproc)
38+
39+
#sycl compiler with cuda
40+
source /opt/intel/opencl/env/compiler_rt_vars.sh
41+
cd $DPCPP_HOME
42+
(if cd llvm; then git pull; else git clone https://github.com/intel/llvm.git -b sycl; fi)
43+
cd llvm
44+
python3 ./buildbot/configure.py \
45+
--cuda \
46+
-t release \
47+
--cmake-opt="-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/deploy" \
48+
--cmake-opt="-DCUDA_SDK_ROOT_DIR=$CUDA_ROOT" \
49+
--cmake-opt="-DLLVM_BINUTILS_INCDIR=/usr/local/include" \
50+
--cmake-opt="-DLLVM_ENABLE_PROJECTS=clang;sycl;llvm-spirv;libunwind;opencl;libdevice;xpti;xptifw;libclc;lld;lldb;libc;libcxx;libcxxabi;openmp;clang-tools-extra;compiler-rt" \
51+
--cmake-opt="-DLLVM_BUILD_TESTS=$cmake_test" \
52+
--cmake-opt="-DCMAKE_CXX_STANDARD=17" \
53+
--cmake-opt="-DLLVM_ENABLE_LTO=off" \
54+
--cmake-opt="-DLLVM_ENABLE_LLD=ON" \
55+
--cmake-opt="-DSYCL_ENABLE_WERROR=OFF" \
56+
--cmake-opt="-Wno-dev"
57+
cd build
58+
ninja install -j $(nproc)
59+
if $run_test; then
60+
echo "testing llvm"
61+
ninja check -j $(nproc)
62+
fi
63+
64+
#Lapack Reference
65+
cd $DPCPP_HOME
66+
(if cd lapack; then git pull; else git clone https://github.com/Reference-LAPACK/lapack.git; fi)
67+
cd lapack/
68+
mkdir -p build
69+
cd build/
70+
cmake \
71+
-DBUILD_SHARED_LIBS=ON \
72+
-DCBLAS=ON \
73+
-DLAPACKE=ON \
74+
-DBUILD_TESTING=$cmake_test \
75+
-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/lapack/install \
76+
..
77+
cmake --build . -j $(nproc) --target install
78+
if $run_test; then
79+
cmake --build . -j $(nproc) --target test
80+
fi
81+
82+
#oneTBB
83+
cd $DPCPP_HOME
84+
(if cd oneTBB; then git pull; else git clone https://github.com/oneapi-src/oneTBB.git; fi)
85+
cd oneTBB
86+
mkdir -p build
87+
cd build
88+
cmake -GNinja \
89+
-DCMAKE_CXX_COMPILER=$DPCPP_HOME/deploy/bin/clang++ \
90+
-DCMAKE_BUILD_TYPE=Release \
91+
-DTBB_STRICT=OFF \
92+
-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/deploy/ \
93+
-DTBB_TEST=$cmake_test \
94+
..
95+
ninja install -j $(nproc)
96+
if $run_test; then
97+
ninja test -j $(nproc)
98+
fi
99+
100+
#oneMKL
101+
cd $DPCPP_HOME
102+
(if cd oneMKL; then git pull; else git clone https://github.com/oneapi-src/oneMKL.git; fi)
103+
cd oneMKL
104+
mkdir -p build
105+
cd build
106+
cmake -GNinja \
107+
-DCMAKE_CXX_COMPILER=$DPCPP_HOME/deploy/bin/clang++ \
108+
-DCMAKE_BUILD_TYPE=Release \
109+
-DCMAKE_CXX_STANDARD=17 \
110+
-DTARGET_DOMAINS=blas \
111+
-DENABLE_MKLGPU_BACKEND=OFF \
112+
-DENABLE_CURAND_BACKEND=OFF \
113+
-DENABLE_MKLCPU_BACKEND=OFF \
114+
-DENABLE_CUBLAS_BACKEND=ON \
115+
-DENABLE_NETLIB_BACKEND=ON \
116+
-DREF_BLAS_ROOT=$DPCPP_HOME/lapack/install \
117+
-DNETLIB_ROOT=$DPCPP_HOME/lapack/install \
118+
-DOPENCL_INCLUDE_DIR=$DPCPP_HOME/OpenCL-Headers \
119+
-DCUDA_TOOLKIT_ROOT_DIR=$CUDA_ROOT \
120+
-DSYCL_LIBRARY=$DPCPP_HOME/deploy/lib/libsycl.so \
121+
-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/deploy/ \
122+
-DBUILD_FUNCTIONAL_TESTS=$cmake_test \
123+
..
124+
ninja install -j $(nproc)
125+
if $run_test; then
126+
ninja test -j $(nproc)
127+
fi
128+
129+
#oneDNN
130+
cd $DPCPP_HOME
131+
(if cd oneDNN; then git pull; else git clone https://github.com/oneapi-src/oneDNN.git; fi)
132+
cd oneDNN
133+
mkdir -p build
134+
cd build
135+
cmake -GNinja \
136+
-DCMAKE_C_COMPILER=$DPCPP_HOME/deploy/bin/clang \
137+
-DCMAKE_CXX_COMPILER=$DPCPP_HOME/deploy/bin/clang++ \
138+
-DCMAKE_BUILD_TYPE=Release \
139+
-DCMAKE_CXX_STANDARD=17 \
140+
-DDNNL_INSTALL_MODE=BUNDLE \
141+
-DDNNL_CPU_RUNTIME=DPCPP \
142+
-DDNNL_GPU_RUNTIME=DPCPP \
143+
-DDNNL_GPU_VENDOR=NVIDIA \
144+
-DNNL_BUILD_EXAMPLES=OFF \
145+
-DTBBROOT=$DPCPP_HOME/deploy \
146+
-DCUDA_SDK_ROOT_DIR=$CUDA_ROOT \
147+
-DOPENCLROOT=$DPCPP_HOME/OpenCL-ICD-Loader/install \
148+
-DOpenCL_INCLUDE_DIR=$DPCPP_HOME/OpenCL-Headers \
149+
-DCUBLAS_LIBRARY=$CUDA_ROOT/lib64/libcublas.so \
150+
-DCUBLAS_INCLUDE_DIR=$CUDA_ROOT/include \
151+
-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/deploy \
152+
-DDNNL_BUILD_TESTS=$cmake_test \
153+
..
154+
ninja install -j $(nproc)
155+
if $run_test; then
156+
LD_LIBRARY_PATH=$DPCPP_HOME/deploy/lib ninja test -j $(nproc)
157+
fi

‎setup-script/build_minimal.sh

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
#!/bin/sh
2+
export DPCPP_HOME=~/sycl_workspace
3+
export CUDA_ROOT=/usr/local/cuda-10.2
4+
export LD_LIBRARY_PATH=$DPCPP_HOME/deploy/lib/:$DPCPP_HOME/deploy/lib64/:$DPCPP_HOME/lapack/install/lib64/:$DPCPP_HOME/OpenCL-ICD-Loader/install/lib64:$CUDA_ROOT/lib:$CUDA_ROOT/lib64:$LD_LIBRARY_PATH
5+
export PATH=$DPCPP_HOME/deploy/bin/:$CUDA_ROOT/bin:$PATH
6+
7+
mkdir -p $DPCPP_HOME
8+
cd $DPCPP_HOME
9+
mkdir -p deploy
10+
11+
#export LD_PRELOAD=/opt/intel/opencl/libOpenCL.so.1
12+
13+
run_test=false
14+
cmake_test="OFF"
15+
16+
if [[ -z "$DPCPP_TESTS" ]]; then
17+
echo "Not testing"
18+
else
19+
echo "testing"
20+
run_test=true
21+
cmake_test="ON"
22+
fi
23+
24+
export CXXFLAGS="${CXXFLAGS} -D_GLIBCXX_USE_CXX11_ABI=1"
25+
26+
# OpenCL headers+ICD
27+
cd $DPCPP_HOME
28+
(if cd OpenCL-Headers; then git pull; else git clone https://github.com/KhronosGroup/OpenCL-Headers.git; fi)
29+
(if cd OpenCL-ICD-Loader; then git pull; else git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader.git; fi)
30+
cd OpenCL-ICD-Loader
31+
mkdir -p build
32+
cd build
33+
cmake \
34+
-DOPENCL_ICD_LOADER_HEADERS_DIR=$DPCPP_HOME/OpenCL-Headers \
35+
-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/OpenCL-ICD-Loader/install \
36+
..
37+
make install -j $(nproc)
38+
39+
#sycl compiler with cuda
40+
cd $DPCPP_HOME
41+
(if cd llvm; then git pull; else git clone https://github.com/intel/llvm.git -b sycl; fi)
42+
cd llvm
43+
python3 ./buildbot/configure.py \
44+
--cuda \
45+
-t release \
46+
--cmake-opt="-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/deploy" \
47+
--cmake-opt="-DCUDA_SDK_ROOT_DIR=$CUDA_ROOT" \
48+
--cmake-opt="-DLLVM_BINUTILS_INCDIR=/usr/local/include" \
49+
--cmake-opt="-DLLVM_ENABLE_PROJECTS=clang;sycl;llvm-spirv;opencl;libdevice;xpti;xptifw;libclc;openmp" \
50+
--cmake-opt="-DLLVM_BUILD_TESTS=$cmake_test" \
51+
--cmake-opt="-DCMAKE_CXX_STANDARD=17" \
52+
--cmake-opt="-DLLVM_ENABLE_LTO=off" \
53+
--cmake-opt="-DLLVM_ENABLE_LLD=ON" \
54+
--cmake-opt="-Wno-dev"
55+
cd build
56+
ninja install -j $(nproc)
57+
if $run_test; then
58+
echo "testing llvm"
59+
ninja check -j $(nproc)
60+
fi
61+
62+
#Lapack Reference
63+
cd $DPCPP_HOME
64+
(if cd lapack; then git pull; else git clone https://github.com/Reference-LAPACK/lapack.git; fi)
65+
cd lapack/
66+
mkdir -p build
67+
cd build/
68+
cmake \
69+
-DBUILD_SHARED_LIBS=ON \
70+
-DCBLAS=ON \
71+
-DLAPACKE=ON \
72+
-DBUILD_TESTING=$cmake_test \
73+
-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/lapack/install \
74+
..
75+
cmake --build . -j $(nproc) --target install
76+
if $run_test; then
77+
cmake --build . -j $(nproc) --target test
78+
fi
79+
80+
#oneTBB
81+
cd $DPCPP_HOME
82+
(if cd oneTBB; then git pull; else git clone https://github.com/oneapi-src/oneTBB.git; fi)
83+
cd oneTBB
84+
mkdir -p build
85+
cd build
86+
cmake -GNinja \
87+
-DCMAKE_CXX_COMPILER=$DPCPP_HOME/deploy/bin/clang++ \
88+
-DCMAKE_BUILD_TYPE=Release \
89+
-DTBB_STRICT=OFF \
90+
-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/deploy/ \
91+
-DTBB_TEST=$cmake_test \
92+
..
93+
ninja install -j $(nproc)
94+
if $run_test; then
95+
ninja test -j $(nproc)
96+
fi
97+
98+
#oneMKL
99+
cd $DPCPP_HOME
100+
(if cd oneMKL; then git pull; else git clone https://github.com/oneapi-src/oneMKL.git; fi)
101+
cd oneMKL
102+
mkdir -p build
103+
cd build
104+
cmake -GNinja \
105+
-DCMAKE_CXX_COMPILER=$DPCPP_HOME/deploy/bin/clang++ \
106+
-DCMAKE_BUILD_TYPE=Release \
107+
-DCMAKE_CXX_STANDARD=17 \
108+
-DTARGET_DOMAINS=blas \
109+
-DENABLE_MKLGPU_BACKEND=OFF \
110+
-DENABLE_CURAND_BACKEND=OFF \
111+
-DENABLE_MKLCPU_BACKEND=OFF \
112+
-DENABLE_CUBLAS_BACKEND=ON \
113+
-DENABLE_NETLIB_BACKEND=ON \
114+
-DREF_BLAS_ROOT=$DPCPP_HOME/lapack/install \
115+
-DNETLIB_ROOT=$DPCPP_HOME/lapack/install \
116+
-DOPENCL_INCLUDE_DIR=$DPCPP_HOME/OpenCL-Headers \
117+
-DCUDA_TOOLKIT_ROOT_DIR=$CUDA_ROOT \
118+
-DSYCL_LIBRARY=$DPCPP_HOME/deploy/lib/libsycl.so \
119+
-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/deploy/ \
120+
-DBUILD_FUNCTIONAL_TESTS=$cmake_test \
121+
..
122+
ninja install -j $(nproc)
123+
if $run_test; then
124+
ninja test -j $(nproc)
125+
fi
126+
127+
#oneDNN
128+
cd $DPCPP_HOME
129+
(if cd oneDNN; then git pull; else git clone https://github.com/oneapi-src/oneDNN.git; fi)
130+
cd oneDNN
131+
mkdir -p build
132+
cd build
133+
cmake -GNinja \
134+
-DCMAKE_C_COMPILER=$DPCPP_HOME/deploy/bin/clang \
135+
-DCMAKE_CXX_COMPILER=$DPCPP_HOME/deploy/bin/clang++ \
136+
-DCMAKE_BUILD_TYPE=Release \
137+
-DCMAKE_CXX_STANDARD=17 \
138+
-DDNNL_INSTALL_MODE=BUNDLE \
139+
-DDNNL_CPU_RUNTIME=DPCPP \
140+
-DDNNL_GPU_RUNTIME=DPCPP \
141+
-DDNNL_GPU_VENDOR=NVIDIA \
142+
-DTBBROOT=$DPCPP_HOME/deploy \
143+
-DCUDA_SDK_ROOT_DIR=$CUDA_ROOT \
144+
-DOPENCLROOT=$DPCPP_HOME/OpenCL-ICD-Loader/install \
145+
-DOpenCL_INCLUDE_DIR=$DPCPP_HOME/OpenCL-Headers \
146+
-DCUBLAS_LIBRARY=$CUDA_ROOT/lib64/libcublas.so \
147+
-DCUBLAS_INCLUDE_DIR=$CUDA_ROOT/include \
148+
-DCMAKE_INSTALL_PREFIX=$DPCPP_HOME/deploy \
149+
-DDNNL_BUILD_TESTS=$cmake_test \
150+
..
151+
ninja install -j $(nproc)
152+
if $run_test; then
153+
LD_LIBRARY_PATH=$DPCPP_HOME/deploy/lib ninja test -j $(nproc)
154+
fi

0 commit comments

Comments
 (0)