Compiler setup and path to thrust for porting RapidCFD to AMD GPUs

[klausbu]

I am trying to port RapidCFD to AMD GPUs:

ROCm is installed under /opt/rocm-6.0.0: Relevant sections are:
/opt/rocm-6.0.0/bin
/opt/rocm-6.0.0/lib
/opt/rocm-6.0.0/include
/opt/rocm-6.0.0/include/thrust
/opt/rocm-6.0.0/include/rocprim

the compiler is called "hipcc"

Using rocThrust in a project, here RapidCFD:

We recommended including rocThrust into a CMake project by using its package configuration files.

On ROCm rocThrust requires rocPRIM
find_package(rocprim REQUIRED CONFIG PATHS "/opt/rocm-6.0.0/include/rocprim")

"/opt/rocm" - default install prefix
find_package(rocthrust REQUIRED CONFIG PATHS "/opt/rocm-6.0.0/include/thrust")

...
includes rocThrust headers and roc::rocprim_hip target
target_link_libraries(<your_target> roc::rocthrust)

Where your_target refers to the GPU architecture e.g. gfx1030 for a RX6800

The problem is, that the porting tools don't port the make and configuration files. How to change the setup to use hipcc instead of nvcc and to find thrust and rocprim, a dependency for thrust?

I think the following section in "settings.sh" needs to be changed to make it happen, maybe .bashrc, too?: Can someone tell me how to specify the compiler(s) to be used and set the path to thrust'

# Location of compiler installation
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
if [ -z "$foamCompiler" ]
then
    foamCompiler=system
    echo "Warning in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
    echo "    foamCompiler not set, using '$foamCompiler'" 1>&2
fi

case "${foamCompiler}" in
OpenFOAM | ThirdParty)
    case "$WM_COMPILER" in
    Gcc | Gcc++0x | Gcc48 | Gcc48++0x)
        gcc_version=gcc-11.4.1
        gmp_version=gmp-5.1.2
        mpfr_version=mpfr-3.1.2
        mpc_version=mpc-1.0.1
        ;;
    Gcc49 | Gcc49++0x)
        gcc_version=gcc-4.9.0
        gmp_version=gmp-5.1.2
        mpfr_version=mpfr-3.1.2
        mpc_version=mpc-1.0.1
        ;;
    Clang)
        # using clang - not gcc
        export WM_CC='clang'
        export WM_CXX='clang++'
        clang_version=llvm-3.4.2
        ;;
    *)
        echo 1>&2
        echo "Warning in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
        echo "    Unknown OpenFOAM compiler type '$WM_COMPILER'" 1>&2
        echo "    Please check your settings" 1>&2
        echo 1>&2
        ;;
    esac

    # optional configuration tweaks:
    _foamSource `$WM_PROJECT_DIR/bin/foamEtcFile config/compiler.sh`

    if [ -n "$gcc_version" ]
    then
        gccDir=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER_ARCH/$gcc_version
        gmpDir=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER_ARCH/$gmp_version
        mpfrDir=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER_ARCH/$mpfr_version
        mpcDir=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER_ARCH/$mpc_version

        # Check that the compiler directory can be found
        [ -d "$gccDir" ] || {
            echo 1>&2
            echo "Warning in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
            echo "    Cannot find $gccDir installation." 1>&2
            echo "    Please install this compiler version or if you wish to use the system compiler," 1>&2
            echo "    change the 'foamCompiler' setting to 'system'" 1>&2
            echo
        }

        _foamAddMan     $gccDir/man
        _foamAddPath    $gccDir/bin

        # add compiler libraries to run-time environment
        _foamAddLib     $gccDir/lib$WM_COMPILER_LIB_ARCH

        # add gmp/mpfr libraries to run-time environment
        _foamAddLib     $gmpDir/lib
        _foamAddLib     $mpfrDir/lib

        # add mpc libraries (not need for older gcc) to run-time environment
        if [ -n "$mpc_version" ]
        then
            _foamAddLib     $mpcDir/lib
        fi

        # used by boost/CGAL:
        export MPFR_ARCH_PATH=$mpfrDir
        export GMP_ARCH_PATH=$gmpDir
    fi
    unset gcc_version gccDir
    unset gmp_version gmpDir  mpfr_version mpfrDir  mpc_version mpcDir

    if [ -n "$clang_version" ]
    then
        clangDir=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER_ARCH/$clang_version

        # Check that the compiler directory can be found
        [ -d "$clangDir" ] || {
            echo 1>&2
            echo "Warning in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
            echo "    Cannot find $clangDir installation." 1>&2
            echo "    Please install this compiler version or if you wish to use the system compiler," 1>&2
            echo "    change the 'foamCompiler' setting to 'system'" 1>&2
            echo 1>&2
        }

        _foamAddMan     $clangDir/share/man
        _foamAddPath    $clangDir/bin
    fi
    unset clang_version clangDir
    ;;
system)
    case "$WM_COMPILER" in
    Clang)
        export WM_CC='clang'
        export WM_CXX='clang++'
        clang_version=llvm-9.0.0

        cudaHome="$(dirname $(dirname $(which nvcc)))"
        [ -d "$cudaHome" ] || {
            echo 1>&2
            echo "Error in $WM_PROJECT_DIR/etc/config/settings.sh:" 1>&2
            echo "    Cannot find CUDA installation." 1>&2
            echo "    Found only $cudaHome." 1>&2
            echo 1>&2
            exit 1
        }
        export CUDA_HOME=$cudaHome
        #an option to override thrust
        if [ -d "$THRUST_HOME" ]
        then
            export THRUST_INCLUDE=-I$THRUST_HOME
        else
            export THRUST_INCLUDE=/opt/rocm-6.0.0/include/thrust
        fi
        unset cudaHome
        ;;
    esac
    # okay, use system compiler
    ;;
*)
    echo "Warn: foamCompiler='$foamCompiler' is unsupported" 1>&2
    echo "   treating as 'system' instead" 1>&2
    ;;
esac


#
# add c++0x flags for external programs
#
if [ -n "$WM_CXXFLAGS" ]
then
    case "$WM_COMPILER" in
    Gcc*++0x)
        WM_CXXFLAGS="$WM_CXXFLAGS -std=c++0x"
        ;;
    esac
fi

[wtkaczewski]

Hello
Check my fork of RapidCFD :) I've manage to compile and run cases with amd gpus.
There is no issues with rocm-sdk 4.7.2 (based on clang-13) - I've started with this version because it is the last one that supports polaris gpus. In newer versions based on clang 17 there is a compile error which is not fixed yet.

[klausbu]

Great! Thank you!

[daniel-jasinski]

Great job @wtkaczewski with your AMD port!!!

I can see that with minor modifications, it could be compatible with both CUDA and ROCM.
Would you be interested in making a few changes so that we could merge your fork into this repository?
I can provide some suggestions if needed.

Daniel

[wtkaczewski]

I'm working right now on wmake rules that allows to build cuda version by AMD hipcc compiler. I will fill pill request when I will be sure that source is compiling for NVIDIA and AMD gpus.

[daniel-jasinski]

Super! Looking forward to it!

[klausbu]

@wtkaczewski

I have been trying to compile your AMD version but when I source the .bashrc file as usual for an OpenFOAM installation, the terminal crashes, when I add source $HOME/OpenFOAM/RapidCFD-dev/etc/bashrc to the user .bashrc file to load it at boot time, the terminal crashes, when I try to open it.

My OpenFOAM installation works fine and of course, I am changing the .bashrc settings so they don't conflict.

I am using Ubuntu 22.04.

Do you have any idea what could cause the problem?

[klausbu]

Here is the fix to install RapidCFD with AMD GPU support:
1: In the bashrc change Clang to hipcc (this will call ROCm Clang or Nvcc depending on the GPU architecture if Nvcc i.e. cuda is enabled as a ROCm backend)
2: Rename the rules folder in wmake/rules/ called linux64Clang to linux64hipcc
3: Change foamInstall to foamInstall=$HOME/$WM_PROJECT for a standard installation in your home directory under RapidCFD

There another problem: OSHA1stream.H is outdated as it doesn't meet c++11 standards hence many simulations will not run due to IO errors. Current compilers like LLVM included in ROCm expect c++11 compliance. This has been fixed in OpenFOAM in 2022 but RapidCFD has not been updated accordingly. The OpenFOAM fix details can be found here: OpenFOAM/OpenFOAM-9@b0c15be unfortunately I lack the skills to implement it in RapidCFD