2DECOMP&FFT: Run sclability tests for Archer2

This README contains basic instructions for building the 2DECOMP&FFT library and run scalabilitity tests. The tests are based on

• Transposes of 3D real array
• Transposes of 3D complex array
• 3D R2C FFT starting from a X-pencil decomposition
• 3D C2C FFT starting from a X-pencil decomposition
• 3D R2C FFT starting from a Z-pencil decomposition
• 3D C2C FFT starting from a Z-pencil decomposition

Building 2DECOMP&FFT:

The build system is driven by CMake`and detailed instructions can be found here. Here a specific summary with in mind Archer2 is given.

$ git clone git@github.com:2decomp-fft/2decomp-fft.git
$ cd 2decomp-fft
$ git checkout -b v2.1

It is good practice to directly point to the MPI Fortran wrapper that you would like to use to guarantee consistency between Fortran compiler and MPI. It is recomended to use version 2.1 of the library. This can be done by setting the default Fortran environmental variable

$ export FC=ftn

The build system can then be generated by running

$ cmake -S $path_to_sources -B $path_to_build_directory -DBUILD_TESTING=ON -DENABLE_OMP=on

This will build also the testing framework and enable OpenMP. If the build directory does not exist it will be generated and it will contain the configuration files. By default a RELEASE build will built for CPU using MPI and the generic FFT backend included. In case FFTW is also available use

$ cmake -S $path_to_sources -B $path_to_build_directory -DBUILD_TESTING=ON -DENABLE_OMP=on -DFFT_Choice=fftw_f03

Once the build system has been configured, you can build 2DECOMP&FFT by running

$ cmake --build $path_to_build_directory -j <nproc>

appending -v will display additional information about the build, such as compiler flags.

Finally, the build library can be installed by running

$ cmake --install $path_to_build_directory

The default location for libdecomp2d.a is $path_to_build_directory/opt/libor $path_to_build_directory/opt/lib64 unless the variable CMAKE_INSTALL_PREFIX is modified.

Running the tests

This repository provides two bash scripts:

• SubmitJob.sh receives as input several parameters and generates a submission script using SLURM. The submission script is specific for Archer2
• An example file that given a variation of parameters generates and submit a series of submission scripts. The parameters are the following
  1. build: the path to the build directory for 2DECOMP&FFT
  2. nx: resolution in the X-direction
  3. ny: resolution in the Y-direction
  4. nz: resolution in the Z-direction
  5. tt: number of repetition of the test
  6. NN: list with the number of nodes
  7. NT: list of the tasks per node
  8. hh: max execute time hours
  9. mm: max execute time minutes

The example file is located in the folder Example and it is suggested to invoke the genration script from a directory located at the same location of the script SubmitJob.sh. The individual submission scripts will have a name that follows the structure Res${nx}x${ny}x${nz}_NP${nn}x${nt}_2D${nr}x${nc}. The .out file with all the timing will have the same root name and will be located in the same directory as the Example script. For the whole procedure to work an enviromental variable with the path to the 2DECOMP&FFT build needs to be provided as

$ export export D2D_build_path=$path_to_build_directory