GPU Utilisation for Computation

[ABruce2]

Hi All, does anyone have any experience with logging jobs to GPU (and making jobs 'very' parallel rather than limited to CPU cores). I have read CUDA may be useful but am not sure how it all pieces together. Any feedback would be a great help, thanks.

[rmcdermo]

This is an area of active development, but you cannot do this now with the main branch of FDS. @marcosvanella has a research branch where he has started linking to some specific libraries with GPU capabilities.

[marcosvanella]

Hi @ABruce2, we have started looking into porting parts of FDS to GPUs making use of 3rd party libraries (PETSc for Poisson solver, Sundials for detailed chemistry). Also, we have been rewriting parts of the code to make it more amenable to OpenMP threading, with the intention of doing GPU offloading through it. The main reason behind this choice is to be able to run the code in any architecture, not only NVIDIA hardware. The code to interface with PETSc is on the FireX research branch under my user. PETSc provides linear solver implementations for NVIDIA an AMD GPUs, with INTEL hardware in the works. Ideally we want to have main parts of the time stepping algorithm running on GPU, reducing back and forth data movement between CPU and GPU memory.
Do you have experience with GPU programming in scientific computing?

[ABruce2]

Hey, thank you for both of your reply's. @marcosvanella No I do not have any experience with that but would be keen to learn if it is possible for a layperson to do, otherwise when solving fds parallel, is there any increased optimisations that can be done when solving with CPU other than spreading meshes across cores?

[marcosvanella]

Have a look at the FDS User Guide. The other option is to use threading (concurrency) on each mesh. To do this you would use the fds_openmp executable and set the OMP_NUM_THREADS environment variable. Note you need to have at least as many free cpu cores as threads you set, so each core executes one thread. The parallel scaling is not as good as splitting in more meshes and using MPI, as explained on the guide.

[KingCZE]

Do you have any advice on the compilation with some i.e. CPU specific instruction sets (AVX2, AVX-512, FMA,...), memory array (i.e. 64byte),...? Do these flags generally improve the performance on the CPU, or is the default settings the best?

[ABruce2]

I do not know about these things, however I image @marcosvanella would have more of an idea

[marcosvanella]

Well, we leave SIMD vectorization/optimization to the compiler wherever possible. For our production executable we use -O2 -ipo in the intel compiler target being distributed (-O2 for gnu compilers). There are no specific SIMD instructions in the FDS source I'm aware of.
Some info in automatic vectorization here:
https://www.intel.com/content/www/us/en/docs/fortran-compiler/developer-guide-reference/2024-2/use-automatic-vectorization.html

[12maxi21]

I also try to do fds simulations on multiple GPU cores in LINUX. At the moment i am only able to do simple simulations on one GPU Core with the command fds abcd.fds Is there any way to do the Simulation on multiple GPU Cores similiar to doing it in windows with fds_local -p 8 -o1 abcd.fds

[mcgratta]

There is a brief discussion in the User's Guide of how to run jobs on linux. However, if you are interested in GPUs, you really should learn how to write job scheduling scripts for either Torque/PBS or Slurm.