TACC Open Hackathon 2024

Some notes for organizing our efforts

!!! Need at least three people for every day

Agenda (all times CST)

• Tues Oct 8 10 AM – 11:30 AM online
  • Meet with mentor
• Tues Oct 15 9 AM – 5 PM online
  • Cluster intro
  • Introductory team presentations
  • Work with mentor
• Tues Oct 22 – Thurs Oct 24 9 AM – 5 PM hybrid
  • Work on code with mentor

Our problem

• Parthenon codes (especially ones that aren’t AthenaPK) scale poorly (~75% efficiency) on GPU clusters, especially with small meshblocks – highest priority
• Single precision?
• Multigrid performance
• Automated performance regression testing
• Optimize kernel launch properties for few large meshblocks
• AMD communication library (need MPI abstraction layer?)
• Revive host-pinned buffers with GH shared memory

Our ideas

• Too many MPI messages? Combine messages prior to send. Less trivial with sparse data.
• Per-boundary kernel launches? Or use new par_for_outer
• Autotuning in Kokkos
• Ask Forrest how we can leverage Grace-Hopper features
• Ask Forrest about a more formal way of switching between a default Kokkos backend and optional hardware (e.g. CUDA)-specific specializations for performance tuning.

Our proposed work

• Create simple test problems (with/without sparse fields?) that illustrate this issue at scale
• Try dense first, then generalize to sparse?
• Be able to use the machines soon!
• Implement our ideas
• Measure performance improvement