Issue running simulation with finer mesh Xcompact3d

[Fahimeh-Goodarzi]

Hello Team,

I hope you’re doing well.

I’m running a jet simulation with XCompact3D on Compute Canada’s Niagara system. A previous run using a mesh of 2001 × 1000 × 800 completed successfully. After refining the mesh to 2701 × 1250 × 1024 (all other inputs unchanged), the job now crashes at iteration 200 with Δt = 0.001 and Re=350.
Runtime settings (unchanged):
icheckpoint = 50
ioutput = 20
ilist = 10
nvisu = 1
Batch-script modules:
module load NiaEnv/2022a
module load gcc/11.3.0
module load openmpi/5.0.2+ucx-1.15.0
Error excerpt
==== backtrace (tid: 3161) ====
0 0x00000000000248e0 uct_ud_ep_deferred_timeout_handler() ud_ep.c:0
1 0x0000000000053018 ucs_callbackq_slow_proxy() callbackq.c:0
2 0x0000000000049a9a ucp_worker_progress() ???:0
3 0x00000000000237c4 opal_progress() ???:0
4 0x000000000008e1f5 ompi_request_default_wait_all() ???:0
5 0x00000000001612fc mca_fcoll_vulcan_file_write_all() ???:0
6 0x00000000001a9ace mca_io_ompio_file_write_all() ???:0
7 0x00000000000ae9ae MPI_File_write_all() ???:0
8 0x000000000004e91b MPI_File_write_all_f08() ???:0
9 0x000000000043bbb8 __decomp_2d_io_MOD_mpiio_write_real_coarse() ???:0
10 0x000000000061298a __tools_MOD_restart() ???:0
11 0x000000000040313f main() ???:0
12 0x0000000000022555 __libc_start_main() ???:0
13 0x0000000000403289 _start() ???:0
...
[nia0041:3161] ud_ep.c:278 Fatal: UD endpoint … : unhandled timeout error
MPI_File_write_all
__decomp_2d_io_MOD_mpiio_write_real_coarse
__tools_MOD_restart
xcompact3d
Could you please advise on:

Whether the error points to an MPI / I/O bottleneck triggered by the larger mesh size.

Recommended adjustments—e.g., UCX environment variables, file striping, checkpoint frequency, or memory-per-core settings.

An alternative OpenMPI module known to work reliably with XCompact3D on Niagara.

Thank you very much for your help.

[pbartholomew08]

Hi Fahimeh, I'm afraid we can't provide specific advice for systems that we don't have experience using. Xcompact3d should work with OpenMPI though.

Since it appears to have failed during I/O I'd suggest making sure you've not exceeded your file quota on the system.
Also please be aware the Jet case is unsupported and has been removed from later versions of Xcompact3d.

[Fahimeh-Goodarzi]

Hi Paul, thank you for the response, I modeled jet in BC User , every thing works with my regular mesh and the results are reasonable, but for the finer mesh it fails and has error, Re=350 .

[pbartholomew08]

On ARCHER2 it was generally beneficial to use the maximum Lustre striping possible for I/O performance. If you are using a reasonably recent version of Xcompact3d it should retain only a single checkpoint (although it temporarily creates 2 in case the new one fails).

Otherwise for UCX settings if your site has any recommendations for "3-D FFT codes" these might be worth trying.

[mathrack]

Hello,

Have you tried to write only one restart during the simulation or to avoid restart validation ? (logical flag validation_restart in the section InOutParam of the namelist)

[Fahimeh-Goodarzi]

Thank you, I will try this one.

[Fahimeh-Goodarzi]

Hello , thank you for the comments , I put icheckpoint to 1000 and ilast 1000, the simulation run until 1000 and saved without failure, the problem is the one restart during simulation as you said, now I want to continue my simulation from 1001 to 2001, for the rest of simulation do I need checkpoint from previous simulation? or I can delete it?(because of memory I think I need to delete it)

[pbartholomew08]

You would need that checkpoint to continue the simulation from timestep 1001. I think as @mathrack suggested trying disabling the restart validation may help if that's what is failing. If the problem is space related then I think your only option would be to disable checkpointing for your restarted run (set icheckpoint = 1000000 or some similarly large number beyond ilast)

[Fahimeh-Goodarzi]

Thanks ,I think the problem is memory because one node goes out of memory for this finer mesh. I will try this (I checkpoint = 100000).

[Fahimeh-Goodarzi]

the problem solved. thank you all very much , as you suggested, I put I checkpoint=500, and run simulations for 500 iterations so it works for each simulation.