Problem with parallel calculation FDS in PyroSim

[RGB-Green]

Good afternoon,

We have a server with the following specifications:
• Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz 3.59 GHz (processors: 2)
• 640 GB RAM
• NVIDIA Quadro RTX 8000
• Windows 10 Pro 22H2 19045.5965
• PyroSim 2024.2.1209

We want to use this server for parallel calculation, but we have encountered the following problems:
We use the same file but with different mesh properties
1. We removed the MPI_PROCCESS property from each mesh and started a normal calculation. FDS writes Remaining time ~4000 hrs. Calculating for half a year is not good))) But it works)
2. Removed MPI_PROCCESS property from each mesh and ran parallel calculation. FDS does not write the remaining time, that is, it is unknown how long to wait. And also we can't really understand the logs:

ENV: {PATH=C:\Program Files\PyroSim 2024\fds;C:\Program Files\PyroSim 2024\fds\mpi;C:\Windows;C:\Windows\system32, OMP_STACKSIZE=16M, OMP_NUM_THREADS=1, I_MPI_WAIT_MODE=1}
CMD: "C:\Program Files\PyroSim 2024\fds\runfds.exe" mpiexec.exe -localonly -n 16 fds.exe Scenariy01_02x02x02.fds
WDIR: P:\MOS-TLP\GROUPS\ALLGEMEIN\06_HKLS\VDA\FDSmodeling\without_MPI\Scenariy01_02x02x02

Starting FDS ...

MPI Process 13 started on MOSTLPVDI250
MPI Process 5 started on MOSTLPVDI250
MPI Process 9 started on MOSTLPVDI250
MPI Process 1 started on MOSTLPVDI250
MPI Process 11 started on MOSTLPVDI250
MPI Process 3 started on MOSTLPVDI250
MPI Process 7 started on MOSTLPVDI250
MPI Process 15 started on MOSTLPVDI250
MPI Process 10 started on MOSTLPVDI250
MPI Process 4 started on MOSTLPVDI250
MPI Process 2 started on MOSTLPVDI250
MPI Process 6 started on MOSTLPVDI250
MPI Process 14 started on MOSTLPVDI250
MPI Process 8 started on MOSTLPVDI250
MPI Process 12 started on MOSTLPVDI250
MPI Process 0 started on MOSTLPVDI250

Reading FDS input file ...

WARNING: SPEC MY_FUEL is not in the table of pre-defined species. Any unassigned SPEC variables in the input were assigned the properties of nitrogen.

Fire Dynamics Simulator

Current Date : April 22, 2025 12:43:55
Revision : FDS-6.9.1-0-g889da6a-release
Revision Date : Sun Apr 7 17:05:06 2024 -0400
Compiler : Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.9.0 Build 20230302_000000
Compilation Date : Tue 04/09/2024 01:25 PM

Number of MPI Processes: 16

MPI version: 3.1
MPI library version: Intel(R) MPI Library 2021.6 for Windows* OS

Job TITLE :
Job ID string : Scenariy01_02x02x02

forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown
forrtl: severe (41): insufficient virtual memory
Image PC Routine Line Source
fds.exe 00007FF641247AFD Unknown Unknown Unknown
fds.exe 00007FF64122A099 Unknown Unknown Unknown
fds.exe 00007FF6411F77DB Unknown Unknown Unknown
fds.exe 00007FF641BB2250 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown

3. We set MPI_PROCESS from 0 to 21 in the MPI_PROCESS mesh and started parallel calculation. FDS also does not write the remaining time and we cannot understand the logs very well:

ENV: {PATH=C:\Program Files\PyroSim 2024\fds;C:\Program Files\PyroSim 2024\fds\mpi;C:\Windows;C:\Windows\system32, OMP_STACKSIZE=16M, OMP_NUM_THREADS=2, I_MPI_WAIT_MODE=1}
CMD: "C:\Program Files\PyroSim 2024\fds\runfds.exe" mpiexec.exe -localonly -n 16 fds_openmp.exe Scenariy01_02x02x02.fds
WDIR: P:\MOS-TLP\GROUPS\ALLGEMEIN\06_HKLS\VDA\FDSmodeling!Archive\Scenariy01_02x02x02

Starting FDS ...

MPI Process 9 started on MOSTLPVDI250
MPI Process 0 started on MOSTLPVDI250
MPI Process 13 started on MOSTLPVDI250
MPI Process 1 started on MOSTLPVDI250
MPI Process 5 started on MOSTLPVDI250
MPI Process 8 started on MOSTLPVDI250
MPI Process 14 started on MOSTLPVDI250
MPI Process 10 started on MOSTLPVDI250
MPI Process 2 started on MOSTLPVDI250
MPI Process 6 started on MOSTLPVDI250
MPI Process 4 started on MOSTLPVDI250
MPI Process 11 started on MOSTLPVDI250
MPI Process 3 started on MOSTLPVDI250
MPI Process 7 started on MOSTLPVDI250
MPI Process 15 started on MOSTLPVDI250
MPI Process 12 started on MOSTLPVDI250

Reading FDS input file ...

WARNING: SPEC MY_FUEL is not in the table of pre-defined species. Any unassigned SPEC variables in the input were assigned the properties of nitrogen.

Fire Dynamics Simulator

Current Date : April 16, 2025 16:00:49
Revision : FDS-6.9.1-0-g889da6a-release
Revision Date : Sun Apr 7 17:05:06 2024 -0400
Compiler : Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.9.0 Build 20230302_000000
Compilation Date : Tue 04/09/2024 01:31 PM

Number of MPI Processes: 16
Number of OpenMP Threads: 2

MPI version: 3.1
MPI library version: Intel(R) MPI Library 2021.6 for Windows* OS

Job TITLE :
Job ID string : Scenariy01_02x02x02

forrtl: , .

forrtl: severe (38): error during write, unit 4, file P:\MOS-TLP\GROUPS\ALLGEMEIN\06_HKLS\VDA\FDSmodeling!Archive\Scenariy01_02x02x02\Scenariy01_02x02x02.smv
Image PC Routine Line Source
fds_openmp.exe 00007FF7076C2E0D Unknown Unknown Unknown
fds_openmp.exe 00007FF70EFEBC84 Unknown Unknown Unknown
fds_openmp.exe 00007FF70E7DADFE Unknown Unknown Unknown
fds_openmp.exe 00007FF70E795DCB Unknown Unknown Unknown
fds_openmp.exe 00007FF70F14EBB0 Unknown Unknown Unknown
KERNEL32.DLL 00007FFE44EB7374 Unknown Unknown Unknown
ntdll.dll 00007FFE4533CC91 Unknown Unknown Unknown

4. We set MPI_PROCESS to 0 in the MPI_PROCESS mesh and started parallel calculation. FDS immediately generates an error:

ENV: {PATH=C:\Program Files\PyroSim 2024\fds;C:\Program Files\PyroSim 2024\fds\mpi;C:\Windows;C:\Windows\system32, OMP_STACKSIZE=16M, OMP_NUM_THREADS=2, I_MPI_WAIT_MODE=1}
CMD: "C:\Program Files\PyroSim 2024\fds\runfds.exe" mpiexec.exe -localonly -n 16 fds_openmp.exe Scenariy01_02x02x02.fds
WDIR: P:\MOS-TLP\GROUPS\ALLGEMEIN\06_HKLS\VDA\FDSmodeling\with_0_MPI\Scenariy01_02x02x02

Starting FDS ...

MPI Process 0 started on MOSTLPVDI250
MPI Process 4 started on MOSTLPVDI250
MPI Process 6 started on MOSTLPVDI250
MPI Process 8 started on MOSTLPVDI250
MPI Process 14 started on MOSTLPVDI250
MPI Process 10 started on MOSTLPVDI250
MPI Process 2 started on MOSTLPVDI250
MPI Process 3 started on MOSTLPVDI250
MPI Process 5 started on MOSTLPVDI250
MPI Process 11 started on MOSTLPVDI250
MPI Process 9 started on MOSTLPVDI250
MPI Process 7 started on MOSTLPVDI250
MPI Process 15 started on MOSTLPVDI250
MPI Process 13 started on MOSTLPVDI250
MPI Process 12 started on MOSTLPVDI250
MPI Process 1 started on MOSTLPVDI250

Reading FDS input file ...

ERROR(122): Too many MPI processes have been assigned to this job. (CHID: Scenariy01_02x02x02)

ERROR: FDS was improperly set-up - FDS stopped (CHID: Scenariy01_02x02x02)
[runfds.exe] subprocess exit: 0

Please advise which option we should use and how to solve problems with it. Or are there any other options?

Thanks in advance!

[Er9y714]

Since you are trying to run this simulation through Pyrosim interface, I think it is more relevant to ask this question to Thunderhead support.

https://support.thunderheadeng.com/

[drjfloyd]

The remaining time is something Pyrosim generates not an FDS output.

How big is this case? The first error message is not enough virtual memory.

The second error message with the smv write error has been fixed (there have been changes to MPI routines since 6.9.1). Recommend you update to the latest release of FDS and try running using the FDS CMD window rather than through Pyrosim.

The third error with too many MPI processes suggests an issue with the input file vs. the number of requested processes. Without an input file no real way to diagnose further.