Excessive RAM Usage – 1 TB RAM not sufficient for 12 Mio cells – any guidance?

[stephanB79]

Hello,

I want to simulate gas dispersion in a process plant module (30x30x30m) using a total of approx. 12 Mio grid cells. The geometry is highly complex involving piping, structural steel and equipment. The CAD file was converted to &OSBT using PyroSim.

With the target grid resolution of 0.125m, I´m getting around 250.000 &OBST lines from PyroSim. I was trying to run it on a cluster with 64 or 128 or 192 cores using AMD EPYC with 64 cores per CPU and 512 GB of RAM per CPU but all have the same problem.

The geometry setup for smokeview works, but FDS-6.9.1-0 is running out of memory before the 1st iteration starts. It does not reach the point where the *.out file is generated.

It`s MPI version: 3.1 with MPI library version: Intel(R) MPI Library 2021.11. With the same set-up, less complex geometries worked flawless.

The error message returned by slurm is not helping but it`s obvious that the complete memory is filled prior to the termination.

[0] MPI startup(): I_MPI_ROOT=/hpc/appl/intel/mpi/2021.11
[0] MPI startup(): I_MPI_MPIRUN=mpirun
[0] MPI startup(): I_MPI_BIND_WIN_ALLOCATE=localalloc
[0] MPI startup(): I_MPI_HYDRA_TOPOLIB=hwloc
[0] MPI startup(): I_MPI_HYDRA_BOOTSTRAP=slurm
[0] MPI startup(): I_MPI_RETURN_WIN_MEM_NUMA=0
[0] MPI startup(): I_MPI_INTERNAL_MEM_POLICY=default
[0] MPI startup(): I_MPI_DEBUG=5



   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
   RANK 0 PID 3038159 RUNNING AT ….
   KILLED BY SIGNAL: 9 (Killed)

   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
   RANK 1 PID 3038160 RUNNING AT  …
   KILLED BY SIGNAL: 9 (Killed)

In a different workstation with windows but only 128GB of RAM , the error message is

MPI version: 3.1
 MPI library version: Intel(R) MPI Library 2021.6 for Windows* OS


 Job TITLE        :
 Job ID string    : …….


ERROR: Memory allocation failed for CONNECTED_ZONES in the routine INIT (MPI Process: 0, CHID:….)

ERROR: FDS was improperly set-up - FDS stopped (CHID: ….)

Apart from the obstruction, the fds input is rather simple:

It contains slices for velocity vectors and volume fraction as well as mass integrals over the entire computational domain and two isosurfaces for volume fraction.

The remaining set-up is:

&DUMP NFRAMES=5000/
&DUMP DT_ISOF = 0.2/
&DUMP DT_SLCF = 0.1/
&DUMP DT_PART = 0.1 /
&DUMP DT_DEVC = 0.01 /
&DUMP DT_MASS = 0.01 /
&DUMP MASS_FILE=.TRUE. /

&MISC SHARED_FILE_SYSTEM=F  

&MISC SIMULATION_MODE='LES' / 
&MISC VN_MIN=0.4/
&MISC VN_MAX=0.5/

&SPEC ID='LJ AIR', BACKGROUND=T/
&SPEC ID='HYDROGEN' /

&RADI RADIATION=F/

Do you have any guidance other than lowering the grid resolution which would impair the dispersion modelling in the highly complex geometry?

Reducing the number of slices or integrals so far did not help.

Is there a way to estimate how much RAM would be required other than trial and error with meshing only smaller segments of the geometry?

Would it help if the model is distributed to less cores not using all cores available to increase the number of RAM per core?

[drjfloyd]

Unless you are using HT3D or BURNAWAY everywhere I wouldn't expect a 12 million cell case to need more than 100 Gb.

You could try 6.10.1. There have been some edits to allocations and MPI calls since 6.9.1.

It is difficult to suggest possible causes without the actual input file.

Are you exporting OBST from Pyrosim that are completely subgrid? If so, removing those would reduce the memory required.
You could try removing portions of the input file. For example, does it run if you remove half the OBST and if it doesn't does it run if you instead remove the other half?

[stephanB79]

Thanks, I´m not using HT3D or BURNAWAY. Its a simple gas dispersion calculation without combustion.

The OBST exported from Pyrosim are on grid level only, but its 250 thousand lines like this which increase the *.inp file size to 30 MB as the geometry is highly complex.

&OBST ID='Object_26462', XB=-156.0,-155.875,-189.25,-189.125,114.75,114.75, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26462', XB=-156.0,-155.875,-188.875,-188.75,114.75,114.75, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26463', XB=-156.0,-155.875,-189.25,-189.125,124.0,124.0, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26463', XB=-156.0,-155.875,-188.875,-188.75,124.0,124.0, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-155.625,-155.5,-177.625,-177.5,113.25,113.25, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-155.625,-155.5,-177.625,-177.5,113.375,113.375, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-156.0,-155.625,-178.0,-178.0,113.25,113.375, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-156.0,-155.625,-177.375,-177.375,113.25,113.375, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-155.625,-155.5,-177.75,-177.75,113.25,113.375, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-155.625,-155.5,-177.625,-177.625,113.25,113.375, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-155.625,-155.5,-177.5,-177.5,113.25,113.375, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-155.875,-155.875,-177.375,-177.25,113.25,113.375, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-155.625,-155.625,-178.0,-177.75,113.25,113.375, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-155.625,-155.625,-177.625,-177.375,113.25,113.375, RGB=44,44,44, SURF_ID='MatID8504'/ 
&OBST ID='Object_26467', XB=-155.5,-155.5,-177.75,-177.5,113.25,113.375, RGB=44,44,44, SURF_ID='MatID8504'/ 

I will try smaller files with 100 thousand and 50 thousand lines of OSBT and get the latest version of FDS installed.

[johodges]

You probably have a lot of small holes in the domain which FDS is assigning to separate zones. Looks like we are building the CONNECTED_ZONES array before filling in small zones according to MINIMUM_ZONE_VOLUME. My guess is this is intentional so that we can check if two independently identified zones in separate meshes are actually the same zone before applying the volume check, but @mcgratta would know better. Could we fill the small created zones before building the CONNECTED_ZONES array?

[mcgratta]

I think we allocate CONNECTED_ZONES after filling in unwanted zones. In any case, I will see if there is a way to avoid having this large 2-D array of connected zones. In the meantime try to reduce the number of zones. There must be hundreds or thousands of them, and I cannot imagine that these are necessary.

[stephanB79]

Thanks for your guidance! I tried 1/5th of the exported OBST resulting in a slowly running model at 1.6 TB of RAM with approximately 3000 volumes in Pressure Zone Information.

But with &MISC NO_PRESSURE_ZONES=.TRUE. the complete model is running smoothly with a RAM usage below 200GB.

I`m not sure if NO_PRESSURE_ZONES=.TRUE. was a good idea, but from my perspective I'm only interested in the (open) atmospheric flow around the equipment/piping. What is happening inside of the equipment/piping is irrelevant, so increasing the MINIMUM_ZONE_VOLUME might also be an option which I did not yet test.