Results conversion to HDF5

[fra-kd]

Hi all. I've got a simple question: is it possible to compress the simulation's results to HDF5 format after finishing the simulation?

[marchdf]

You can output the results compressed with hdf5 and zfp if you want. Here's an example input file: https://github.com/AMReX-Combustion/PeleC/blob/development/Exec/RegTests/PMF/pmf-hdf5-zfp.inp. You can search for hdf5 in the repo to see other examples. You do need to enable this in the build. If you want both you can,

USE_HDF5  = TRUE
USE_HDF5_ZFP = TRUE

or there are cmake options

option(PELE_ENABLE_HDF5 "Enable plot file output using HDF5" OFF)
option(PELE_ENABLE_HDF5_ZFP "Enable ZFP compression in HDF5" OFF)

[fra-kd]

@marchdf Thanks so much for your fast answer! However, the problem is that I've already run the simulation, and I have the results written without compression (e.g., folders with the different Levels and so on...). How can I re-write them with the hdf5 compression? I'm basically wondering if I can do this task as "post-processing"...
Thanks in advance!

[baperry2]

I don't believe there is an exiting tool for this purpose. It should be pretty simple for you to write one though, following some of the tools in PeleAnalysis (e.g., https://github.com/AMReX-Combustion/PeleAnalysis/blob/development/Src/subPlt.cpp) - this would actually be much simpler.

[marchdf]

most likely you could do that but you would have to write another application. One that would read the existing files, and then write out new ones. We do not have such a utility. You could take inspiration from other utilities we have for plot file manipulation: https://peleanalysis.readthedocs.io/en/latest/. And then combine that with the HDF5 way of writing out plt files: https://github.com/AMReX-Combustion/PeleC/blob/development/Source/PeleCAmr.cpp#L251.

[hsitaram]

https://github.com/AMReX-Codes/amrex/tree/development/Tools/Plotfile has some nice stand-alone codes for reading and manipulating plot files. Perhaps all you need to do is read in the plot file (PlotFileData pf_a(plotfilename);) and write it out as HDF5.

…

On Thu, Mar 20, 2025 at 11:05 AM Marc T. Henry de Frahan < ***@***.***> wrote: most likely you could do that but you would have to write another application. One that would read the existing files, and then write out new ones. We do not have such a utility. You could take inspiration from other utilities we have for plot file manipulation: https://peleanalysis.readthedocs.io/en/latest/. And then combine that with the HDF5 way of writing out plt files: https://github.com/AMReX-Combustion/PeleC/blob/development/Source/PeleCAmr.cpp#L251 . — Reply to this email directly, view it on GitHub <#912 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYOQM6R3MUBZVMMEIEQMX32VLYPJAVCNFSM6AAAAABZM43QQSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENJWG4YDQNY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[fra-kd]

Thanks for your answers! Now I understood what I should do. I'll try to implement it and eventually succeed; I'll share the routine with you!

[marchdf]

Can you open a PR to the PeleAnalysis repo?

[fra-kd]

Hi all! Thanks to your suggestions, I managed to write down the routine for performing the conversion. I'm happy to share it with you! Please suggest to me how to do it!
However, after testing it, I realized that writing the results in hdf5 format does not provide any significant reduction in the file size (if compared to the folder size)! This is very upsetting for me! Do you have comments on this? Is there a way to improve it?
Thanks again!

[jrood-nrel]

pelec-hdf5-plot-file-size.tex.pdf
These are some results when I tested using compression in HDF5 files with ZFP. You can choose lossless or lossy compression and you can also choose the accuracy when using lossy compression.

[fra-kd]

@jrood-nrel thanks for your reply!
These are my settings:
USE_HDF5 = TRUE
USE_HDF5_ZFP = TRUE
hdf5_compression ="ZFP_ACCURACY@0.01"

Can you suggest something to obtain better results?