zdc_lyso benchmark started to occasionally fail

[veprbl]

Here is an example failure from https://eicweb.phy.anl.gov/EIC/benchmarks/detector_benchmarks/-/jobs/5811057

Traceback (most recent call last):
  File "/builds/EIC/benchmarks/detector_benchmarks/benchmarks/zdc_lyso/analysis/analysis.py", line 217, in <module>
    plt.tight_layout()
  File "/opt/local/lib/python3.12/site-packages/matplotlib/pyplot.py", line 2349, in tight_layout
    return gcf().tight_layout(pad=pad, h_pad=h_pad, w_pad=w_pad, rect=rect)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/figure.py", line 3509, in tight_layout
    engine.execute(self)
  File "/opt/local/lib/python3.12/site-packages/matplotlib/layout_engine.py", line 178, in execute
    kwargs = get_tight_layout_figure(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/_tight_layout.py", line 266, in get_tight_layout_figure
    kwargs = _auto_adjust_subplotpars(fig, renderer,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/_tight_layout.py", line 82, in _auto_adjust_subplotpars
    bb += [martist._get_tightbbox_for_layout_only(ax, renderer)]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/artist.py", line 1415, in _get_tightbbox_for_layout_only
    return obj.get_tightbbox(*args, **{**kwargs, "for_layout_only": True})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/axes/_base.py", line 4385, in get_tightbbox
    ba = martist._get_tightbbox_for_layout_only(axis, renderer)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/artist.py", line 1415, in _get_tightbbox_for_layout_only
    return obj.get_tightbbox(*args, **{**kwargs, "for_layout_only": True})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/axis.py", line 1323, in get_tightbbox
    ticks_to_draw = self._update_ticks()
                    ^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/axis.py", line 1262, in _update_ticks
    major_locs = self.get_majorticklocs()
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/axis.py", line 1484, in get_majorticklocs
    return self.major.locator()
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/ticker.py", line 2341, in __call__
    return self.tick_values(vmin, vmax)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/lib/python3.12/site-packages/matplotlib/ticker.py", line 2358, in tick_values
    raise ValueError(
ValueError: Data has no positive values, and therefore can not be log-scaled.

I've added some debug outputs in #177, and it looks like we have some sort of nan occurring somewhere:

Energy [0.005, 0.01, 0.05, 0.1, 0.5, 1.0]
emean [nan, nan, nan, nan, nan, nan]
hmean [nan, nan, nan, nan, nan, nan]

[veprbl]

I'm able to reproduce locally, actually.

It appears that there might be an issue with the dataframe produced (indices are not aligned?):

     ecal_reco_energy_5 ecal_rec_energy_5 ecal_sim_energy_5 hcal_reco_energy_5  ... ecal_reco_nhits_1000 par_x_1000 par_y_1000 par_z_1000
0              0.004028          0.004028             0.005                  0  ...                   10  -0.027169  -0.002004   0.999629
1              0.004761          0.004761             0.005                  0  ...                    9  -0.025042  -0.000404   0.999686
2              0.004272          0.004272          0.004489                  0  ...                    8  -0.024918  -0.000189   0.999689
3                     0                 0                 0                  0  ...                    8  -0.028294   0.001299   0.999599
4              0.004028          0.004028          0.004612                  0  ...                   12  -0.027247  -0.002065   0.999627
...                 ...               ...               ...                ...  ...                  ...        ...        ...        ...
996            0.004639          0.004639          0.004601                  0  ...                    0  -0.028836   0.000257   0.999584
997                   0                 0                 0                  0  ...                   15   -0.02725  -0.003691   0.999622
998                   0                 0                 0                  0  ...                    0  -0.024453   0.000447   0.999701
999                   0                 0                 0                  0  ...                    9  -0.024919   0.000514   0.999689
1000                  0                 0               NaN                  0  ...                    0        NaN        NaN        NaN

@bschmookler, @JiaJunHuang120000 could you check?

[bschmookler]

Ok. We will look into this.