Benchmark to verify simulation beamline optics

[simonge]

The benchmark aims to verify that the optics used in the dd4hep epic simulation match those expected by the eic machine.

Proposed components of the finalized benchmark:

• Plots showing the position, momentum and correlations for beam electrons passing through the beamline magnets. Create Beamline benchmark #154
• Link directly to machine distributions.
• Generate events using the afterburner.
• Energy-theta acceptance at each virtual detector.
  Beamline acceptance benchmark #167
• Labelled beamline geometry visualization output.
• Extend to include far-forward protons
• Create report pdf, tidying up all plots.
• Train onnx neural network to reconstruct initial particle and produce resolutions.
  Beamline neural network reconstruction training #171
• Create pipeline to automatically update the onnx network used in the reconstruction.
  Close the loop on retraining Low-Q2 onnx models #201

[simonge]

@veprbl @wdconinc
Before I try and implement the task at the end of my list I would like to check that my proposed approach is reasonable and possible.

Proposed workflow (1, 2 and 3 are planned for #171, 4 onwards need to be developed)

1. Check if reconstruction currently meets test criteria
2. If it fails, retrain the model
3. Check the new model meets the test criteria
4. If it succeeds, create a PR to epic-data which updates the model from the benchmark artefact.
5. Create a PR to the epic branch which triggered the benchmarks with the updated permalink.

It will probably only be changes to the geometry which will cause the tests to fail, it would be less obvious how to handle a change to EICrecon breaking the reconstruction.

[wdconinc]

Sounds good to me. Since the pipeline runs on eicweb, and we don't pass a token with permissions to open PRs on other repositories, step 4 will likely not work out of the box.

[veprbl]

Yes, we were thinking about a similar approach for material maps. We can provide benchmark with tokens with permissions to open github PRs when needed.

[simonge]

For a first step would it be ok to add some github CI to epic-data? With logic such that if onnx/Low-Q2_Steering_Reconstruction.onnx is updated it either creates a new PR to epic, or if the commit message contains a specific epic repository PR it creates a review comment and suggestion there.

Or should epic-data be kept purely as a place to store calibrations etc. without any moving parts? The update of epic-data and epic could both be handled by the detector_benchmarks CI.

It feels like there's a lot of potential for the file on epic-data to be updated several times if the user doesn't understand the workflow.

[wdconinc]

Or should epic-data be kept purely as a place to store calibrations etc. without any moving parts? The update of epic-data and epic could both be handled by the detector_benchmarks CI.

Just from a practical point of view, we had considered epic-data to be a temporary repository of those artifacts only, with the idea that we would ultimately move to a more permanent (non-github) space. If we start adding CI to epic-data then we would at some point have to port that over as well. There's lower probability that we will move away from doing CI in epic or detector_benchmarks.

[veprbl]

Thanks for raising this important point, that I forgot to mention. We can not continuously add things to epic-data as we want to stay away from hitting github limits. I'd like to propose that we store calibrations on XRootD. We already have a setup for uploading artifacts from eicweb to XRootD. The only missing component is adding root:///roots:// protocol support to https://github.com/eic/epic/blob/main/src/FileLoader.cpp using xrdcp instead of curl. We might have to implement some cleanup policy for files that are not part of any calibration (e.g. automated PRs submitted but never accepted), but I think that's not something we need to overthink right now.

[simonge]

Ok, that sounds more reasonable and doesn't look like much work to allow an xrdcp alternative command. I could attempt to make that update unless you're already on it.

Where is the appropriate space on XRootD to store these and where can I find an example which uploads from eicweb?

[veprbl]

Where is the appropriate space on XRootD to store these and where can I find an example which uploads from eicweb?

This is the code

detector_benchmarks/.gitlab-ci.yml

Lines 191 to 195 in 5f54c8b

	- wget https://dl.pelicanplatform.org/7.13.0/pelican_Linux_x86_64.tar.gz
	- sha256sum -c <(echo '38ac8548c67302299e50a1b81c159ed418e90d84a6606ddd377fd2c8b164d114 pelican_Linux_x86_64.tar.gz')
	- tar zxf pelican_Linux_x86_64.tar.gz
	- mv results pipeline-$CI_PIPELINE_ID; tar cf pipeline-$CI_PIPELINE_ID.tar pipeline-$CI_PIPELINE_ID/; mv pipeline-$CI_PIPELINE_ID results
	- ./pelican-*/pelican object copy pipeline-$CI_PIPELINE_ID.tar $OSDF_ENDPOINT$OSDF_OUTPUT_PREFIX/pipeline-$CI_PIPELINE_ID.tar

we can switch from pelican. I think, for initial tests you can write to (a subdirectory of) /volatile/eic/EPIC/xrdtest, and we can setup a more official directory, like /volatile/eic/EPIC/CALIB.

Ok, that sounds more reasonable and doesn't look like much work to allow an xrdcp alternative command. I could attempt to make that update unless you're already on it.

Please, go ahead.

[simonge]

Please could you manually copy the Low-Q2_Steering_Reconstruction.onnx file into /volatile/eic/EPIC/xrdtest/... for the meantime to allow testing.

I don't think the automatic triggering of a PR on more manual updating of the file would be possible in this approach but hopefully that doesn't matter.

[veprbl]

Here: root://dtn-eic.jlab.org//volatile/eic/EPIC/xrdtest/CALIB/2025/5a202b05d865214e3c399883ee13859318044678f87d7b3d9fa8ff526b4909b6/Low-Q2_Steering_Reconstruction.onnx

[simonge]

Thanks @veprbl. Where are OSDF_ENDPOINT and OSDF_OUTPUT_PREFIX being defined?

[veprbl]

Those are pipeline variables defined on eicweb instance. I've added OSDF_CALIB_OUTPUT_PREFIX to point to /eic/volatile/EPIC/xrdtest/CALIB (eic and volatile have to be swapped here)