Storage of job execution environment in output files [2/4]

[PetrilloAtWork]

This PR implements a system for storing into some output files information ("SBN metadata") about the execution environment of each job.

From user perspective, adding a "plugin" to the job output module (RootOutput) configuration will have this metadata saved into all the art/ROOT output files and to the TFileService file if available.

Users can read back the metadata in art/ROOT files with a special output module (configuration dump_sbnjobmetadata.fcl provided in sbncode), and can unpack the one in the TFileService file with EvtInfo->dump(std::cout) provided that the dictionary of the class is available.

What code is where

The system includes in sbnobj the data class holding the metadata (sbn::JobEnvironmentInfo) and the small CMake library to extract the GIT branch version, and in sbncode the art-based modules and plugins and the job configuration file for dumping the metadata from art/ROOT file, plus extensive documentation of the whole system in Doxygen format.
The system requires a modification in each repository that wants the GIT branch version extracted.
This PR provides those modifications for icarusalg and icaruscode. In addition, since ICARUS uses a template RootOutput configuration, that configuration has been changed to include the plugin that will save the information.

PR summary

• Storage of job execution environment in output files [1/4] sbnobj#118
• Storage of job execution environment in output files [2/4] #497 (primary — this one)
• Storage of job execution environment in output files [3/4] icarusalg#90
• Storage of job execution environment in output files [4/4] icaruscode#794

What is included in the metadata

This PR includes in the metadata:

• the full environment variable set of the job execution environment, with the exception of the shell function;
• art version;
• art process name;
• GIT branch description for all instrumented GIT repositories.

I attach an example of output from dump_sbnmetadata.fcl on a test file created by two "empty" jobs in chain.

If art/ROOT input files contain SBN metadata, that will also be replicated in the art/ROOT output file(s), but not in the TFileService one.

Some details of how this works and its limits

The system writes its metadata in a Results-level data product, which is unlike the ones we are used to in that is accessible basically only by the output modules. While this makes some sense, it also makes the programmatic usage of the metadata much, much harder.

The extraction of the execution environment information is pretty straightforward.
Conversely, the extraction of the GIT branch information, is not.
The pattern of the system is that the CMake building files of a repository need to include instructions to extract from GIT the branch information (git describe) and to put that into a C++ shared library (might have been something else, with different pros and cons). This is a fairly simple set of instruction, but long enough that the PR provides that in a SBNutil.cmake library in sbnobj that the repository CMakeLists.txt need to include.
Then, the art-aware repositories (so, for example sbncode but not sbnobj) need to define a art tool that links to the metadata and returns it.
An algorithm class, sbn::JobEnvironmentInfoExtractor, is provided in sbncode which calls all the tools it knows (from its configuration) and fills the list of metadata.
Finally, the RootOutput plugin SaveJobEnvironment (sbncode) is the front-end executing that algorithm (and passing it the list of known repositories/tools) and storing the result into the output.

Another important limit is that the CMake macros used here are stored in sbnobj, which is the lowest level repository we have in SBN. For once, it is questionable that they belong here; and, more fatally, there are repositories which do not depend on sbnobj (e.g. sbnanaobj) and that as a consequence can't use them. sbnana would have been a natural candidate for inclusion in the system, but it seems unlikely given that it does not depend on sbnobj either (it does depend on sbndata, which is a questionable workaround but still one; sbnanaobj, on the other end, depends almost only on ROOT, and by design).

This system is extensively described in SBNsourceMetadataSystem.dox file.

Testing

The system, in the final incarnation in this PR, has been tested with eight combinations of builds including or not including sbnobj, sbncode, icarusalg and icaruscode.
In the process, a few defects have been found and corrected in the build scripts (typically, missing stuff that was overlooked because was being fortuitously provided by some other package in the build).

Review

I am calling for the review:

• @SFBayLaser, who five days ago was lamenting the lack of information like this;
• @absolution1, especially for his considerations on including this system in SBND too.

This system is complicate, and honestly should have been introduced at art level.
A lot of design was involved, and there were a lot of choices made in the process.
The sooner these choices are pondered, tested, discussed and challenged, the better: it is unlikely that design reconsideration will result in a backward-compatible change. Backward compatibility here is not a strong requirement, but it does not hurt.

I don't know how badly this interact with the Spack-based build system. Provided that CMake is still there, this system should still work, although not necessarily with satisfaction.
For example, the dump of the environment implies that all UPS products are trackable; this might not be the case with Spack any more. However, when it is the time, the system can be tuned to the new build system.

[PetrilloAtWork]

• need to include the GIT extraction macros from sbnobj

[PetrilloAtWork]

• added a new job configuration and new directory

[PetrilloAtWork]

• added a new job configuration and new directory

[PetrilloAtWork]

• a new job configuration for dumping the SBN metadata

[PetrilloAtWork]

• metadata "framework":
  • added algorithm to extract SBN metadata
  • added output plugin to write the metadata into art output
  • added the output module to dump that metadata from files to screen
• package metadata:
  • added macros for extraction of the version of this repository
  • added plugin for reporting versions of sbncode and sbnobj

[PetrilloAtWork]

• output module dumping SBN metadata from an input file to screen

[PetrilloAtWork]

• template source for sbncode repository version library

[PetrilloAtWork]

• template interface for sbncode repository version library

[PetrilloAtWork]

• implementation of the SBN metadata extraction algorithm

[PetrilloAtWork]

• interface of the algorithm for extracting SBN metadata

[PetrilloAtWork]

• helpers to add information to the data product class (which we keep simple...r)

[PetrilloAtWork]

• helpers to add information to the data product class (which we keep simple...r)

[PetrilloAtWork]

• interface of the art tool to collect version information from the various repositories

[PetrilloAtWork]

• complete documentation of the system in Doxygen format

[PetrilloAtWork]

• output module plugin to save SBN metadata into art output

[PetrilloAtWork]

• sbncode-specific tool reporting the version of sbncode and sbnobj repositories

[SFBayLaser]

I love this! But I need to digest... so give me a bit of time...

[kjplows]

Hi @PetrilloAtWork, @SFBayLaser, and @absolution1 - just pinging to check in on this PR. I've merged develop into the branch - does this still look good to you? Would be great if we could merge this. Thanks!