Naming of metadata file

[bharsesh]

There is agreement on the location of metadata file under $OMNIBOR_DIR/metadata/context.
(#50)
But we need to discuss how the metadata file itself can be named as it is very important that we can corelate the metadata file with the output artifact. The build tool that generates the metadata file and the consumer tool/scripts that read in the metadata file need to agree on the rules to establish this correlation. A few options and their potential issues are presented below based on the meeting discussions.

1. <output_filename>.metadata : Eg, hello.metadata
   Suppose we generate an executable hello for both x86 and aarch64 and if we store the metadata in the same directory, then there will be a conflict.

2. <Omnibor_Id>.metadata : Eg, aeb4e9c2c5fc73f9811e5cd16438345a2744bb86.metadata
   Note that Omnibor_id is the gitoid of the Input Manifest. There could be potential conflicts in the naming if two different executables have the same input manifest.
   Also, there comes the question of whether we should use sha1 or sha256 for the Omnibor Id, as using both in the naming can make it awkward.

3. <Output_Artifact_id>.metadata: Eg., 3f429a63d3c65c84b3a20772b80d4244555084bc.metadata
   In this example, 3f429a63d3c65c84b3a20772b80d4244555084bc is the gitoid of the output executable.
   Here too, there comes the question of whether we should use sha1 or sha256 for the Artifact Id, as using both in the naming can make it awkward. May be we could create <sha1_artifact>.metadata and create a symlink to this file for <sha256_artifact_id>.metadata ?

It is not possible for two non-identical output artifacts to have the same gitoid hash, so this brings uniqueness to the name, but it is possible for two output artifacts to be built differently, but end up being identical.
For example, consider
gcc -O2 file.c -o exe1
gcc -O3 file.c -o exe2
clang -O2 file.c -o exe3
gcc_v2 -O2 file.c -o exe4
Both exe1 and exe2 could be identical, even when they are built differently. It is highly unlikely for exe1 and exe3 to be identical (due to differences in sections and .comment sections generated by different compilers), but exe1 and exe4 could again be identical.

The concern with this option is that if the metadata captures information about the build (build command or license info, for example), then overwriting the metadata file for identical output artifacts, could result in loss of information and potential inaccuracy of the metadata. So we need to find a way to mitigate this conflict (would concatenation of metadata help?)

[edwarnicke]

Differentiating which output artifact a metadata file is about can be achieved by using something like:

${OMNIBOR_DIR}/metadata/${context}/${output artifact id}

However, this has two potential issues:

1. It is conceivable that the same artifact could be build with different metadata depending on rebuild, different tooling on rebuild, etc.
2. It is conceivable that you may want to separate different kinds of metadata for an output artifact into separate files

(2) is simply addressed by using a directory which may contain files using the same naming convention rather than a file, so for example:

${OMNIBOR_DIR}/metadata/${context}/${output artifact id}/fileindex

For (1) we have three solutions:

1. Decide we don't care, and simply stomp the existing file.
2. Would be to use some differentiating suffix to accumulate multiple metadata files/directories for a given output artifact
3. Concatenate metadata to the end of the file, so multiple different metadata results in multiple copies of metadata in the same file

[alilleybrinker]

First, I think #3 is my least favorite option. Concatenation is likely to be confusing, error prone, in my opinion.

I think my preference is #2, with the suffix being a timestamp. These files are attestations of metadata at a particular point in time, so annotating them with the time of the attestation seems natural.

[RobMarion]

I am in agreement that solution 3 is my least favorite. One effect could be, as mentioned, unbounded growth in file size.

[alilleybrinker]

Further discussion raised the challenge of unbounded file growth in both solution 2 and solution 3. In either case, builds are frequent and we're likely to write out a large number of metadata files which will frequently be identical.

Coming up with a scheme to reduce this duplication is possible but also complicated, and there exist other mechanisms for preserving / communicating attestations that we'd like not to reinvent. Better to be orthogonal and fit together with other standards nicely.

Given this, I feel my opinion is changing toward solution 1. If users want to differentiate metadata / preserve it, then they can use an environment variable to modify the OmniBOR metadata directory.

[AevaOnline]

I'm strongly in favor of option (1), clobber the file if it already exists, because it provides a stronger guarantee of the integrity of a set of artifacts and correlated (accurate) metadata files at the end of a build.

In the event of a partial re-run of a build, whereby multiple copies of some metadata files might be generated (more likely in hand-run test/debug environments, and unlikely to be encountered in production build pipelines) it is still possible to preserve each output file by timestamping a copy at the top-level directory.

[edwarnicke]

We seem to have reached a consensus of 'stomp'

[yonhan3]

For a single build, Bomsh actually just writes to a single metadata file: bomsh_hook_raw_logfile

Then after the build, a bomsh script parses this single metadata file to get what is recorded in this metadata file.

If there are multiple metadata for the same output artifact ID, then it is up to the post-processing bomsh script to make the decision.