Consider replacing GitOID Construction with plain SHA-256

[alilleybrinker]

(This suggestion came from @dpp, this GitHub Discussion reproduces notes from the meeting discussion)

Proposal

Currently, the OmniBOR Artifact ID construction is a GitOID construction under the hood, meaning that we first hash in a "prefix" of the format blob <number of bytes for file>\0 before hashing in the contents of a file.

The proposal is to replace this "GitOID construction" with just plain hashing of the file itself, equivalent to what you'd get from passing a file to the sha256sum command or the equivalent for another hash algorithm.

Note that this is distinct from considerations of what hashes to officially support.

Discussion

Need to clarify the tradeoffs between the options here.

Key thoughts:

• Loss of Git compatibility
  • With this, we'd lose out on Git compatibility; though we're likely also losing this compatibility with the sole use of SHA-256 we've been considering. Few Git projects have moved to SHA-25.6
• Change of cryptographic commitment
  • This loses out on the cryptographic commitment of the number of bytes in the file, which in theory could make collisions easier because an attacker wouldn't need to worry about matching the number of bytes in the original file
  • We also lose out on the "blob" type commitment, though the Artifact ID construction doesn't vary the object type in the underlying GitOID construction, so this isn't particularly meaningful for us
• Speedup
  • Avoiding calculating the file length should speed things up for us
  • The cost means something like
    • If the file fits in memory, then you're just getting the length of the buffer with in-memory calculations
    • If it's on disk, you're seeking front-to-back; with spinning media that's quite slow.
  • We haven't benchmarked this, so it's all theoretical
• Storage
  • Having distinct GitOIDs vs. SHA-256 means more things to store, which may be challenging for OmniBOR data aggregators that want to also index SHA-256 to support cross-querying with hashes people may already have around
• Others?
  • There may be more consequences which haven't yet been considered here, for example for indexers of OmniBOR data

We're currently planning a deeper conversation around this for Monday April 22nd, as we'll be missing some WG members at the April 15th meeting.

[dpp]

But gitoids are not universally searchable, only within a single git repo

[edwarnicke]

Lots of folks have large indexes of git data across multiple repos already... and even within a single git repo, it spares a lot of reindexing.

[dpp]

Please include examples of git indexes that support sha256.

[edwarnicke]

If it's on disk, you're seeking front-to-back; with spinning media that's quite slow.

If it's on disk, the OS will happily tell us the length :) (and with Content-Length if retrieving via HTTP).

[edwarnicke]

With this, we'd lose out on Git compatibility; though we're likely also losing this compatibility with the sole use of SHA-256 we've been considering. Few Git projects have moved to SHA-25.6

Effectively, at the source level, it means having to re-index incredible amounts of source code that is already indexed via gitoid in order to search for it.

[edwarnicke]

I think historically we've wanted our artifact (and by extension manifest) identifiers to have the following characteristics:

• Canonical - Independent parties, presented with equivalent artifacts, derive the same artifact identity.
• Unique - Non-equivalent artifacts have distinct identities.
• Immutable - An identified artifact can not be modified without also changing its identity.

Clearly, at any given moment in time, any well formed hash meets these characteristics.

I'd suggest we consider these 'tier one' requirements.

At tier 2, we have questions like:

• How many useful artifacts are already indexed by this hash?
• How expensive is this hash to compute?
• How expensive is this hash to store?
• How durable is this hash likely to be? (ie, how long before it's broken, note: this is a guess generally)

Its also worth considering "Required of what when?". We have a few distinct cases we've already identified:

• Required of build tools at build time
• Required of metadata indexing tools at indexing time
• Required of ADG/metadata stores at query time

[dpp]

And the tools available by default on a system. sha256sum is available on most Linux distributions. omnibor is not.

[edwarnicke]

@dpp

git hash-object

is pretty broadly available :)