GGUF File Format Improvement Wishlist

[mofosyne]

Still currently investigating gguf file format and ways to improve it. So far this is my wishlist/investigation/strategy points to consider:

• Make GGUF easier to validate for testing / regression checking
  • Let's have a dummy json file format that can act in place of a real gguf layers but contains only hashes for compact regression testing of conversion processes.
  • Let's have GGUF tensor info structure contain a hash (and hash type for futureproofing) as well as offset and size of data area to ensure we can self validate the integrity of a GGUF file.
  • Let's have some way of self describing the superblock, block internal structure? (blue sky wish)
  • For hash functions...
    • Considering xxhash or metrohash for 'per layer' hash as we are only aiming for consistency checks if main whole model hash fails (or for regression ci testing).
    • Considering SHA1 or SHA256 for whole tensor data area hashing (depending on how much we care about security or consistency). The idea of that would be that the same model and quantization will hash to the same hash.
    • Consider using https://github.com/multiformats/multihash to ensure futureproofing of hash standards
• Add UUID to kv store... but haven't decided how unique it should be. (Should same models but different metadata have different UUID?)
• Add mechanism to jump over the metadata kv area... or maybe relocate the meta kv at the end of the tensor files so it can be more easily edited
• Maybe KV store is done using CBOR? (Could we get away with using CBOR as much as possible? Not sure how to deal with alignment in tensor data payloads)
• Every section should have an offset and size to make it easier to jump over unknown section
• Type IDs that are harder to coordinate should really be a random 64bit ID (or UUID)
• Each tensor info should contain
  • hash of each layer for validation
  • size of each tensor area so hashing of future arch is easier, in addition to offset

Bonus cookie (but rather half baked because of potential security complications)

• Should we append a zip file to store stuff like image and compressed model cards?

Also let's address any file format security issues:

https://www.databricks.com/blog/ggml-gguf-file-format-vulnerabilities (As pointed out by #7890)

Longer shot investigation

Could Kaitai struct be a possible path to a more well defined and safe gguf file format?

https://doc.kaitai.io/

[ggerganov]

Overall, I'm against complicating the file format with anything that is not really necessary. For every addition, there should be a meaningful usecase that it will enable and no other alternative to accomplish it in a reasonable way. I don't think any of the suggestion in this issue qualifies as such