Add a spare 1KiB field to end of GGUF metadata when writing the model

[KerfuffleV2]

I propose we add an unused field when creating GGUF files (can allow an option to disable this) that's just something like key __SPARE and an array of 1,024 UINT8s or something like that. This space can be used to add fields to the metadata when necessary, without having to recreate the entire model file.

For example, if you wanted to add a tokenizer.ggml.bos_token_id = 1 (UINT32) field, you can just overwrite the __SPARE field and then write an adjusted __SPARE field into the remaining space. (This is something tools can calculate pretty easily, the user wouldn't have to worry about it.)

This doesn't need to be a actual format change or anything, just a policy.

[cmp-nct]

Should definitely be part of it, disabling 1kb of data wouldn't even be useful imho. The model sizes are so much larger in proportion. Even on a raspi 1kb is nothing

[KerfuffleV2]

Your 1kb spare area idea sounds like one step in that direction!

Hmm, I don't think it will help with file format compatibility stuff for the most part. It just makes modifying the metadata of an existing file easier, because there's space for a new key, etc. Like if you wanted to insert a new key, right now there's no space so to do this all the data past the end of the metadata would have to be shifted forward. Having an unused chunk of space just means we can use some of that for new fields.

you'll likely find a significant portion of gguf models being not functional and it's impossible to tell that upfront before downloading it.

Hmm, do you meant they're in GGUFv1 format or just that they have other issues (like bugs that got fixed) which cause them to not function correctly?

Converting from GGUFv1 is actually pretty simple, if that's the issu. The other one is, unfortunately probably true in a lot of cases and converting with a version that has those fixes is basically the only way to deal with it.

[cmp-nct]

I've not tried again in the past 3 weeks but for example every single Falcon model I could find was not functional, minor tokenizer changes and stuff like that which rendered the files invalid.
I probably downloaded 300GB until I gave up and started fixing my anaconda environment to convert them myself to gguf. (Which also failed that moment until I manually fixed padding additions in the python script).
Not long after I had similar problems with llama-2 models, it was again some problem with tokens. Another issue came up with a tensor error.
Nothing of that matters if you convert the models yourself of course and keep the original HF backed up. (Well except for python versioning conflicts that can cause similar problems but that's out of our control)

It's a real problem when you look at it at a longterm perspective, like junk in the earth orbit that builds up.
I can imagine a lot of new people gave up in frustration already when the files they download only error out with messages they can not understand.
I'm not sure what the best solution would be, though at the very minimum the error messages upon loading need to be clear that it's not a invalid model file, it's a version X conflict.

[KerfuffleV2]

Yeah, unfortunately that problem is harder to solve because it's probably just that there were bugs in the conversion process, or maybe issues in the original model metadata which got fixed at some point.

Hmm, actually, I thought of a way the __SPARE field could possibly help in that scenario. There could be a tool that just rebuilds the metadata (includes vocab, etc) in a GGUF file while preserving the tensor data. Sort of like how the convert-llama-ggml-to-gguf.py script works with metadata supplied, if you're familiar with that. Then if you downloaded a GGUF file and know where the original HF metadata is, you could just clone the metadata and rebuild only the metadata section in the file.

Then if the new version of the metadata was longer than the original, you could use some of the __SPARE part. Of course, this would only work where the original GGUF file was new enough to even have __SPARE.

Now that #3981 is merged, there are actually a lot of interesting things that can be done. For example, creating a vocab-only GGUF file from a full GGUF, extracting tensors and metadata to a directory, etc.

though at the very minimum the error messages upon loading need to be clear that it's not a invalid model file, it's a version X conflict.

I'm pretty sure you get a reasonably clear error if you try to load a GGUF file of the wrong version. The error is a bit more arcane (something about bad magic) if you try to do something like load an old GGML file.

That's a different issue from what you were talking about with Falcon though, I think.

[cmp-nct]

Sounds like a good idea, updating and working with the gguf files to repair or modernize them.

Regarding the error: I've rarely had one that was useful, some tensor that was not found, some tokenization error, etc.
Any non developer is totally helpless with the typical errors I've seen.
I believe at one point, which should come sooner than later, breaking changes should incorporate a proper error message for the breaking.

ggml/llama.cpp have mostly been demos so far (hell .. the executables are even called 'examples' as if it was still a ggml example project where it originated from) but it's becoming more and more refined and useful for real usage.
So at a certain point it should be a doctrine that any breaking change, should at the minimum come with a great error message so people know what to do. Kind of a balance between development progress and user-friendlyness

[KerfuffleV2]

Regarding the error: I've rarely had one that was useful [...] Any non developer is totally helpless with the typical errors I've seen.

For the most part, with actual bugs or whatever there isn't too much a user can do to fix it. What is helpful is making sure developers can see those messages. They're (at least in the case of "good errors") are aimed at providing enough for developers to figure out what happened so the problem can be fixed.

breaking changes should incorporate a proper error message for the breaking.

As far as I know, changes that knowingly and deliberately break stuff do tend to have an announcement in the README and better errors.

I think you might be under the misapprehension that the kind of errors you're talking about are something someone deliberately broke, but that's probably not the case in most of these situations. llama.cpp is a big, complicated project and it has a lot of moving parts. It supports a bunch of different features, different model types, different hardware, etc. It's really hard to make changes without affecting other stuff in unpredictable ways.

Also, my impression (I have no authority to speak for the project or anything) is that GGML/llama.cpp aims to move fast even when there's a risk of breaking stuff. And I think it's that philosophy is why it's so competitive: it's not afraid to add new features, make changes that are believed to be better, etc and then fix the issues that crop up later.

There are other projects that try to track llama.cpp and provide a more stable experience, like koboldcpp is one example I know of. But you can't really have both the stability and the bleeding edge features.