Extract model's metadata without loading tensors

[matiaslin]

Background:

• Model’s metadata: type, architecture, ftype, name, vocab, GGUF metadata, and hyperparameter.
• When loading a model from file, llama_model_load is being used, which couples the loading of hyperparameters and tensors.

Question

• Is there a way to extract a model’s metadata without loading its tensors?

[slaren]

If you set vocab_only on llama_model_params, the tensor data will not be loaded, but the llama_meta functions should still work. You could also use the gguf functions from ggml, or from a different library such as gguf-py.

[matiaslin]

Thank you @slaren for the quick reply. The parameter vocab_only does indeed load some useful parameters without loading the tensors.

However, there are other variables within a model’s hyperparameter that would be useful to extract as well, such as n_layer. If we enable vocab_only, then llm_load_hparams would return before being able to set this variable.

I propose to complement the llama_model_params with a boolean parameter (e.g. skip_load_tensors) that will ensure the aforementioned hyperparameters to be set while skipping the loading of tensors, as its name suggests.

[slaren]

You can still obtain the number of layers by reading the <arch>.block_count metadata. It would be good to have a better understanding of your use case before adding more parameters.

[WilliamTambellini]

Tks @slaren
Our usecase is just the one from the title: getting all/most metadata (incl nlayers, ...) without loading any data (vocab, tensors, ...).
Perhaps a bool param 'metadata_only' to llama_model_params would be better?
Yours
W.

[christianazinn]

For some use cases you can also convert a model from safetensors/pytorch with the --vocab-only flag, which writes exclusively the metadata to an output file with no tensors at all. But this wouldn't work if you're downloading pre-converted GGUFs - I'm not sure what the use case is so just putting this out there.