Add huggingface to gguf format conversion for a new model?

[rlucas7]

I am trying to generate a GGUF formatted Roberta style model. However, both the base Roberta model and a masked language model variant (*-mlm) raise errors when running from main on Roberta.

What I've tried:

(1) The Hf -> gguf convert your repo browser option which raises errors (see screenshots)

(2) The convert python module from main branch-after downloading a Roberta model from hugginface to my local machine. When I try to do the conversion to GGUF this raises the same error as the converter in (1).

I'd like to be able to convert a Roberta model into a GGUF format so that it can be used in the lembed sqlite extension which requires the model to be in the GGUF format locally.

Note: If there are other ways to enable the conversion I'm happy to try that-please let me know the preferred conversion method if my current approach isn't preferred.

From spelunking the transformers code I see that the architecture is ostensibly the same for XLMRoberta as for Roberta (ref https://github.com/huggingface/transformers/blob/bdb29ff9f3b8030772bd4be037d061f253c0e928/src/transformers/models/xlm_roberta/modeling_xlm_roberta.py#L152)

So given that xlm roberta model is supported I'd like to be try using the same conversion for a Roberta model, is there a way to achieve conversion for a roberta model using the xlm roberta code?

some related items:

PR 5423 seems to have added BERT model support

discussion 7712 shows how to generate gguf for other, supported, model types.

discussion 2948 is a, possibly outdated-tutorial from 2023 on conversion to gguf format.

PR 8205 adds some documentation for BERT support.

[slaren]

If the architecture is the same, you can try adding it to the annotation.
https://github.com/ggerganov/llama.cpp/blob/0cc63754b831d3a6c37bc5d721d12ce9540ffe76/convert_hf_to_gguf.py#L2624-L2625

[rlucas7]

So that works to complete the processing of the model conversion except that the tokenizer discrepancies are showing up now,

... stdout from previous layers elided ... 
INFO:hf-to-gguf:blk.11.ffn_up.bias,              torch.float32 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.11.ffn_down.weight,          torch.float32 --> F16, shape = {3072, 768}
INFO:hf-to-gguf:blk.11.ffn_down.bias,            torch.float32 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.layer_output_norm.weight, torch.float32 --> F32, shape = {768}
INFO:hf-to-gguf:blk.11.layer_output_norm.bias,   torch.float32 --> F32, shape = {768}
INFO:hf-to-gguf:Set meta model
INFO:hf-to-gguf:Set model parameters
INFO:hf-to-gguf:gguf: context length = 512
INFO:hf-to-gguf:gguf: embedding length = 768
INFO:hf-to-gguf:gguf: feed forward length = 3072
INFO:hf-to-gguf:gguf: head count = 12
INFO:hf-to-gguf:gguf: layer norm epsilon = 1e-05
INFO:hf-to-gguf:gguf: file type = 1
INFO:hf-to-gguf:Set model tokenizer
Traceback (most recent call last):
  File "/Users/rlucas/gguf_make/llama.cpp/convert_hf_to_gguf.py", line 4439, in <module>
    main()
  File "/Users/rlucas/gguf_make/llama.cpp/convert_hf_to_gguf.py", line 4433, in main
    model_instance.write()
  File "/Users/rlucas/gguf_make/llama.cpp/convert_hf_to_gguf.py", line 435, in write
    self.prepare_metadata(vocab_only=False)
  File "/Users/rlucas/gguf_make/llama.cpp/convert_hf_to_gguf.py", line 428, in prepare_metadata
    self.set_vocab()
  File "/Users/rlucas/gguf_make/llama.cpp/convert_hf_to_gguf.py", line 2648, in set_vocab
    raise FileNotFoundError(f"File not found: {tokenizer_path}")
FileNotFoundError: File not found: codebert-hf/sentencepiece.bpe.model

The contents of the model directory locally are:

Lucass-MacBook:gguf_make rlucas$ ls codebert-hf/
README.md		flax_model.msgpack	pytorch_model.bin	special_tokens_map.json	tokenizer_config.json
config.json		merges.txt		rust_model.ot		tf_model.h5		vocab.json

so I think the tokenizer needs to take a different codepath than the one it is choosing here

A couple of the other models in this module look like they're working off json files, e.g. here is one

is this jina model tokenizer code a good one to crib from for the branched code for this Roberta tokenizer?

FWIW the tokenizer structures are (very) similar:

>>> from transformers import AutoTokenizer, AutoModelForMaskedLM
xlm_tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-base')
>>> xlm_tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-base')
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/codebert-base")
>>> tokenizer
RobertaTokenizerFast(name_or_path='microsoft/codebert-base', vocab_size=50265, model_max_length=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'mask_token': '<mask>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
	0: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
	1: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
	2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
	3: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
	50264: AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False, special=True),
}
>>> xlm_tokenizer
XLMRobertaTokenizerFast(name_or_path='xlm-roberta-base', vocab_size=250002, model_max_length=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'mask_token': '<mask>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
	0: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	1: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	3: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	250001: AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False, special=True),
}

[slaren]

That's as far as my knowledge of the converter goes. I suggest looking if there are other supported models that use the same tokenizer type and use that as a template.

[rlucas7]

That's as far as my knowledge of the converter goes. I suggest looking if there are other supported models that use the same tokenizer type and use that as a template.

Thanks @slaren I'm looking at this a bit closer now for the tokenizer, I'm unclear on the interaction between the two modules, convert_hf_to_gguf.py and convert_hf_to_gguf_update.py is this documented anywhere in an issue or pr, or can you provide some context on this part?

FWIW, once I get this working I'd be happy to open a PR for the full changes to get it working-perhaps as a new class with it's own @Model.register() etc.

[rlucas7]

or if anyone else involved in the project knows, maybe @ggerganov can provide some info?

[ggerganov]

I'm unclear on the interaction between the two modules, convert_hf_to_gguf.py and convert_hf_to_gguf_update.py is this documented anywhere in an issue or pr, or can you provide some context on this part?

See if #6920 helps.