Mistral-7B Support

[bgyss]

https://mistral.ai/news/announcing-mistral-7b/

https://huggingface.co/mistralai/Mistral-7B-v0.1
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

Highlights from the announcement:

Mistral 7B is a 7.3B parameter model that:

- Outperforms Llama 2 13B on all benchmarks
- Outperforms Llama 1 34B on many benchmarks
- Approaches CodeLlama 7B performance on code, while remaining good at English tasks
- Uses Grouped-query attention (GQA) for faster inference
- Uses Sliding Window Attention (SWA) to handle longer sequences at smaller cost

We’re releasing Mistral 7B under the Apache 2.0 license, it can be used without restrictions.

[bgyss]

Didn't see the pull request already created.

[psugihara]

I've been able to play with it and it seems to work. Are you seeing problems?

Here are GGUFs from TheBloke https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF

[psugihara]

ah sorry, just saw your comment

[jooray]

How can I convert Mistral model to GGUF? convert.py does not work for me. I do not want to use the one from TheBloke (I am fine tuning).

[slaren]

convert.py should work with Mistral, what problem do you see?

[jooray]

Loading vocab file '/home/user/mistral-up/tokenizer.model', type 'spm'
Traceback (most recent call last):
File "/home/user/llama.cpp/convert.py", line 1193, in
main()
File "/home/user/llama.cpp/convert.py", line 1175, in main
vocab = load_vocab(vocab_dir, args.vocabtype)
File "/home/user/llama.cpp/convert.py", line 1086, in load_vocab
return SentencePieceVocab(path, added_tokens_path if added_tokens_path.exists() else None)
File "/home/user/llama.cpp/convert.py", line 372, in init
raise Exception(f"Expected added token IDs to be sequential and start at {len(added_tokens)}; got {actual_ids}")
Exception: Expected added token IDs to be sequential and start at 6; got [0, 1, 2, 32000, 32001, 32002]

[slaren]

I think there was an issue with the added_tokens.json file in the HF version of the model, removing this file should fix it. It looks like they have already removed it from the repository in HF as well, so downloading the latest version of the model should also work.

[jooray]

This did not help. I am actually trying to convert this finetuned model that has added tokens:

https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca/blob/main/added_tokens.json

If I remove added_tokens.json, it does not work:

Writing /home/user/jrj-mistral.gguf, format 1
Traceback (most recent call last):
  File "/home/user/llama.cpp/convert.py", line 1193, in <module>
    main()
  File "/home/user/llama.cpp/convert.py", line 1188, in main
    OutputFile.write_all(outfile, ftype, params, model, vocab, special_vocab, concurrency = args.concurrency)
  File "/home/user/llama.cpp/convert.py", line 907, in write_all
    check_vocab_size(params, vocab)
  File "/home/user/llama.cpp/convert.py", line 802, in check_vocab_size
    raise Exception(msg)
Exception: Vocab size mismatch (model has 32003, but /home/user/mistral-up/tokenizer.model has 32000).  Most likely you are missing added_tokens.json (should be in /home/user/mistral-up).

[slaren]

In that case, I suggest opening a new issue. There were some issues with this model that I think should have been fixed in #3455, @goerch may know more about it.

[goerch]

Don't we have some reports about Mistral-7B-OpenOrca already? I'm trying to understand #3502 first before continuing.

[paulocoutinhox]

Hi,

It was solved? How i can use this model with llama.cpp?

Thanks.

[Galunid]

It should work out of the box. Download .gguf model and run it like you'd any other. I use it like this, change -ngl and -m
to fit your needs:
./main -ngl 50 --log-disable -i -ins -m ~/llama.cpp/models/dolphin-2.1-mistral-7b.Q4_K_M.gguf -c 4096 -b 1024 --temp 0.7 --top-k 40 --top-p 0.95 --repeat_penalty 1.1 --no-penalize-nl -n -1 --color -r -e

[paulocoutinhox]

Im trying, can you help me understand what im doing wrong here?
#3724

[paulocoutinhox]

How i can translate all your params to c++ code here?

// prompt and model
const std::string prompt = "Write a simple Python code";
const std::string modelPath = "codellama-7b.Q2_K.3.gguf";

// params
gpt_params params;
params.model = modelPath;
params.prompt = "### Instruction:" + prompt + "\n\n### Response:";
params.numa = 32;

// init LLM
llama_backend_init(params.numa);

// initialize the model
llama_model_params modelParams = llama_model_default_params();
llama_model *model = llama_load_model_from_file(params.model.c_str(), modelParams);

Current error:

gguf_init_from_file: tensor 'token_embd.weight' number of elements (131072000) is not a multiple of block size (0)
error loading model: llama_model_loader: failed to load model from /Users/paulo/Downloads/mistral-7b-v0.1.Q2_K.gguf

[jooray]

Remove the first tokens (numbered 0, 1, ..) from additional_tokens.json