Does llama.cpp use the correct tokens for the system prompt / inst for the mistral models?

[PhilArmstrong]

The documentation for the mistral instruct models says to prompt them like so in order to get the best results out of them:

<s>[INST] What is your favourite condiment? [/INST]"
"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s>
[INST] Do you have mayonnaise recipes? [/INST]

i.e., you wrap the "system" prompt in the <s> & </s> tokens, and put the instruction you want a response to inside "[INST]" & "[/INST]" text strings.

Does lama.cpp do this automatically? As in, if I define a prompt with "-p" on the command line, will main add the correct tokens, or do I have to include them myself?

[PhilArmstrong]

Addendum: it seems I should be defining the pre & post prompt text with --in-prefix and --out-prefix ?

Like so:

./build-cpu/bin/main -t 16 -m /mnt/models/mixtral-8x7b-instruct-v0.1.Q3_K_M.gguf --in-prefix "[INST]" --in-suffix "[/INST]" --color -i --prompt "" -ins -c 0

It would be nice if that text maybe wasn’t printed to the output, but I guess main is just a proof of concept code.

[eskeletor97]

I have a question related to this. Is [INST] and [/INST] being broken down into their individual parts like this is an intended behavior, or is it supposed to be treated as one token?

[1703092227] embd_inp.size(): 771, n_consumed: 0
[1703092227] eval: [ '':1, ' [':733, 'INST':16289, ']':28793, ' Contin':13718, 'ue':441, ' writing':3653, ' this':456 ]
[1703092228] n_past = 8

As you can see above, [INST] is not getting processed as a whole.

[eminence]

This is correct and expected. See this section of the mistral docs where it says:

Note that <s> and </s> are special tokens for beginning of string (BOS) and end of string (EOS) while [INST] and [/INST] are regular strings.

[Jeximo]

i.e., you wrap the "system" prompt in the & tokens, and put the instruction you want a response to inside "[INST]" & "[/INST]" text strings. Does lama.cpp do this automatically?

main adds the a bos token <s> for Mistral Instruct.

Here's an example:

-i --penalize-nl --temp 0 --in-prefix "[INST] " --in-suffix "[/INST]" -p "[INST] What's 5+5?[/INST] 5+5=10</s>"