Output shows <|endoftext|> tokens and

[UltraWelfare]

I'm trying two models converted to gguf using the GGUF-my-repo space
Model 1
Model 2

They both face the same issue where they have <|endoftext|> or <|im_end> tokens in their output and they start questioning and answering themselves.

I'm starting the llama-server like this : .\llama-server --model .\teuken-7b-instruct-commercial-v0.4-q6_k.gguf. Setting the context size (parameter -c) doesn't change the output...

Other models work correctly like Llama3.2 and Llama3.1.. I'm not sure what is up with these two specifically

[slaren]

This is most likely caused due to incorrect configuration of the model tokenizer. From a quick look at https://huggingface.co/openGPT-X/Teuken-7B-instruct-commercial-v0.4/blob/main/tokenizer_config.json it seems that this model does not have a chat_template, so llama.cpp does not know how to format the chat messages. There may be other issues such as missing a list of stop tokens.

[UltraWelfare]

Hm I see... I just looked at the output of the server

main: The chat template that comes with this model is not yet supported, falling back to chatml. This may cause the model to output suboptimal responses
main: chat template, built_in: 0, chat_example: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'

It says built_in:0 but it has an example.. Unless this is the chatml one.

Looking at the Meltemi model:

main: chat template, built_in: 1, chat_example: '<|system|>
You are a helpful assistant<|endoftext|>
<|user|>
Hello<|endoftext|>
<|assistant|>
Hi there<|endoftext|>
<|user|>
How are you?<|endoftext|>
<|assistant|>
'

It seems more correct... although still not quite sure why it doesn't work D:

[cristianadam]

See also the discussion I've started at #10539

llama.cpp doesn't actually support the Teuken models.