How do you format with the prompt template in the server UI?

[RahulVivekNair]

The mistral finetunes are repeating unless I use the chatml format but I don't understand how to format it properly in the server UI. Cant find much documentation about the parameters in the ui like {{name}}: {{message}} , {{prompt}},{{history}},{{char}}: either. Any insight into this?

[jooray]

Probably not a good direct answer, but I had success with using llama-cpp-python server module.

CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python[server]

export MODEL=“PATH_TO_YOUR_MODEL.gguf”
python3 -m llama_cpp.server --port 4891 --chat_format chatml --n_gpu_layers -1 --use_mmap 1 --n_ctx 4096 --n_threads 8

And then you get openAI-like API on port 4891 that will correctly format your prompt as chatml

[francisco-lafe]

This is very much needed, I can't find any info either.

[mounta11n]

i would suggest to open an issue regarding this. especially because {{char}} is an extremly confusing and misleading wording, since char is used a lot in C code. and aside of the this, as you mentioned @RahulVivekNair, one cant find good documentation. even if you use the --verbose flag in server command, the output is not clarifying in which way exactly the model received the request and what the response looks like exactly.