Best way to apply chat templates locally

[bwilkie]

Hi Everyone.

Im sure this is a silly question but Ive been at it for hours not. I think im just not getting something obvious.

So each model will have a prefferd chat template and EOS/BOS token. If running models online you can use HF apply_chat_template.

I found that when using llama_cpp locally I can get the metadata and the jinja template from the LLM_Model with;

(

metadata = LLM_Model.metadata

chat_template = metadata.get('tokenizer.chat_template', None)

)

Is this a good method?

How do other people pull and apply chat templates locally for various models?

Thanks!

[rrrusst]

If I remember right, llama-cpp-python tries to guess the chat template based on the metadata available in the loaded model.

So for example:
llm = Llama(model_path = llm_path, n_ctx = int(ct_size), chat_format = None)
will let it auto select what it thinks is the correct chat template, and (correct me if I'm wrong) falling back on the llama-2 chat template if it does not have that chat template or if the LLM metadata does not contain the required info.

To manually set the chat template, using the "mistral-instruct" template found in llama-cpp-python:
llm = Llama(model_path = llm_path, n_ctx = int(ct_size), chat_format = mistral-instruct)

To get the full list of llama-cpp-python's built-in chat templates, try the following:
from llama_cpp.llama_chat_format import LlamaChatCompletionHandlerRegistry
print(sorted(list(LlamaChatCompletionHandlerRegistry._chat_handlers.keys())))