How is tokenizer's add_prefix_space handled in llama.cpp?

[anhnami]

I'm debugging certain performance issue and I found that HF tokenizer adds prefix space by default. I'm wondering if llama.cpp does the same since it's not obvious in the code.

[dranger003]

I think what you are looking for is here?
https://github.com/ggerganov/llama.cpp/blob/ad3a0505e3b6cd777259ee35e61d428357ffc565/llama.cpp#L10912-L10923

[anhnami]

Thank you.