how make llamacpp inference fast as possible on SNAPDRAGON732G termux

[akac97]

i want to run phi-3.5-mini-instruct gguf fast as possible on my phone what flags can i use llamacpp ./llama-cli if possible i want use q3_k_l or q3_k_m or q4_k_m

[ExtReMLapin]

For arm you can use the Q4_0_4_4 quants but does Kryo 470 counts as ARM ?

[akac97]

i think it counts as arm But i think about thinks like batch processing and something like that. (cli flags)

[ExtReMLapin]

You could use QNN backend from there https://github.com/chraac/llama.cpp/tree/dev-refactoring

It's originated from this PR, but it's a mess, read the comments #6869