[Feat]: llama.cpp vs mnn-llm speed

[bi4key]

llama.cpp vs mnn-llm speed

---

https://www.reddit.com/r/LocalLLaMA/s/tDh1l0cvVe

.

https://github.com/alibaba/MNN/blob/master/project/android/apps/MnnLlmApp/README.md

.

https://github.com/alibaba/MNN

[a-ghorbani]

Interesting. The multimodality aspect sounds great, but the LLM inference speed doesn’t seem to be significantly different on phones, from what folks report:

• https://www.reddit.com/r/LocalLLaMA/comments/1ia7v0x/comment/m9akra7/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
• https://www.reddit.com/r/LocalLLaMA/comments/1ia7v0x/comment/m9a9i0l/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[Mcxiaocaibug]

I think there's still a big difference

[Mcxiaocaibug]

pocketpal 5tokens/s
mnn 50token/s
DeepSeek r1 1.5B
There is a big difference between them and hopefully it can be improved
@a-ghorbani

[HiWill]

MNN beat llama.cpp

pocketpal 5tokens/s mnn 50token/s DeepSeek r1 1.5B There is a big difference between them and hopefully it can be improved @a-ghorbani

[HiWill]

llama.cpp 易用性要好，希望能提升性能

[Disonantemus]

Speed test:

PocketPal vs MNN Chat

Prompt:

"Why the sky is blue?"

Results

t/s	model	q
10.06	Qwen2.5-3B-instruct-MNN	4-bit
6.02	Qwen2.5-3B-instruct	iq4_xs
5.84	Qwen2.5-3B-instruct	q2_k
5.32	Qwen2.5-3B-instruct	q4_0
5.01	Qwen2.5-3B-instruct	q8_0
4.16	Qwen2.5-3B-instruct	iq2_m

PocketPal Models: bartowski/Qwen2.5-3B-Instruct-GGUF

MNN Models

I was expecting faster "t/s" when using models with lower q, but iq4_xs was the fastest, why?

---

Device: Samsung SM-G780G
OS: Android 13
App Version: 1.8.5
8 cores
7.4GB

[iAdanos]

As of now, mnn is still much (10x) faster on android. Worth researching.

[bi4key]

MNN vs PocketPal

Pixel 6a

Unsloth model

MNN in 3x faster
But I don't know what quant they use, I use q4

---

MNN

---

PocketPal

[bi4key]

New version MNN is even faster that previous

https://github.com/alibaba/MNN/blob/master/apps/Android/MnnLlmChat/README.md#version-040

[eozalp]

Mnn is faster. I don't have statistics. Actually no need. Trust me.

But... Everything is not about speed. Ecosystem matter.