What models run inside Orange Pi 5 32GB?

[paulocoutinhox]

Hi,

What models run inside Orange Pi 5 32GB? 65B?

Reference:
http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/details/Orange-Pi-5-32GB.html

Thanks.

[ianscrivener]

Here are 2 small models to choose from;

• https://huggingface.co/Aryanne/Mamba-gpt-3B-v4-ggml-and-gguf/resolve/main/q4_0-gguf-mamba-gpt-3B_v4.gguf
• https://huggingface.co/Aryanne/Bean-3B-ggml-and-gguf/resolve/main/bean-3B-ggufv2-q4_0.gguf

Run with the latest llama.cpp release.

Please share your progress... keen to know how you go.

[KerfuffleV2]

With absolutely nothing else running, you could actually probably load a Q2_K quantized 70B. You'd probably have to set the context low. Obviously 70B and 65B is going to be very, very slow. For practical use, probably 7-13B, 30B if you're patient. There are some really good 7B Mistral models, that's probably the best option.