privateGPT is very slow.

[lsmpascal]

Hi. I installed privateGPT with Mistral 7b on some powerfull (and expensive) servers proposed by Vultr.
I tested on :

• Optimized Cloud : 16 vCPU, 32 GB RAM, 300 GB NVMe, 8.00 TB Transfer
• Bare metal : Intel E-2388G / 8/16@3.2 GHz / 128 GB RAM
• Cloud GPU : A16 - 1 GPU / GPU : 16 GB / 6 vCPUs / 64 GB RAM

All are >300$/month, but the perf are very poor (words per second < 4), far worse than the same mistral 7b installed on my mac Intel 2020 with Jan.
So I do not really understand ...
Is there settings I missed ?
Thank you.

[TobiasJu]

Did you install it with CUDA enabled? You should seee BLAS=1 in the cmd when its launching. GPU usage makes it 10x faster.

[lsmpascal]

I don't really know.
By the way I changed my mind a little bit and gave ollama a try.

I made this :
curl https://ollama.ai/install.sh | sh
then
ollama run mistral:7b-instruct-v0.2-fp16
The perf are still terrible even of I have been told that ollama was GPU friendly.

I think that cuda is installed on the machine :
When I do :

# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

But I can't find BLAS=1 anywhere...