Inference speed on windows server 2022 which has Nvidia RTX A4000

[adeelhasan19]

i am doing mistral 7b openorca inference using llamacpp-python but its is taken lot of timing.How can i fix that

llama-cpp-python version is 0.2.11
Server Configuration:
1)Windows Server 2022 Standard
2) two Nvidia rtx A4000 gpu

[tk-master]

you sure it's using the gpu? it seems like it isn't.. so start there.. make sure cublas is enabled and you are offloading to gpu (n_gpu_layers)

[adeelhasan19]

@tk-master yes it is using gpu but as you can see it is giving some warning of pinned memory is that the issue?

[sadaisystems]

Got the same problem on a server with 2 A100. Llama model is initialized with main_gpu=0, tensor_split=None. With this settings,however, vram is evenly allocated from both GPUS. Which is unexpected. This also causes a slowdown, shown on the screenshots above (speeds, comparable to CPU inference when using GPU)

I've managed to solve this by introducing environment variable CUSA_VISIBLE_DEVICES="0". This helps, but limits the process to only 1 gpu.

Is this a bug, or what? In my case, I have llama-cop-python 2.11

[abetlen]

@mmagnesium any luck with the latest version?