OpenBLAS and CUBLAS

[aneeshjoy]

I have just have 6GB NVIDIA GPU. So most of the time I will be offloading some of the model layers to GPU.

Does it make sense to compile with both LLAMA_OPENBLAS=1 and LLAMA_CUBLAS=1 enabled?

Will that give any overall performance improvement?

[SlyEcho]

If a layer is not loaded to the GPU, it will still use cuBLAS, only that it needs to copy the data to the device before calculation.

[aneeshjoy]

Got it. Thanks