performance much worse on windows

[v4lentin1879]

I'm using mistral-7b-openorca.Q5_K_S.gguf for a specific summarization task on macos. An inference takes me approx. 40-90s on my MacBook Pro 2019. Running the same code with the same data on the same device on windows, however, takes much longer (15 mins). I realized that windows is not letting me use the full computing power. This problem is also not related to my bootcamp setup since I have the same problem on a powerful windows computer of a friend of mine.
Any idea how one could fix this/what could be the problem?
Thank you very much in advance for your help!

[giladgd]

@v4lentin1879 On macOS, Metal is enabled by default, so you benefit from GPU utilization out-of-the-box.
On Windows, you have to run the download command to build llama.cpp from source to enable support for the GPU of your computer.

AFAIK 2019 Intel MacBooks have an AMD GPU, so you'll have to set custom environment variables before using the download command to configure the AMD GPU support of llama.cpp

[giladgd]

@v4lentin1879 If that's the case then I think it's on llama.cpp's end, so it might be a good idea to open a ticket about it there.
But before you do that, try to reproduce it with llama.cpp's main binary on both macOS and Windows.
If it doesn't reproduce with llama.cpp's main binary then let me know the details of the tests you've done so I can investigate it.

[v4lentin1879]

@giladgd You are right, the problem seems to be llama.cpp. When building it with cmake, the CPU-performance is much worse compared to if I build it with w64devkit.exe using make. Have you experienced this before? Do you know of any specific settings for cmake which could fix this?

[v4lentin1879]

@giladgd Is there a way to use w64devkit to compile llama.cpp? Or can I somehow paste my own, compiled llama.cpp in there?

[giladgd]

@v4lentin1879 I have no idea of how to use w64devkit.
If a cmake binary is found in the current PATH, node-llama-cpp will use it, so you can try make it be the case.

If you're using mingw or something like that, it should be sufficient to just try to run npx --no node-llama-cpp download inside of a mingw's bash to make it utilize the tools provided by mingw.
I'm not sure if that's what you're trying to achieve.

[v4lentin1879]

Yes, this seems to be working! Thanks a lot!