Vulkan support on arm64 (Raspberry Pi eGPU)

[geerlingguy]

While testing some AMD GPUs on a Raspberry Pi (64-bit arm64 architecture), I ran into some issues getting llama.cpp to compile on that system, at least using the Docker image in the .devops directory, with support for Vulkan (I am testing with an AMD RX 6700 XT, as well as a Radeon Pro W7700, but will test with other GPUs if I can get one of these working).

See my testing so far, here: geerlingguy/raspberry-pi-pcie-devices#680 (comment)

It seems that the Vulkan support is geared towards x86 right now, and probably untested on arm64.

My question is: would there be interest in supporting multiple architectures? As GPUs are supported on more than x86 now (RISC-V is soon joining the party, Arm has been there a few years with Ampere machines), it would be nice to get this running smoothly with GPU acceleration where CUDA is not available (Nvidia cards work on Ampere and their own systems, but so far not on any other Arm architecture, leaving us with AMD as the reasonable alternative, maybe Intel Arc at some point).

[MathiasSchindler]

Further reading: https://www.jeffgeerling.com/blog/2024/llms-accelerated-egpu-on-raspberry-pi-5 and geerlingguy/ollama-benchmark#1

Thank you for documenting it, @geerlingguy

[AndreasKunar]

@geerlingguy - do you think your eGPU work could also run on a Snapdragon X PC (probably in WSL2), connected via USB4, with some software-tweaking?

[geerlingguy]

No clue on that, I haven't seen anyone working on that but it would be a bit complicated... it'd be nicer to have Windows on Arm support PCIe GPUs natively, but so far no drivers have been released.

[netrunnereve]

It's worth noting that Vulkan has had a lot of updates these couple days which should help your AMD cards. Not sure which commit you did your tests on but my Q4_0 inference speeds have more than doubled compared to last week.