Whisper CPP on Jetson Nano

[keshara2032]

Hi everyone!

I am trying to run whisper.cpp on a Jetson Nano for a real-time speech recognition task. I can run the stream method with the tiny model, but the latency is too high. I tried the CuBLAS instructions, but I could not get it to work (maybe my bad or GPU incompatibility) I would appreciate it if you guys could give me a tip or some advice.

Thanks!

[bobqianic]

I tried the CuBLAS instructions, but I could not get it to work (maybe my bad or GPU incompatibility)

Could you please offer a more detailed explanation of this problem? (eg. console output)

[keshara2032]

Yes of course! Sorry!

My Setup:
Jetson Nano
Jetpack : 4.6.1
Cuda : 10.2

I first started compiling with WHISPER_CUBLAS=1 make stream -j and got the following issue.

Then I came across this issue #876 and was able to fix it using the available gpu-arch

Then I got this issue.

I also found this issue #1018.

Despite following their suggestions, It did not compile. The error I got was '__builtin__assume' undefined.

However, in #1018, the commit mentioned #b948361 works with the changes they mention.

Hope this is helpful.
Thanks!

[bobqianic]

I suspect you're facing a problem similar to this one: ggml-org/llama.cpp#3131.

There are a couple of solutions. You could either upgrade CUDA to version 11.1 or higher, or I can tweak the .cu file for you to ensure compatibility with older CUDA versions, much like what was done in this pull request: ggml-org/llama.cpp#3132.

[keshara2032]

Thanks for your response! Unfortunately due to a hard limitation, I cannot upgrade CUDA beyond 10.2. It will be great if you could tweak the .cu files as you said. However, the performance on the Jetson is not good as I thought and the latency is quite high. I am not sure if implementing streaming speech recognition on jetson is a good idea.

[DVS70]

I suspect you're facing a problem similar to this one: ggml-org/llama.cpp#3131.

There are a couple of solutions. You could either upgrade CUDA to version 11.1 or higher, or I can tweak the .cu file for you to ensure compatibility with older CUDA versions, much like what was done in this pull request: ggml-org/llama.cpp#3132.

I would also appreciate that tweak. I'm having same issue with 10.2 cuda :(

[mazz]

@keshara2032 Would your port work on an Orin Nano?