CUDA Memory Requirements

[pman07]

Apologies if this is covered somewhere and I missed it, what do I need to change to get this running with 4 GB of VRAM? I've tried editing the tts_memory_threshold but I'm still running out of CUDA VRAM while "Warming models". Any tips? Thanks!

[kristiankielhofner]

What hardware are you using with 4GB VRAM?

[pman07]

I had an old GTX 1050 Ti laying around, was trying that out. Is that too low end?

[kristiankielhofner]

I just pushed a change to the wisng branch. I had to guess at the actual available reported CUDA memory but it should be close. I'm showing all of the Whisper models using 3.245GB of VRAM here in int8.

wisng will likely be merged to main this week so we'd really appreciate any testing you can do with it!

[pman07]

Cool, thanks for the quick response! I got wisng running after disabling TTS. Hopefully will get around to tinkering with it tomorrow.

[kristiankielhofner]

Great, let us know how it goes. The 1050 Ti is officially the lowest supported card anyone has ever run WIS on, interested to see how it performs.

[pman07]

Working pretty well, I think! I noticed the commit to revert to default willow response right about the time I was running into that issue. I'm seeing inference times of just under a second and still feels pretty quick for most uses so far. Thanks again for your awesome help! Is there anything you'd like me to test?

[kristiankielhofner]

Great! If you'd like to experiment with faster response times you can try one of the smaller models (such as small or base). A couple of options:

1. Copy settings.py to custom_settings.py and override the default model there.

2. Select the model on a per-request basis by adding the model param to the URI:

/api/willow?model=small

Theoretically the accuracy will be lower but for most voice assistant tasks, etc our default of medium probably leans a little heavy to the accuracy side - but I'd be very interested to hear what your experience is like! A couple of other general tips for reducing latency:

1. If it's acceptable for you and your environment you can use HTTP on port 19001 (by default) from Willow to WIS. This shaves some time off the initial connection to WIS on each session - we're looking at utilizing a persistent websocket connection from Willow to WIS to eliminate this. Additionally, we've identified a few performance issues with some of the TLS cipher suites on the ESP that we're investigating.

2. Speaking of which, you mentioned reverting to the default willow response - that should have only impacted you if you were using Willow + WIS (wisng) with Home Assistant pre 2023.5. If you use 2023.5 or later we maintain a persistent websocket connection to Home Assistant and in addition to being more capable it's also faster because of connection reuse.

3. If you speak relatively quickly (like I do) you can go to the Advanced Options section of the Willow Configuration menu and reduce your VAD timeout. The default of 300ms is also pretty conservative (I use 100ms).

I'll be adding formal benchmark support later this week and if you could run them on the 1050ti we'd like to add the results to our benchmarks.

[ceinstaller]

pman, can you list the settings you used for your 1050ti? I'm using the same card but can't seem to find the right combination of settings to get the server to start. Always hit the 'CUDA failed with error out of memory'.

Thanks in advance,

~Mark

[pman07]

@ceinstaller Sorry for the delay, I was out of town. Are you using the 'wisng' branch? I don't believe I had to do anything special to get that branch running with the 1050ti.

[ceinstaller]

Hey pman, I now owe you an apology for a late reply. I didn't get it to work, but I was using the instructions from the main page. (I assume that means I wasn't using the wsing branch.) I ended up pulling a 1060 out of another machine and using that and all is well!