Gradio Interface, Docker and Runpod Template

[alisson-anjos]

Congratulations on the optimization work, it really was incredible, it couldn't get any better than that, I forked your repository and implemented a gradio interface, a docker image and a template in runpod for anyone who wants to test it, thanks for the optimization work.

https://github.com/alisson-anjos/YuE-exllamav2-UI

[turboderp]

For completeness, the EXL2 path for stage 2 relies on a cache size rather than batch size. It should probably be documented a bit better, but essentially you set the --stage2_cache_size to however many tokens you can fit (default is 8192). The actual batch size is dynamically chosen during inference.

[jrked]

Congratulations on the optimization work, it really was incredible, it couldn't get any better than that, I forked your repository and implemented a gradio interface, a docker image and a template in runpod for anyone who wants to test it, thanks for the optimization work.

https://github.com/alisson-anjos/YuE-exllamav2-UI

how can i run this in google colab?

[alisson-anjos]

For completeness, the EXL2 path for stage 2 relies on a cache size rather than batch size. It should probably be documented a bit better, but essentially you set the --stage2_cache_size to however many tokens you can fit (default is 8192). The actual batch size is dynamically chosen during inference.

Yes, I only realized this a short time ago, I just removed the field from the interface, Tky.

[alisson-anjos]

Congratulations on the optimization work, it really was incredible, it couldn't get any better than that, I forked your repository and implemented a gradio interface, a docker image and a template in runpod for anyone who wants to test it, thanks for the optimization work.
https://github.com/alisson-anjos/YuE-exllamav2-UI

how can i run this in google colab?

so as long as you install all the dependencies and configure the same structure that is used in terms of directory for models, outputs .... I believe it works, but of course you would have to clone the repository and then install the dependencies and configure the paths within the interface.py file that is in the src folder. You can also search for how to run docker on google colab, it seems that there is an application called https://github.com/indigo-dc/udocker and you can try to see how to use it and then just follow the tutorial in the repository that teaches you how to run with docker.

[brcisna]

@alisson-anjos

You need to make slight adjustment to your repo to introduce something simliar to the original YuE-exllamv2 to have the requirements-rocm.txt as an alternative for AMD GPUs using ROCm HIP
Just for kicks,,on my Debian 13 Trixie with an AMD Radeo Pro W6600...I pulled down the docker file but in the end when the docker tried to run,,"no suitable gpu driver found"
The GUI makes this so much nicer for novices like me to use YuE-exllamav2

Thanks

[alisson-anjos]

@alisson-anjos

You need to make slight adjustment to your repo to introduce something simliar to the original YuE-exllamv2 to have the requirements-rocm.txt as an alternative for AMD GPUs using ROCm HIP Just for kicks,,on my Debian 13 Trixie with an AMD Radeo Pro W6600...I pulled down the docker file but in the end when the docker tried to run,,"no suitable gpu driver found" The GUI makes this so much nicer for novices like me to use YuE-exllamav2

Thanks

I will find some time and make this adjustment.