[Question] Why awq-int4-flux.1-t5xxl.safetensors causes CUDA out of memory, while t5xxl_fp8_e4m3fn_scaled.safetensors does work without any issues ?

[Greatz08]

I was curious to test awq-int4-flux.1-t5xxl.safetensors to save more VRAM as i have RTX 4060 - 8 GB VRAM only, but it caused CUDA out of memory, but t5xxl_fp8_e4m3fn_scaled.safetensors does work in my system. I want to understand the reason behind this issue, so if anyone has any idea why it happened then do let me know. AWQ quant might be issue which i can guess but cant confirm so asking here :-)

[nitinmukesh]

@Greatz08
I am using the quantized text_encoder_2 ( t5xxl_fp8_e4m3fn_scaled.safetensors) and don't face any CUDA OOM issue.
I have 4060 laptop GPU + 16 GB RAM and it only needs 4-6 GB VRAM.

How are you using? Comfy or example script/Gradio?

I responded here too
#523 (reply in thread)

[Greatz08]

@nitinmukesh As i said in my main message, i am also able to run this -> t5xxl_fp8_e4m3fn_scaled.safetensors without any issues. This post was mainly about problem in running -> awq-int4-flux.1-t5xxl.safetensors , which i thought could save some more ram or vram of my device (this awq-int4-flux.1-t5xxl.safetensors was developed by nunchaku teamm so i thought to ask here).

[lmxyy]

Yeah, the main issue may be that we move the entire 4-bit AWQ model to the GPU when running it. The FP8 model may have some offloading, which we have not implemented for now.