Issues with 0.6B model using either EXL2 or EXL2ASYNC

[zeropointnine]

I've been using the 1.0B model with the EXL2 backend to great effect, but am having issues with the 0.6B model.

Using Backend.EXL2 with the 0.6B model throws an error: RuntimeError: torch.cat(): expected a non-empty list of Tensors

Using Backend.EXL2ASYNC with the 0.6B model does work, but VRAM usage spills over into shared memory significantly for some reason, making it impractical.

My ModelConfig settings:
MODEL_CONFIG = outetts.ModelConfig( model_path=r"C:\Users\me\.cache\huggingface\hub\models--OuteAI--OuteTTS-1.0-0.6B\snapshots\12345", interface_version=outetts.InterfaceVersion.V3, backend=outetts.Backend.EXL2, # or EXL2ASYNC device="cuda", dtype=torch.bfloat16 )

Running Windows 11, RTX 3080Ti, latest version of the oute lib, latest version of the exllama2 lib.

[edwko]

Are you using 0.3.0 EXL2 release? It's required to run the Qwen3 architecture.

For VRAM, since EXL2 async is designed for batched generation, it uses a larger cache to maximize GPU usage. You might try lowering the exl2_cache_seq_multiply. By default, it's set to 16, meaning 8192 * 16 token cache is used.

In your MODEL_CONFIG, you can lower this setting to reduce VRAM usage. It should be set equal to the max_batch_size used in GenerationConfig.

[zeropointnine]

Yes, I am using exllamav2 0.3.0.

Adjusting exl2_cache_seq_multiply and max_batch_size resolved my VRAM issues when using EXL2ASYNC. Thanks!