[BUG] Runtime error when trying to load Qwen3 32B

[umar-mq]

OS

Windows

GPU Library

CUDA 12.x

Python version

3.12

Pytorch version

2.6.0+cu124

Model

https://huggingface.co/CAPsMANyo/Qwen3-32B_exl2/tree/4.25

Describe the bug

When trying to load the new Qwen3 32B model, it loads then immediately crashes with the given error.

Reproduction steps

Just by running start.sh

Expected behavior

It was expected to load successfully

Logs

Activating venv
pip 24.3.1 from /home/thomas/tabbyAPI_NEW/venv/lib/python3.12/site-packages/pip (python 3.12)
Loaded your saved preferences from `start_options.json`
Starting TabbyAPI...
2025-04-30 13:18:44.754 INFO:     ExllamaV2 version: 0.2.8
2025-04-30 13:18:44.786 WARNING:  Disabling authentication makes your instance vulnerable. Set the `disable_auth` flag to False in config.yml if you want to share
this instance with others.
2025-04-30 13:18:44.787 INFO:     Generation logging is enabled for: prompts
2025-04-30 13:18:44.817 WARNING:  Draft model is disabled because a model name wasn't provided. Please check your config.yml!
2025-04-30 13:18:44.818 WARNING:  An unsupported GPU is found in this configuration. Switching to compatibility mode.
2025-04-30 13:18:44.818 WARNING:  This disables parallel batching and features that rely on it (ex. CFG).
2025-04-30 13:18:44.818 WARNING:  To disable compatability mode, all GPUs must be ampere (30 series) or newer. AMD GPUs are not supported.
2025-04-30 13:18:44.819 INFO:     Attempting to load a prompt template if present.
2025-04-30 13:18:44.820 WARNING:  TemplateLoadError: Model JSON path "/home/thomas/text-generation-webui/models/Qwen3-32B-exl/chat_template.json" not found.
2025-04-30 13:18:44.842 INFO:     Using template "from_tokenizer_config" for chat completions.
2025-04-30 13:18:45.388 INFO:     Loading model: /home/thomas/text-generation-webui/models/Qwen3-32B-exl
2025-04-30 13:18:45.388 INFO:     Loading with tensor parallel
Loading model modules ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 131/131 0:00:00
Traceback (most recent call last):
  File "/home/thomas/tabbyAPI_NEW/start.py", line 291, in <module>
    entrypoint(args, parser)
  File "/home/thomas/tabbyAPI_NEW/main.py", line 166, in entrypoint
    asyncio.run(entrypoint_async())
  File "/home/thomas/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/home/thomas/tabbyAPI_NEW/main.py", line 71, in entrypoint_async
    await model.load_model(
  File "/home/thomas/tabbyAPI_NEW/common/model.py", line 112, in load_model
    async for _ in load_model_gen(model_path, **kwargs):
  File "/home/thomas/tabbyAPI_NEW/common/model.py", line 90, in load_model_gen
    async for module, modules in load_status:
  File "/home/thomas/tabbyAPI_NEW/backends/exllamav2/model.py", line 570, in load_gen
    async for value in iterate_in_threadpool(model_load_generator):
  File "/home/thomas/tabbyAPI_NEW/common/concurrency.py", line 30, in iterate_in_threadpool
    yield await asyncio.to_thread(gen_next, generator)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI_NEW/common/concurrency.py", line 20, in gen_next
    return next(generator)
           ^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI_NEW/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 57, in generator_context
    response = gen.send(request)
               ^^^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI_NEW/backends/exllamav2/model.py", line 731, in load_model_sync
    self.model.forward(input_ids, cache=self.cache, preprocess_only=True)
  File "/home/thomas/tabbyAPI_NEW/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI_NEW/venv/lib/python3.12/site-packages/exllamav2/model.py", line 900, in forward
    r = self.forward_chunk(
        ^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI_NEW/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^

Additional context

No response

Acknowledgements

• I have looked for similar issues before submitting this one.
• I understand the developers of this program are human, and I will ask my questions politely.
• I understand that the developers have lives and my issue will be answered when possible.

[Thireus]

Your logs show ExllamaV2 version: 0.2.8. Partial support for Qwen3 was added to the dev branch of 0.2.9 yesterday - a344009.

[pivtienduc]

Your logs show ExllamaV2 version: 0.2.8. Partial support for Qwen3 was added to the dev branch of 0.2.9 yesterday - a344009.

Hi @Thireus

It seem that Oobaboga can't use Qwen 32B exllamav2 yet right? I manage to convert Qwen 32B successfully using Exllamav2 dev brand but can't find any tools to run it yet. Can you tell me how to run it. I often run Oobaboga then using API to connect with Silly Tavern.
Are there any ways to run something similar ? Thank you so much

[Thireus]

@pivtienduc you'll need to compile the dev branch and replace the current version you've got installed with pip pip freeze | grep exllama

Make sure you are in the same conda venv (or whichever venv if any) of your tool. For example, if you're using text-generation-webui on linux:

# Clone exllamav2
git clone https://github.com/turboderp-org/exllamav2

# Go to text-generation-webui virtual env
conda deactivate
cd ~/text-generation-webui && source activate.sh && cd ~/exllamav2 # See Note* about activate.sh
git pull
git checkout dev # Switch to dev branch which implements partial support for Qwen3

# Full installation of exllamav2 dev branch or upgrade
rm -rf ~/exllamav2/build # It's important to do this, especially if the previous build was corrupted!
pip install -r requirements.txt # WARNING: This will replace numpy-2.2.5 by numpy-1.26.4
pip install .

# Check the install was successful
pip freeze | grep exllama # Should display exllamav2 @ file:///home/yourusername/exllamav2

*Note: The content of activate.sh is the content of the start_linux.sh script you're using, just without the last line python one_click.py $@.

[pivtienduc]

@pivtienduc you'll need to compile the dev branch and replace the current version you've got installed with pip pip freeze | grep exllama

Make sure you are in the same conda venv (or whichever venv if any) of your tool. For example, if you're using text-generation-webui on linux:

# Clone exllamav2
git clone https://github.com/turboderp-org/exllamav2

# Go to text-generation-webui virtual env
conda deactivate
cd ~/text-generation-webui && source activate.sh && cd ~/exllamav2 # See Note* about activate.sh
git pull
git checkout dev # Switch to dev branch which implements partial support for Qwen3

# Full installation of exllamav2 dev branch or upgrade
rm -rf ~/exllamav2/build # It's important to do this, especially if the previous build was corrupted!
pip install -r requirements.txt # WARNING: This will replace numpy-2.2.5 by numpy-1.26.4
pip install .

# Check the install was successful
pip freeze | grep exllama # Should display exllamav2 @ file:///home/yourusername/exllamav2

*Note: The content of activate.sh is the content of the start_linux.sh script you're using, just without the last line python one_click.py $@.

Sound too complicated, I know how to rebuild and compile but change more things like this it a bit too much since it need to understand more about Linux so I will wait until Obbaboga upgrade to support or exllamav2 merge dev into main. Thank you so much

[mongoostin]

I'm having a different problem when loading Qwen3 dense models in exl2 (same error for any of them, tried 32B and 0.6B, different quants, different authors):

2025-05-02 22:46:26.498 WARNING:  Draft model is disabled because a model name wasn't provided. Please check your
config.yml!
2025-05-02 22:46:26.499 WARNING:  The given cache_size (16384) is less than 2 * max_seq_len and may be too small for
requests using CFG.
2025-05-02 22:46:26.499 WARNING:  Ignore this warning if you do not plan on using CFG.
2025-05-02 22:46:26.499 INFO:     Attempting to load a prompt template if present.
2025-05-02 22:46:26.500 WARNING:  TemplateLoadError: Model JSON path
"/home/ubuntu/tabbyAPI/models/Qwen3-0.6B_4.25bpw_exl2/chat_template.json" not found.
2025-05-02 22:46:26.509 INFO:     Using template "from_tokenizer_config" for chat completions.
2025-05-02 22:46:26.819 INFO:     Loading model: /home/ubuntu/tabbyAPI/models/Qwen3-0.6B_4.25bpw_exl2
2025-05-02 22:46:26.820 INFO:     Loading with autosplit
Loading model modules ╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   2%  1/59 -:--:--
2025-05-02 22:46:27.045 ERROR:    Exception in ASGI application
2025-05-02 22:46:27.045 ERROR:    Traceback (most recent call last):
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
2025-05-02 22:46:27.045 ERROR:        result = await app(  # type: ignore[func-returns-value]
2025-05-02 22:46:27.045 ERROR:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
2025-05-02 22:46:27.045 ERROR:        return await self.app(scope, receive, send)
2025-05-02 22:46:27.045 ERROR:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
2025-05-02 22:46:27.045 ERROR:        await super().__call__(scope, receive, send)
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
2025-05-02 22:46:27.045 ERROR:        await self.middleware_stack(scope, receive, send)
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
2025-05-02 22:46:27.045 ERROR:        raise exc
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
2025-05-02 22:46:27.045 ERROR:        await self.app(scope, receive, _send)
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
2025-05-02 22:46:27.045 ERROR:        await self.app(scope, receive, send)
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
2025-05-02 22:46:27.045 ERROR:        await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2025-05-02 22:46:27.045 ERROR:        raise exc
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2025-05-02 22:46:27.045 ERROR:        await app(scope, receive, sender)
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/routing.py",
line 714, in __call__
2025-05-02 22:46:27.045 ERROR:        await self.middleware_stack(scope, receive, send)
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/routing.py",
line 734, in app
2025-05-02 22:46:27.045 ERROR:        await route.handle(scope, receive, send)
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/routing.py",
line 288, in handle
2025-05-02 22:46:27.045 ERROR:        await self.app(scope, receive, send)
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/routing.py",
line 76, in app
2025-05-02 22:46:27.045 ERROR:        await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2025-05-02 22:46:27.045 ERROR:        raise exc
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2025-05-02 22:46:27.045 ERROR:        await app(scope, receive, sender)
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/starlette/routing.py",
line 73, in app
2025-05-02 22:46:27.045 ERROR:        response = await f(request)
2025-05-02 22:46:27.045 ERROR:                   ^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/fastapi/routing.py",
line 301, in app
2025-05-02 22:46:27.045 ERROR:        raw_response = await run_endpoint_function(
2025-05-02 22:46:27.045 ERROR:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/fastapi/routing.py",
line 212, in run_endpoint_function
2025-05-02 22:46:27.045 ERROR:        return await dependant.call(**values)
2025-05-02 22:46:27.045 ERROR:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/tabbyAPI-dev/endpoints/OAI/router.py", line 112, in
chat_completion_request
2025-05-02 22:46:27.045 ERROR:        await load_inline_model(data.model, request)
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/tabbyAPI-dev/endpoints/OAI/utils/completion.py", line 184, in
load_inline_model
2025-05-02 22:46:27.045 ERROR:        await model.load_model(
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/tabbyAPI-dev/common/model.py", line 161, in load_model
2025-05-02 22:46:27.045 ERROR:        async for _ in load_model_gen(model_path, **kwargs):
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/tabbyAPI-dev/common/model.py", line 137, in load_model_gen
2025-05-02 22:46:27.045 ERROR:        async for module, modules in load_status:
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/tabbyAPI-dev/backends/exllamav2/model.py", line 470, in load_gen
2025-05-02 22:46:27.045 ERROR:        async for value in iterate_in_threadpool(model_load_generator):
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/tabbyAPI-dev/common/concurrency.py", line 30, in
iterate_in_threadpool
2025-05-02 22:46:27.045 ERROR:        yield await asyncio.to_thread(gen_next, generator)
2025-05-02 22:46:27.045 ERROR:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/miniconda3/lib/python3.12/asyncio/threads.py", line 25, in to_thread
2025-05-02 22:46:27.045 ERROR:        return await loop.run_in_executor(None, func_call)
2025-05-02 22:46:27.045 ERROR:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/miniconda3/lib/python3.12/concurrent/futures/thread.py", line 58, in
run
2025-05-02 22:46:27.045 ERROR:        result = self.fn(*self.args, **self.kwargs)
2025-05-02 22:46:27.045 ERROR:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/tabbyAPI-dev/common/concurrency.py", line 20, in gen_next
2025-05-02 22:46:27.045 ERROR:        return next(generator)
2025-05-02 22:46:27.045 ERROR:               ^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 57, in generator_context
2025-05-02 22:46:27.045 ERROR:        response = gen.send(request)
2025-05-02 22:46:27.045 ERROR:                   ^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/tabbyAPI-dev/backends/exllamav2/model.py", line 619, in
load_model_sync
2025-05-02 22:46:27.045 ERROR:        for value in self.model.load_autosplit_gen(
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/exllamav2/model.py",
line 622, in load_autosplit_gen
2025-05-02 22:46:27.045 ERROR:        module.load()
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-05-02 22:46:27.045 ERROR:        return func(*args, **kwargs)
2025-05-02 22:46:27.045 ERROR:               ^^^^^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/exllamav2/attn.py", line
252, in load
2025-05-02 22:46:27.045 ERROR:        if self.q_norm is not None: self.q_norm.load()
2025-05-02 22:46:27.045 ERROR:                                    ^^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File
"/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-05-02 22:46:27.045 ERROR:        return func(*args, **kwargs)
2025-05-02 22:46:27.045 ERROR:               ^^^^^^^^^^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:      File "/home/ubuntu/exllamav2/venv/lib/python3.12/site-packages/exllamav2/headnorm.py",
line 57, in load
2025-05-02 22:46:27.045 ERROR:        if len(weight.shape) == 1 and weight.shape[0] == self.model.config.head_dim:
2025-05-02 22:46:27.045 ERROR:               ^^^^^^^^^^^^
2025-05-02 22:46:27.045 ERROR:    AttributeError: 'NoneType' object has no attribute 'shape'

Exllamav2 at 68976a0 (dev branch).
The 30B MoE model loads and works fine, the problem is only with the dense models.

[Katehuuh]

Exllamav2 at 68976a0 (dev branch). The 30B MoE model loads and works fine, the problem is only with the dense models.

I hear Qwen3-30B-A3B-exl2 is supposed to be faster due to MoE. I've compiled Exllamav2 (dev branch) in ooba, but tokens/s are even lower than with the dense model.

Compile:

call C:\text-generation-webui\cmd_windows.bat
call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" x64
cd /d C:\ && git clone https://github.com/turboderp-org/exllamav2 && cd C:\exllamav2 && git switch dev
set DISTUTILS_USE_SDK=1
set MSSdk=1
set CUDA_HOME=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8
pip install .

(C:\text-generation-webui\installer_files\env) C:\exllamav2>python -c "import exllamav2; print(exllamav2.__version__)"
0.2.9

Benchmark:

INFO     Loading "turboderp_Qwen3-30B-A3B-exl2_4.0bpw"
INFO     Loaded "turboderp_Qwen3-30B-A3B-exl2_4.0bpw" in 92.94 seconds.
INFO     LOADER: "ExLlamav2"
INFO     TRUNCATION LENGTH: 40960
INFO     INSTRUCTION TEMPLATE: "Custom (obtained from model metadata)"
Output generated in 17.08 seconds (31.56 tokens/s, 539 tokens, context 95, seed 2070023933)
Output generated in 7.70 seconds (38.70 tokens/s, 298 tokens, context 95, seed 682143513)
Output generated in 13.08 seconds (29.27 tokens/s, 383 tokens, context 700, seed 1473531359)
Output generated in 13.51 seconds (30.58 tokens/s, 413 tokens, context 1710, seed 1638064417)
INFO     Loading "Qwen2.5-Coder-32B-exl2-bpw4.7-h8"
INFO     Loaded "Qwen2.5-Coder-32B-exl2-bpw4.7-h8" in 24.95 seconds.
INFO     LOADER: "ExLlamav2"
INFO     TRUNCATION LENGTH: 40960
INFO     INSTRUCTION TEMPLATE: "Custom (obtained from model metadata)"
Output generated in 14.29 seconds (36.18 tokens/s, 517 tokens, context 1727, seed 2129712675)
Output generated in 10.82 seconds (41.50 tokens/s, 449 tokens, context 1727, seed 1830443184)

• Windows+2.7.0+cu128+py3.12

Edit1: exl3 v0.0.2: Qwen3-30B-A3B-exl3

INFO     Loaded "turboderp_Qwen3-30B-A3B-exl3_4.0bpw" in 11.88 seconds.
INFO     LOADER: "ExLlamav3_HF"
INFO     TRUNCATION LENGTH: 32768

Output generated in 53.95 seconds (15.40 tokens/s, 831 tokens, context 1629, seed 1738499746)
Output generated in 17.16 seconds (19.34 tokens/s, 332 tokens, context 1528, seed 2138735888)
Output generated in 94.57 seconds (21.66 tokens/s, 2048 tokens, context 1542, seed 2455159)
Output generated in 55.62 seconds (21.32 tokens/s, 1186 tokens, context 1507, seed 1716390959)

Edit2: When ever i use oobabooga/text-generation-webui running MMLU-Pro ctx32768+flash-attn:

• Qwen2.5-Coder-32B-Instruct-abliterated-Q4_K_M.gguf 57t/s
• ArtusDev/huihui-ai_Qwen2.5-Coder-32B-Instruct-abliterated-EXL2:4.0bpw_H6 ExLlamav2_HF: 38t/s, ExLlamav2: 40t/s 21.4GB VRAM
• ArtusDev/huihui-ai_Qwen2.5-Coder-32B-Instruct-abliterated-EXL3:4.0bpw_H6 ExLlamav3_HF: 24t/s - 20GB VRAM

Sometimes, exl3 is half the speed of gguf:

• mistralai_Mistral-Small-3.2-24B-Instruct-2506-EXL3_4.0bpw_H6 and get 40 tokens/s
• Mistral-Small-3.2-24B-Instruct-2506-UD-Q4_K_XL.gguf 80 tokens/s
  But GGUF isn't really faster in batch processing because MMLU-Pro outputs a small number of tokens (<200), so whenever processing long context 4k token in/out EXL wins.

[umar-mq]

@pivtienduc you'll need to compile the dev branch and replace the current version you've got installed with pip pip freeze | grep exllama

Make sure you are in the same conda venv (or whichever venv if any) of your tool. For example, if you're using text-generation-webui on linux:

# Clone exllamav2
git clone https://github.com/turboderp-org/exllamav2

# Go to text-generation-webui virtual env
conda deactivate
cd ~/text-generation-webui && source activate.sh && cd ~/exllamav2 # See Note* about activate.sh
git pull
git checkout dev # Switch to dev branch which implements partial support for Qwen3

# Full installation of exllamav2 dev branch or upgrade
rm -rf ~/exllamav2/build # It's important to do this, especially if the previous build was corrupted!
pip install -r requirements.txt # WARNING: This will replace numpy-2.2.5 by numpy-1.26.4
pip install .

# Check the install was successful
pip freeze | grep exllama # Should display exllamav2 @ file:///home/yourusername/exllamav2

*Note: The content of activate.sh is the content of the start_linux.sh script you're using, just without the last line python one_click.py $@.

I have done as you have said and upgraded to the dev branch. But when trying to load the 30B dense model I have the following error:

Traceback (most recent call last):
  File "/home/thomas/tabbyAPI/start.py", line 293, in <module>
    entrypoint(args, parser)
  File "/home/thomas/tabbyAPI/main.py", line 166, in entrypoint
    asyncio.run(entrypoint_async())
  File "/home/thomas/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/home/thomas/tabbyAPI/main.py", line 71, in entrypoint_async
    await model.load_model(
  File "/home/thomas/tabbyAPI/common/model.py", line 161, in load_model
    async for _ in load_model_gen(model_path, **kwargs):
  File "/home/thomas/tabbyAPI/common/model.py", line 137, in load_model_gen
    async for module, modules in load_status:
  File "/home/thomas/tabbyAPI/backends/exllamav2/model.py", line 482, in load_gen
    async for value in iterate_in_threadpool(model_load_generator):
  File "/home/thomas/tabbyAPI/common/concurrency.py", line 30, in iterate_in_threadpool
    yield await asyncio.to_thread(gen_next, generator)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI/common/concurrency.py", line 20, in gen_next
    return next(generator)
           ^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 57, in generator_context
    response = gen.send(request)
               ^^^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI/backends/exllamav2/model.py", line 601, in load_model_sync
    for value in self.model.load_tp_gen(
  File "/home/thomas/tabbyAPI/venv/lib/python3.11/site-packages/exllamav2/model.py", line 449, in load_tp_gen
    module.load(device_context = False)
  File "/home/thomas/tabbyAPI/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI/venv/lib/python3.11/site-packages/exllamav2/attn.py", line 252, in load
    if self.q_norm is not None: self.q_norm.load()
                                ^^^^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/tabbyAPI/venv/lib/python3.11/site-packages/exllamav2/headnorm.py", line 57, in load
    if len(weight.shape) == 1 and weight.shape[0] == self.model.config.head_dim:
           ^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'|

[pivtienduc]

Exllamav2 at 68976a0 (dev branch). The 30B MoE model loads and works fine, the problem is only with the dense models.

I hear Qwen3-30B-A3B-exl2 is supposed to be faster due to MoE. I've compiled Exllamav2 (dev branch) in ooba, but tokens/s are even lower than with the dense model.

Compile:

call C:\text-generation-webui\cmd_windows.bat
call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" x64
cd /d C:\ && git clone https://github.com/turboderp-org/exllamav2 && cd C:\exllamav2 && git switch dev
set DISTUTILS_USE_SDK=1
set MSSdk=1
set CUDA_HOME=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8
pip install .
(C:\text-generation-webui\installer_files\env) C:\exllamav2>python -c "import exllamav2; print(exllamav2.version)"
0.2.9
Benchmark:

INFO     Loading "turboderp_Qwen3-30B-A3B-exl2_4.0bpw"
INFO     Loaded "turboderp_Qwen3-30B-A3B-exl2_4.0bpw" in 92.94 seconds.
INFO     LOADER: "ExLlamav2"
INFO     TRUNCATION LENGTH: 40960
INFO     INSTRUCTION TEMPLATE: "Custom (obtained from model metadata)"
Output generated in 17.08 seconds (31.56 tokens/s, 539 tokens, context 95, seed 2070023933)
Output generated in 7.70 seconds (38.70 tokens/s, 298 tokens, context 95, seed 682143513)
Output generated in 13.08 seconds (29.27 tokens/s, 383 tokens, context 700, seed 1473531359)
Output generated in 13.51 seconds (30.58 tokens/s, 413 tokens, context 1710, seed 1638064417)
INFO     Loading "Qwen2.5-Coder-32B-exl2-bpw4.7-h8"
INFO     Loaded "Qwen2.5-Coder-32B-exl2-bpw4.7-h8" in 24.95 seconds.
INFO     LOADER: "ExLlamav2"
INFO     TRUNCATION LENGTH: 40960
INFO     INSTRUCTION TEMPLATE: "Custom (obtained from model metadata)"
Output generated in 14.29 seconds (36.18 tokens/s, 517 tokens, context 1727, seed 2129712675)
Output generated in 10.82 seconds (41.50 tokens/s, 449 tokens, context 1727, seed 1830443184)

• Windows+2.7.0+cu128+py3.12

Dont compile in Oobaboga, you need install Exllamav2 in a separated env then compile only the model after that load the model in Oobaboga then it should work.

[randoentity]

@mongoostin @umar-mq I was stumped by the same error (AttributeError: 'NoneType' object has no attribute 'shape') but I had put it aside as exl3 was working fine. I looked into it the past few days and I noticed that, as the rest of the stack trace shows, the q_norm is missing (cf. https://huggingface.co/Jellon/Qwen3-32B-exl2-6bpw/tree/main?show_file_info=output-00001-of-00004.safetensors and https://huggingface.co/matatonic/Qwen3-32B-6.5bpw-h8-exl2/tree/main?show_file_info=output-00001-of-00003.safetensors and you'll see that the q_norm and k_norm weights are not listed in the former). So the matatonic quants do work, except when I enable tensor parallel (TP):

...exllamav2/exllamav2/attn.py", line 1227, in forward_tp
    ext_c.tp_attn_forward_(
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

[Originalimoc]

Qwen3 32B works fine but A3B MoE is super slow. I now use llama.cpp vulkan for it like 3 to 5x+ speed.