GLM4.5v使用48GB的GPU 进行4卡部署出错

[Jimmy-L99]

System Info / 系統信息

我在RTX 5880 进行 GLM4.5v-fp8 tp=4 部署遇到报错，按理说显存应该足够权重加载和 kvcache 使用，但这个貌似是moe的问题？请教一下该怎么解决，还是得等官方更新优化？

不使用--disable-cuda-graph时，加载权重后报错；
使用--disable-cuda-graph时，权重顺利加载，推理时报错。

triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

sglang issue 9393

Who can help? / 谁可以帮助到您？

@zRzRzRzRzRzRzR

Information / 问题信息

• The official example scripts / 官方的示例脚本
• My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

Expected behavior / 期待表现

模型能够正常推理

[zRzRzRzRzRzRzR]

这看上去好像是你这卡的架构在推理FP8的时候triton的某些算子兼容？
--disable-cuda-graph 正常是不加的，看看详细报错
另外，你装的vlLM是main分支吗，必须是main最新几个commit因为有一个相关的bug刚修复

[Jimmy-L99]

@zRzRzRzRzRzRzR
我用的是sglang:v0.5.0rc2-cu126镜像，启动时换所需的transformers版本。
启动命令：

services:
  sglang-glm4.5v-106b:
    image: lmsysorg/sglang:v0.5.0rc2-cu126
    container_name: sglang-glm4.5v-106b-a12b
    volumes:
      - /home/vg_llm/model:/model
      - /etc/localtime:/etc/localtime:ro
      - /usr/share/zoneinfo/Asia/Shanghai:/usr/share/zoneinfo/Asia/Shanghai:ro
    restart: always
    network_mode: host # required by RDMA
    privileged: true # required by RDMA
    environment:
      - CUDA_VISIBLE_DEVICES=4,5,6,7
    entrypoint: >
        sh -c "
        pip install transformers-v4.55.0-GLM-4.5V-preview -i https://pypi.tuna.tsinghua.edu.cn/simple &&
        python3 -m sglang.launch_server \
        --model-path /model/GLM-4.5V-FP8 \
        --host 192.168.10.87 \
        --port 8012 \
        --context-length 40960 \
        --tp-size 4 \
        --mem-fraction-static 0.80 \
        --chunked-prefill-size 2048 \
        --tool-call-parser glm45 \
        --reasoning-parser glm45 \
        --disable-cuda-graph
        --enable-metrics
        "
    ulimits:
      memlock: -1
      stack: 67108864
    ipc: host
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://192.168.10.87:8012/health || exit 1"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["4", "5", "6", "7"]
              capabilities: [gpu]

详细报错如下：

[2025-08-20 16:14:12] Received sigquit from a child process. It usually means the child failed.
[2025-08-20 16:14:12 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2555, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 315, in __init__
    self.tp_worker = TpWorkerClass(
                     ^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 67, in __init__
    self.worker = TpModelWorker(
                  ^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 84, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 242, in __init__
    self.initialize(min_per_gpu_memory)
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 342, in initialize
    self.init_cuda_graphs()
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1611, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 380, in __init__
    self.capture()
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 485, in capture
    ) = self.capture_one_batch_size(bs, forward)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 649, in capture_one_batch_size
    run_once()
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 637, in run_once
    logits_output_or_pp_proxy_tensors = forward(
                                        ^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/models/qwen2_5_vl.py", line 559, in forward
    hidden_states = general_mm_embed_routine(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py", line 659, in general_mm_embed_routine
    hidden_states = language_model(
                    ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 2102, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/models/glm4_moe.py", line 723, in forward
    hidden_states = self.mlp(hidden_states, forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py", line 466, in forward
    return self.forward_normal(
           ^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/models/glm4_moe.py", line 595, in forward_normal
    final_hidden_states = self.experts(**kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 821, in forward
    final_hidden_states = self.quant_method.apply(
                          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors_moe.py", line 281, in apply
    return fused_experts(
           ^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py", line 1223, in fused_experts
    torch.ops.sglang.inplace_fused_experts(
  File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1243, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py", line 1031, in inplace_fused_experts
    fused_experts_impl(
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py", line 1511, in fused_experts_impl
    invoke_fused_moe_kernel(
  File "/sgl-workspace/sglang/python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py", line 765, in invoke_fused_moe_kernel
    fused_moe_kernel[grid](
  File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 390, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 617, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
    ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 498, in __getattribute__
    self._init_handles()
  File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 483, in _init_handles
    raise OutOfResources(self.metadata.shared, max_shared, "shared memory")
triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

[2025-08-20 16:14:12] Received sigquit from a child process. It usually means the child failed.

[readbyte-ai]

我是4卡L40，一模一样的错误

[zRzRzRzRzRzRzR]

4张L40只有 4 * 48G显存，只能带的动FP8，同时，这个错误看上去这个架构的卡有一定的问题？我们在H100系列上测过，但是没有在A系列架构上测试，A系列目前知道可能存在一些类似你这个报错的问题

[readbyte-ai]

4张L40只有 4 * 48G显存，只能带的动FP8，同时，这个错误看上去这个架构的卡有一定的问题？我们在H100系列上测过，但是没有在A系列架构上测试，A系列目前知道可能存在一些类似你这个报错的问题

我是用SGLang跑FP8，一模一样的错误，开始用Vllm引擎部署，识别响应输出是连续感叹号，应该是vllm引擎的bug！折腾两天了，还是没有办法。

[zRzRzRzRzRzRzR]

vLLM的话，尝试使用源代码安装或者nightly，FP8的bug已经fixl过了

[readbyte-ai]

vLLM的话，尝试使用源代码安装或者nightly，FP8的bug已经fixl过了

谢谢，我试试！

[readbyte-ai]

vLLM的话，尝试使用源代码安装或者nightly，FP8的bug已经fixl过了

谢谢，我试试！

vllm引擎运行GLM4.5-FP8的问题已经解决，我对vllm进行了编译构建，直接做成whl包，同时torch升级到了2.8.0，transformer一样得从源代码构建安装，同时要安装GLM的transformer包。

[zRzRzRzRzRzRzR]

Cool

[ChihayaK]

I encountered exactly this issue using sglang. However, I found that following the instructions here helped. Initially I couldn't run it, but after tuning with the non-v variant of GLM-4.5-fp8, the configuration could be used with the GLM-4.5V-fp8 and worked.
But strangely, the fused MoE does not work on the GLM-4.5-fp8. (Running it shows that it is trying to find configuration E=129, N=352. But the tuned result saves a file that is N=128, E=352), which is really confusing.
But using the fused MoE definitely works though.