Anyone using vLLM on Altra Max M128-30 (ARM Neoverse N1)?

[stelterlab]

Hi!

Did anyone try to run vLLM on an Altra Max M128-30 system yet?

The CPU is reported as:

Vendor ID: ARM
Model name: Neoverse-N1

According to the dev specs it supports NEON (see https://developer.arm.com/Processors/Neoverse%20N1). I tried my luck with Dockerfile.arm (as described in https://docs.vllm.ai/en/latest/getting_started/arm-installation.html), but it fails with not finding trition when I am trying to use GGUF files.

When using the standard *.safetensor format I get an runtime error "rms_norm_impl" not implemented for 'BFloat16'.

Any hints on which format could be used with such ARM CPUs?

Kind regards, cos

[sourcesync]

Christian, I'm curious - did you compile from src? I see several "#ifdef ARM_BF16_SUPPORT" guards in the ARM types file: csrc/cpu/cpu_types_arm.hpp George

…

On Sun, Dec 29, 2024 at 9:15 AM Christian Otto Stelter < ***@***.***> wrote: Hi! Did anyone try to run vLLM on an Altra Max M128-30 system yet? The CPU is reported as: Vendor ID: ARM Model name: Neoverse-N1 According to the dev specs it supports NEON (see https://developer.arm.com/Processors/Neoverse%20N1). I tried my luck with Dockerfile.arm (as described in https://docs.vllm.ai/en/latest/getting_started/arm-installation.html), but it fails with not finding trition when I am trying to use GGUF files. When using the standard *.safetensor format I get an runtime error "rms_norm_impl" not implemented for 'BFloat16'. Any hints on which format could be used with such ARM CPUs? Kind regards, cos — Reply to this email directly, view it on GitHub <#11605>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADL6CK5BWR3YZ3RNNJDWRD2IAU4FAVCNFSM6AAAAABUK7DVCWVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXG42TMNRVHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[stelterlab]

I used the Dockerfile.arm without modifying which installs a gcc-12 and does:

RUN --mount=type=cache,target=/root/.cache/pip \
    --mount=type=cache,target=/root/.cache/ccache \
    --mount=type=bind,source=.git,target=.git \
    VLLM_TARGET_DEVICE=cpu python3 setup.py bdist_wheel && \
    pip install dist/*.whl && \
    rm -rf dist

for installing vLLM in the container. I used a "bleeding edge" version: v0.6.6.post2.dev19+g328841d0.

So, yes I assume some libs will be compiled from source without looking deeper into the magic of the install process of vLLM. ;-)