You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -183,7 +184,6 @@ Currently in vLLM for HPU we support four execution modes, depending on selected
183
184
| 0 | 0 | torch.compile |
184
185
| 0 | 1 | PyTorch eager mode |
185
186
| 1 | 0 | HPU Graphs |
186
-
<figcaption>vLLM execution modes</figcaption>
187
187
188
188
!!! warning
189
189
In 1.18.0, all modes utilizing `PT_HPU_LAZY_MODE=0` are highly experimental and should be only used for validating functional correctness. Their performance will be improved in the next releases. For obtaining the best performance in 1.18.0, please use HPU Graphs, or PyTorch lazy mode.
Copy file name to clipboardExpand all lines: docs/getting_started/installation/ai_accelerator/neuron.inc.md
+28-27Lines changed: 28 additions & 27 deletions
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,8 @@
1
1
# --8<-- [start:installation]
2
2
3
-
[AWS Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/) is the software development kit (SDK) used to run deep learning and
4
-
generative AI workloads on AWS Inferentia and AWS Trainium powered Amazon EC2 instances and UltraServers (Inf1, Inf2, Trn1, Trn2,
5
-
and Trn2 UltraServer). Both Trainium and Inferentia are powered by fully-independent heterogeneous compute-units called NeuronCores.
3
+
[AWS Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/) is the software development kit (SDK) used to run deep learning and
4
+
generative AI workloads on AWS Inferentia and AWS Trainium powered Amazon EC2 instances and UltraServers (Inf1, Inf2, Trn1, Trn2,
5
+
and Trn2 UltraServer). Both Trainium and Inferentia are powered by fully-independent heterogeneous compute-units called NeuronCores.
6
6
This tab describes how to set up your environment to run vLLM on Neuron.
7
7
8
8
!!! warning
@@ -17,11 +17,12 @@
17
17
- Accelerator: NeuronCore-v2 (in trn1/inf2 chips) or NeuronCore-v3 (in trn2 chips)
18
18
- AWS Neuron SDK 2.23
19
19
20
-
## Configure a new environment
20
+
# --8<-- [end:requirements]
21
+
# --8<-- [start:configure-a-new-environment]
21
22
22
23
### Launch a Trn1/Trn2/Inf2 instance and verify Neuron dependencies
23
24
24
-
The easiest way to launch a Trainium or Inferentia instance with pre-installed Neuron dependencies is to follow this
25
+
The easiest way to launch a Trainium or Inferentia instance with pre-installed Neuron dependencies is to follow this
25
26
[quick start guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/setup/neuron-setup/multiframework/multi-framework-ubuntu22-neuron-dlami.html#setup-ubuntu22-multi-framework-dlami) using the Neuron Deep Learning AMI (Amazon machine image).
26
27
27
28
- After launching the instance, follow the instructions in [Connect to your instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html) to connect to the instance
@@ -30,14 +31,14 @@ The easiest way to launch a Trainium or Inferentia instance with pre-installed N
Refer to the [NxD Inference Setup Guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/nxdi-setup.html)
34
+
Refer to the [NxD Inference Setup Guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/nxdi-setup.html)
34
35
for alternative setup instructions including using Docker and manually installing dependencies.
35
36
36
37
!!! note
37
-
NxD Inference is the default recommended backend to run inference on Neuron. If you are looking to use the legacy [transformers-neuronx](https://github.com/aws-neuron/transformers-neuronx)
38
-
library, refer to [Transformers NeuronX Setup](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/transformers-neuronx/setup/index.html).
38
+
NxD Inference is the default recommended backend to run inference on Neuron. If you are looking to use the legacy [transformers-neuronx](https://github.com/aws-neuron/transformers-neuronx)
39
+
library, refer to [Transformers NeuronX Setup](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/transformers-neuronx/setup/index.html).
AWS Neuron maintains a [Github fork of vLLM](https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2) at
63
-
[https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2](https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2), which contains several features in addition to what's
63
+
AWS Neuron maintains a [Github fork of vLLM](https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2) at
64
+
[https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2](https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2), which contains several features in addition to what's
64
65
available on vLLM V0. Please utilize the AWS Fork for the following features:
65
66
66
67
- Llama-3.2 multi-modal support
67
-
- Multi-node distributed inference
68
+
- Multi-node distributed inference
68
69
69
-
Refer to [vLLM User Guide for NxD Inference](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/vllm-user-guide.html)
70
+
Refer to [vLLM User Guide for NxD Inference](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/vllm-user-guide.html)
70
71
for more details and usage examples.
71
72
72
73
To install the AWS Neuron fork, run the following:
@@ -101,11 +102,11 @@ Make sure to use <gh-file:docker/Dockerfile.neuron> in place of the default Dock
The current vLLM and Neuron integration relies on either the `neuronx-distributed-inference` (preferred) or `transformers-neuronx` backend
105
-
to perform most of the heavy lifting which includes PyTorch model initialization, compilation, and runtime execution. Therefore, most
106
-
[features supported on Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/feature-guide.html) are also available via the vLLM integration.
105
+
The current vLLM and Neuron integration relies on either the `neuronx-distributed-inference` (preferred) or `transformers-neuronx` backend
106
+
to perform most of the heavy lifting which includes PyTorch model initialization, compilation, and runtime execution. Therefore, most
107
+
[features supported on Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/nxd-inference/developer_guides/feature-guide.html) are also available via the vLLM integration.
107
108
108
-
To configure NxD Inference features through the vLLM entrypoint, use the `override_neuron_config` setting. Provide the configs you want to override
109
+
To configure NxD Inference features through the vLLM entrypoint, use the `override_neuron_config` setting. Provide the configs you want to override
109
110
as a dictionary (or JSON object when starting vLLM from the CLI). For example, to disable auto bucketing, include
110
111
```console
111
112
override_neuron_config={
@@ -117,33 +118,33 @@ or when launching vLLM from the CLI, pass
to quantize and compile your model using NxD Inference, and then load the compiled artifacts into vLLM.
131
-
- Multi-LoRA serving: NxD Inference only supports loading of LoRA adapters at server startup. Dynamic loading of LoRA adapters at
132
+
- Multi-LoRA serving: NxD Inference only supports loading of LoRA adapters at server startup. Dynamic loading of LoRA adapters at
132
133
runtime is not currently supported. Refer to [multi-lora example](https://github.com/aws-neuron/upstreaming-to-vllm/blob/neuron-2.23-vllm-v0.7.2/examples/offline_inference/neuron_multi_lora.py)
133
134
- Multi-modal support: multi-modal support is only available through the AWS Neuron fork. This feature has not been upstreamed
134
135
to vLLM main because NxD Inference currently relies on certain adaptations to the core vLLM logic to support this feature.
135
136
- Multi-node support: distributed inference across multiple Trainium/Inferentia instances is only supported on the AWS Neuron fork. Refer
136
137
to this [multi-node example](https://github.com/aws-neuron/upstreaming-to-vllm/tree/neuron-2.23-vllm-v0.7.2/examples/neuron/multi_node)
137
138
to run. Note that tensor parallelism (distributed inference across NeuronCores) is available in vLLM main.
138
-
- Known edge case bug in speculative decoding: An edge case failure may occur in speculative decoding when sequence length approaches
139
-
max model length (e.g. when requesting max tokens up to the max model length and ignoring eos). In this scenario, vLLM may attempt
140
-
to allocate an additional block to ensure there is enough memory for number of lookahead slots, but since we do not have good support
141
-
for paged attention, there isn't another Neuron block for vLLM to allocate. A workaround fix (to terminate 1 iteration early) is
139
+
- Known edge case bug in speculative decoding: An edge case failure may occur in speculative decoding when sequence length approaches
140
+
max model length (e.g. when requesting max tokens up to the max model length and ignoring eos). In this scenario, vLLM may attempt
141
+
to allocate an additional block to ensure there is enough memory for number of lookahead slots, but since we do not have good support
142
+
for paged attention, there isn't another Neuron block for vLLM to allocate. A workaround fix (to terminate 1 iteration early) is
142
143
implemented in the AWS Neuron fork but is not upstreamed to vLLM main as it modifies core vLLM logic.
143
144
144
145
145
146
### Environment variables
146
-
-`NEURON_COMPILED_ARTIFACTS`: set this environment variable to point to your pre-compiled model artifacts directory to avoid
147
+
-`NEURON_COMPILED_ARTIFACTS`: set this environment variable to point to your pre-compiled model artifacts directory to avoid
147
148
compilation time upon server initialization. If this variable is not set, the Neuron module will perform compilation and save the
148
149
artifacts under `neuron-compiled-artifacts/{unique_hash}/` sub-directory in the model path. If this environment variable is set,
149
150
but the directory does not exist, or the contents are invalid, Neuron will also fallback to a new compilation and store the artifacts
| QUEUED_RESOURCE_ID | The user-assigned ID of the queued resource request. |
84
-
| TPU_NAME | The user-assigned name of the TPU which is created when the queued |
86
+
| TPU_NAME | The user-assigned name of the TPU which is created when the queued resource request is allocated.|
85
87
| PROJECT_ID | Your Google Cloud project |
86
-
| ZONE | The GCP zone where you want to create your Cloud TPU. The value you use |
87
-
| ACCELERATOR_TYPE | The TPU version you want to use. Specify the TPU version, for example |
88
-
| RUNTIME_VERSION | The TPU VM runtime version to use. For example, use `v2-alpha-tpuv6e` for a VM loaded with one or more v6e TPU(s). For more information see [TPU VM images](https://cloud.google.com/tpu/docs/runtimes).|
89
-
<figcaption>Parameter descriptions</figcaption>
88
+
| ZONE | The GCP zone where you want to create your Cloud TPU. The value you use depends on the version of TPUs you are using. For more information, see [TPU regions and zones]|
89
+
| ACCELERATOR_TYPE | The TPU version you want to use. Specify the TPU version, for example `v5litepod-4` specifies a v5e TPU with 4 cores, `v6e-1` specifies a v6e TPU with 1 core. For more information, see [TPU versions].|
90
+
| RUNTIME_VERSION | The TPU VM runtime version to use. For example, use `v2-alpha-tpuv6e` for a VM loaded with one or more v6e TPU(s). For more information see [TPU VM images]. |
91
+
| SERVICE_ACCOUNT | The email address for your service account. You can find it in the IAM Cloud Console under *Service Accounts*. For example: `tpu-service-account@<your_project_ID>.iam.gserviceaccount.com`|
90
92
91
93
Connect to your TPU using SSH:
92
94
93
95
```bash
94
96
gcloud compute tpus tpu-vm ssh TPU_NAME --zone ZONE
0 commit comments