Skip to content

Commit 59e0250

Browse files
YikunYikun
authored
[CI] Add e2e test frame work and doctest (#730)
### What this PR does / why we need it? Add quickstart doctest CI ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? - CI passed - Run `/vllm-ascend/tests/e2e/run_doctests.sh` Related: #725 Signed-off-by: Yikun Jiang <yikunkero@gmail.com>
1 parent 857f489 commit 59e0250

File tree

5 files changed

+243
-3
lines changed

5 files changed

+243
-3
lines changed

‎.github/workflows/vllm_ascend_doctest.yaml

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
#
2+
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
# This file is a part of the vllm-ascend project.
16+
#
17+
18+
name: 'e2e test / doctest'
19+
20+
on:
21+
workflow_dispatch:
22+
pull_request:
23+
branches:
24+
- 'main'
25+
- '*-dev'
26+
paths:
27+
# If we are changing the doctest we should do a PR test
28+
- '.github/workflows/vllm_ascend_doctest.yaml'
29+
- 'tests/e2e/doctests/**'
30+
- 'tests/e2e/common.sh'
31+
- 'tests/e2e/run_doctests.sh'
32+
schedule:
33+
# Runs every 4 hours
34+
- cron: '0 */4 * * *'
35+
36+
# Bash shells do not use ~/.profile or ~/.bashrc so these shells need to be explicitly
37+
# declared as "shell: bash -el {0}" on steps that need to be properly activated.
38+
# It's used to activate ascend-toolkit environment variables.
39+
defaults:
40+
run:
41+
shell: bash -el {0}
42+
43+
jobs:
44+
test:
45+
strategy:
46+
# Each version should be tested
47+
fail-fast: false
48+
matrix:
49+
vllm_verison: [main, v0.7.3-dev, main-openeuler, v0.7.3-dev-openeuler]
50+
name: vLLM Ascend test
51+
runs-on: linux-arm64-npu-1
52+
container:
53+
image: m.daocloud.io/quay.io/ascend/vllm-ascend:${{ matrix.vllm_verison }}
54+
steps:
55+
- name: Check NPU/CANN and git info
56+
run: |
57+
echo "====> Print NPU/CANN info"
58+
npu-smi info
59+
cat /usr/local/Ascend/ascend-toolkit/latest/"$(uname -i)"-linux/ascend_toolkit_install.info
60+
61+
echo "====> Print vllm-ascend git info"
62+
cd /vllm-workspace/vllm-ascend
63+
git --no-pager log -1 || true
64+
echo "====> Print vllm git info"
65+
cd /vllm-workspace/vllm
66+
git --no-pager log -1 || true
67+
68+
- name: Config OS mirrors - Ubuntu
69+
if: ${{ !endsWith(matrix.vllm_verison, '-openeuler') }}
70+
run: |
71+
sed -i 's|ports.ubuntu.com|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list
72+
apt-get update -y
73+
apt install git curl -y
74+
75+
- name: Config OS mirrors - openEuler
76+
if: ${{ endsWith(matrix.vllm_verison, '-openeuler') }}
77+
run: |
78+
yum update -y
79+
yum install git curl -y
80+
81+
- name: Config pip mirrors
82+
run: |
83+
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
84+
85+
- name: Checkout vllm-project/vllm-ascend repo
86+
uses: actions/checkout@v4
87+
88+
- name: Run vllm-ascend/tests/e2e/run_doctests.sh
89+
run: |
90+
# PWD: /__w/vllm-ascend/vllm-ascend
91+
# Address old branch like v0.7.3:
92+
if [ ! -d /vllm-workspace/vllm-ascend/tests/e2e ]; then
93+
echo "Warning: the doctest path doesn't exists, copy now"
94+
cp -r tests/e2e /vllm-workspace/vllm-ascend/tests/
95+
fi
96+
97+
# Simulate container to enter directory
98+
cd /workspace
99+
100+
# Run real test
101+
echo "Test:"
102+
/vllm-workspace/vllm-ascend/tests/e2e/run_doctests.sh

‎docs/source/quick_start.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ The default workdir is `/workspace`, vLLM and vLLM Ascend code are placed in `/v
6868

6969
You can use Modelscope mirror to speed up download:
7070

71+
<!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->
7172
```bash
7273
export VLLM_USE_MODELSCOPE=true
7374
```
@@ -81,6 +82,7 @@ With vLLM installed, you can start generating texts for list of input prompts (i
8182

8283
Try to run below Python script directly or use `python3` shell to generate texts:
8384

85+
<!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->
8486
```python
8587
from vllm import LLM, SamplingParams
8688

@@ -108,6 +110,7 @@ vLLM can also be deployed as a server that implements the OpenAI API protocol. R
108110
the following command to start the vLLM server with the
109111
[Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) model:
110112

113+
<!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->
111114
```bash
112115
# Deploy vLLM server (The first run will take about 3-5 mins (10 MB/s) to download models)
113116
vllm serve Qwen/Qwen2.5-0.5B-Instruct &
@@ -125,12 +128,14 @@ Congratulations, you have successfully started the vLLM server!
125128

126129
You can query the list the models:
127130

131+
<!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->
128132
```bash
129133
curl http://localhost:8000/v1/models | python3 -m json.tool
130134
```
131135

132136
You can also query the model with input prompts:
133137

138+
<!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->
134139
```bash
135140
curl http://localhost:8000/v1/completions \
136141
-H "Content-Type: application/json" \
@@ -145,10 +150,10 @@ curl http://localhost:8000/v1/completions \
145150
vLLM is serving as background process, you can use `kill -2 $VLLM_PID` to stop the background process gracefully,
146151
it's equal to `Ctrl-C` to stop foreground vLLM process:
147152

153+
<!-- tests/e2e/doctest/001-quickstart-test.sh should be considered updating as well -->
148154
```bash
149-
ps -ef | grep "/.venv/bin/vllm serve" | grep -v grep
150-
VLLM_PID=`ps -ef | grep "/.venv/bin/vllm serve" | grep -v grep | awk '{print $2}'`
151-
kill -2 $VLLM_PID
155+
VLLM_PID=$(pgrep -f "vllm serve")
156+
kill -2 "$VLLM_PID"
152157
```
153158

154159
You will see output as below:

‎tests/e2e/common.sh

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# bash fonts colors
2+
cyan='\e[96m'
3+
yellow='\e[33m'
4+
red='\e[31m'
5+
none='\e[0m'
6+
7+
_cyan() { echo -e "${cyan}$*${none}"; }
8+
_yellow() { echo -e "${yellow}$*${none}"; }
9+
_red() { echo -e "${red}$*${none}"; }
10+
11+
_info() { _cyan "Info: $*"; }
12+
_warn() { _yellow "Warn: $*"; }
13+
_err() { _red "Error: $*" && exit 1; }
14+
15+
CURL_TIMEOUT=1
16+
CURL_COOLDOWN=5
17+
CURL_MAX_TRIES=120
18+
19+
function wait_url_ready() {
20+
local serve_name="$1"
21+
local url="$2"
22+
i=0
23+
while true; do
24+
_info "===> Waiting for ${serve_name} to be ready...${i}s"
25+
i=$((i + CURL_COOLDOWN))
26+
set +e
27+
curl --silent --max-time "$CURL_TIMEOUT" "${url}" >/dev/null
28+
result=$?
29+
set -e
30+
if [ "$result" -eq 0 ]; then
31+
break
32+
fi
33+
if [ "$i" -gt "$CURL_MAX_TRIES" ]; then
34+
_info "===> \$CURL_MAX_TRIES exceeded waiting for ${serve_name} to be ready"
35+
return 1
36+
fi
37+
sleep "$CURL_COOLDOWN"
38+
done
39+
_info "===> ${serve_name} is ready."
40+
}
41+
42+
function wait_for_exit() {
43+
local VLLM_PID="$1"
44+
while kill -0 "$VLLM_PID"; do
45+
_info "===> Wait for ${VLLM_PID} to exit."
46+
sleep 1
47+
done
48+
_info "===> Wait for ${VLLM_PID} to exit."
49+
}
50+
51+
SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)

‎tests/e2e/doctests/001-quickstart-test.sh

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
#!/bin/bash
2+
3+
#
4+
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
5+
#
6+
# Licensed under the Apache License, Version 2.0 (the "License");
7+
# you may not use this file except in compliance with the License.
8+
# You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
# This file is a part of the vllm-ascend project.
18+
#
19+
20+
function simple_test() {
21+
# Do real import test
22+
python3 -c "import vllm; print(vllm.__version__)"
23+
}
24+
25+
function quickstart_offline_test() {
26+
export VLLM_USE_MODELSCOPE=true
27+
# Do real script test
28+
python3 "${SCRIPT_DIR}/../../examples/offline_inference_npu.py"
29+
}
30+
31+
function quickstart_online_test() {
32+
export VLLM_USE_MODELSCOPE=true
33+
vllm serve Qwen/Qwen2.5-0.5B-Instruct &
34+
wait_url_ready "vllm serve" "localhost:8000/v1/models"
35+
# Do real curl test
36+
curl http://localhost:8000/v1/completions \
37+
-H "Content-Type: application/json" \
38+
-d '{
39+
"model": "Qwen/Qwen2.5-0.5B-Instruct",
40+
"prompt": "Beijing is a",
41+
"max_tokens": 5,
42+
"temperature": 0
43+
}' | python3 -m json.tool
44+
VLLM_PID=$(pgrep -f "vllm serve")
45+
_info "===> Try kill -2 ${VLLM_PID} to exit."
46+
kill -2 "$VLLM_PID"
47+
wait_for_exit "$VLLM_PID"
48+
}
49+
50+
_info "====> Start simple_test"
51+
simple_test
52+
_info "====> Start quickstart_offline_test"
53+
quickstart_offline_test
54+
_info "====> Start quickstart_online_test"
55+
quickstart_online_test

‎tests/e2e/run_doctests.sh

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#!/bin/bash
2+
3+
#
4+
# Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved.
5+
#
6+
# Licensed under the Apache License, Version 2.0 (the "License");
7+
# you may not use this file except in compliance with the License.
8+
# You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
# This file is a part of the vllm-ascend project.
18+
#
19+
20+
set -eo errexit
21+
22+
. $(dirname "$0")/common.sh
23+
24+
_info "====> Start Quickstart test"
25+
. "${SCRIPT_DIR}/doctests/001-quickstart-test.sh"
26+
27+
_info "Doctest passed."

0 commit comments

Comments
 (0)