Skip to content

Commit c563a08

Browse files
authored
[CI] Fix nightly benchmark (#1453)
### What this PR does / why we need it? Sometimes the performance benchmark workflow may fail. We hope to add a prompt when the operation fails and not upload the dirty data of the failed operation. --------- Signed-off-by: wangli <wangli858794774@gmail.com>
1 parent 192dbbc commit c563a08

File tree

2 files changed

+10
-6
lines changed

2 files changed

+10
-6
lines changed

.github/workflows/nightly_benchmarks.yaml

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -145,8 +145,8 @@ jobs:
145145
- name: Install elastic_tool
146146
if: github.event_name != 'pull_request'
147147
run: |
148-
pip install escli-tool==0.2.2
149-
148+
pip install escli-tool==0.2.3
149+
150150
- name: Collect pr info from vllm-project/vllm-ascend
151151
if: github.event_name != 'pull_request'
152152
run: |
@@ -176,24 +176,28 @@ jobs:
176176
commit_time=$(git show -s --format=%cd $commit_hash --date=iso-strict)
177177
commit_time_no_tz=${commit_time::19}
178178
pip install -e .
179-
179+
180180
echo "------------------------"
181181
echo "commit_id: $commit_id"
182182
echo "commit_title: $commit_title"
183183
echo "commit_time: $commit_time_no_tz"
184184
echo "vllm branch: ${{ matrix.vllm_branch }}"
185185
echo "vllm-ascend branch: ${{ matrix.vllm_ascend_branch }}"
186186
echo "------------------------"
187-
187+
188188
cd /github/home
189-
bash benchmarks/scripts/run-performance-benchmarks.sh
189+
ERROR_MSG=""
190+
if ! bash benchmarks/scripts/run-performance-benchmarks.sh; then
191+
ERROR_MSG="Benchmark failed to run"
192+
fi
190193
# send the result to es
191194
escli add --vllm_branch ${{ matrix.vllm_branch }} \
192195
--vllm_ascend_branch ${{ matrix.vllm_ascend_branch }} \
193196
--commit_id $commit_id \
194197
--commit_title "$commit_title" \
195198
--created_at "$commit_time_no_tz" \
196199
--res_dir ./benchmarks/results \
200+
--error $ERROR_MSG \
197201
--extra_feat '{"VLLM_USE_V1": "${{ matrix.vllm_use_v1 }}"}'
198202
rm -rf ./benchmarks/results
199203
cd -

benchmarks/scripts/run-performance-benchmarks.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
2-
2+
set -e
33

44
check_npus() {
55
# shellcheck disable=SC2155

0 commit comments

Comments
 (0)