Feat/max error rate - continued #238
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@markurtz This is a continuation of the work done in #171 by @markVaykhansky with the review comments fixed and a few additions.
The main differences to the original PR are:
run_stats
:window_error_rate
,termination_reason
andstatus
.GUIDELLM__ERROR_CHECK_WINDOW_SIZE
completed requests (success/error) if the ratio of errors (or error count if--max-error
>1) in those requests is greater than--max-error
.E2E tests
By default, these will check if the vLLM simulator is available in the local environment, if not - they are skipped and log a warning with the command required to build the vLLM simulator.
The vLLM simulator is built from source in the
tests/e2e/vllm-sim.Dockerfile
and extracted tobin/llm-d-inference-sim
.The tests use a class
VllmSimServer
that wraps the vLLM simulator withstart
andstop
methods, then they call theguidellm
command and assert the values generated in the output report.The 2 test modules have varying run times:
test_successful_benchmark
(2 tests) - 14.12stest_max_error_benchmark
(1 test) - 15.98sThe max error test takes longer since it allows guidellm to start sending requests before killing the vLLM sim, and then waits for the guidellm command to stop gracefully.
The successful tests share a single vLLM sim instance, hence are more efficient.
The new
run_stats
fieldswindow_error_rate
- the error rate within the error check window at the time of run completion (may be a partial window if the--max-error
was not reached.). This is the value that would indicate why the--max-error
was reached, as opposed to theerror_rate
field that is global and does not necessarily reflect that the--max-error
was reached.termination_reason
- either"max_seconds_reached"
,"max_requests_reached"
,"max_error_reached"
or"interrupted"
. This is important for future features such as over-saturation termination or target metric margin termination.status
- either"success"
,"error"
or"interrupted"
. This is a simplification of thetermination_reason
field, will help differentiate success states from error states, for example over-saturation termination is an error state and target metric margin termination is a success state."Chunked" error checking
The precise logic is as follows:
GUIDELLM__ERROR_CHECK_WINDOW_SIZE
, check ifmax_error
is reached.max_error
>1 - check if the accumulated amount of errored requests >max_error
. Otherwise - check if the accumulated errored requests divided by the accumulated completed requests >max_error
.max_error
is reached, break and shutdown.max_error
is not reached, reset the accumulated amounts to 0 and keep going.This is simple and handles all different cases nicely.