You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Feature Description
Please add a --max-error-rate parameter which would stop the execution of a single benchmark.
Why is this needed?
When running benchmarks we want to fail as early as possible in order to save costs of either GPU machine time or remote API calls s.a ChatGPT API.
Further Description
From benchmarks that we've been running a reasonable default value would be 0.05 i.e 5% max error rate.
Also, if a benchmark fails due to reaching the max error rate it should be reflected in the report generated by GuideLLM.
The text was updated successfully, but these errors were encountered:
Feature Description
Please add a
--max-error-rate
parameter which would stop the execution of a single benchmark.Why is this needed?
When running benchmarks we want to fail as early as possible in order to save costs of either GPU machine time or remote API calls s.a ChatGPT API.
Further Description
From benchmarks that we've been running a reasonable default value would be
0.05
i.e5%
max error rate.Also, if a benchmark fails due to reaching the max error rate it should be reflected in the report generated by GuideLLM.
The text was updated successfully, but these errors were encountered: