Skip to content

[Feature Request] Max error rate parameter #105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
markVaykhansky opened this issue Apr 8, 2025 · 0 comments
Open

[Feature Request] Max error rate parameter #105

markVaykhansky opened this issue Apr 8, 2025 · 0 comments

Comments

@markVaykhansky
Copy link
Collaborator

Feature Description
Please add a --max-error-rate parameter which would stop the execution of a single benchmark.

Why is this needed?
When running benchmarks we want to fail as early as possible in order to save costs of either GPU machine time or remote API calls s.a ChatGPT API.

Further Description
From benchmarks that we've been running a reasonable default value would be 0.05 i.e 5% max error rate.
Also, if a benchmark fails due to reaching the max error rate it should be reflected in the report generated by GuideLLM.

@rgreenberg1 rgreenberg1 added good first issue Good for newcomers priority-low and removed good first issue Good for newcomers labels May 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Ready
Development

No branches or pull requests

2 participants