Skip to content

Margin Of Error (MOE) stopping #244

@AlonKellner-RedHat

Description

@AlonKellner-RedHat

Given that #238 will land and that #243 will turn into a PR and land as well, it would be incredible to have target metric MOE stopping.
Basically this feature is 2 new options:

  • --target-metric - the metric to use as the target for MOE stopping, of the form ["all", "ttft", "ttft-p90", "itl-median"....] (default: "all")
  • --target-moe - the allowed error in the metric measurement (in percentages) as reflected in the MOE of that metric.
    In a benchmark run, when the MOE value of the target metric reaches the target MOE - the benchmark will end with the termination reason "target_moe_reached".

This feature will include a simple ETA value.
Essentially, calculate the amount of samples required to reach the MOE (assuming the average sample weight consists), then use the RPS to extrapolate time.

A feature like this would make guidellm much faster and reliable for most use-cases.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions