Skip to content

Sweep mode with --max-seconds fails ungracefully if calculated RPS is too low #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sjmonson opened this issue Apr 15, 2025 · 0 comments
Assignees

Comments

@sjmonson
Copy link
Collaborator

If sweep mode calculates an RPS that results in less then one request being sent out in --max-second range then there is a failure during processing.

Traceback (most recent call last):
  File "/opt/app-root/bin/guidellm", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/guidellm/src/guidellm/__main__.py", line 242, in benchmark
    asyncio.run(
  File "/usr/lib64/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/src/guidellm/src/guidellm/benchmark/entrypoints.py", line 105, in benchmark_generative_text
    async for result in benchmarker.run(
  File "/src/guidellm/src/guidellm/benchmark/benchmarker.py", line 243, in run
    benchmark: BenchmarkT = aggregator.compile()
                            ^^^^^^^^^^^^^^^^^^^^
  File "/src/guidellm/src/guidellm/benchmark/aggregator.py", line 606, in compile
    return GenerativeBenchmark.from_stats(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/guidellm/src/guidellm/benchmark/benchmark.py", line 694, in from_stats
    start_time = min(req.start_time for req in total)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: min() iterable argument is empty
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Backlog
Development

No branches or pull requests

1 participant