Question: Benchmark tools selection for the two secnario

[Sunt-ing]

Hi, I want to benchmark 2 scenarios:

• Batch mode. For like 10000 requests, given the prompt length and number of tokens to generate, what's the total wall time (throughput) of handling them all?
• Server mode. For like 8 concurrent users, what are the latencies (e.g., the first token and per token latency)?

My hardware is a single machine with (2 CPUs with 20 cores) and (1 GPU with 16 GB of memory). What are the most appropriate benchmark tools?

Thanks for your help!

[phymbert]

Hello, please have a look to server bench framework. It support SSE using xk6-sse

[Sunt-ing]

Thanks for your help! server bench framework indeed looks great for my server mode scenario.

btw, do you have any suggestions for the first scenario (batch mode)? It's non-interactive, less sensitive to latency, but more sensitive to throughput.

[Sunt-ing]

I guess the parallel module is suitable for my first scenario, but its doc is incomplete so I'm not quite sure.