Does anyone have FP8 vs. FP16 benchmark results on H100?

[ajtejankar]

I am benchmarking FP8 vs. FP16 on H100 and I don't see much of an improvement. The repro script (uses vllm's benchmarking code) and results are below. I am worried that I may be doing something weird so just want to confirm.

function benchmark_throughput() {
    python benchmark_throughput.py --model "$1" --input-len "$2" --output-len "$3" 2>&1  | tail -n 1 | awk '{ print $2 }'
}

inp_lens=( 2 8 32 128 )
out_lens=( 2 8 32 128 512 )

for inp_len in "${inp_lens[@]}"
do
    for out_len in "${out_lens[@]}"
    do
        res_fp16="$(benchmark_throughput 'mistralai/Mistral-7B-Instruct-v0.3' ${inp_len} ${out_len})"
        res_fp8="$(benchmark_throughput 'neuralmagic/Mistral-7B-Instruct-v0.3-FP8' ${inp_len} ${out_len})"
        python -c "print(f'Inp len: {${inp_len}:5d} | Out len: {${out_len}:5d} | req/sec speedup (FP8/FP16) : ${res_fp8}/${res_fp16} = {${res_fp8}/${res_fp16}:.2f}')"
    done
done

The results are:

Inp len:     2 | Out len:     2 | req/sec speedup (FP8/FP16) : 1867.09/1838.70 = 1.02
Inp len:     2 | Out len:     8 | req/sec speedup (FP8/FP16) : 674.62/597.64 = 1.13
Inp len:     2 | Out len:    32 | req/sec speedup (FP8/FP16) : 193.94/177.22 = 1.09
Inp len:     2 | Out len:   128 | req/sec speedup (FP8/FP16) : 47.48/42.44 = 1.12
Inp len:     2 | Out len:   512 | req/sec speedup (FP8/FP16) : 9.50/8.82 = 1.08
Inp len:     8 | Out len:     2 | req/sec speedup (FP8/FP16) : 1479.89/1151.98 = 1.28
Inp len:     8 | Out len:     8 | req/sec speedup (FP8/FP16) : 607.33/519.54 = 1.17
Inp len:     8 | Out len:    32 | req/sec speedup (FP8/FP16) : 186.48/165.16 = 1.13
Inp len:     8 | Out len:   128 | req/sec speedup (FP8/FP16) : 46.81/42.34 = 1.11
Inp len:     8 | Out len:   512 | req/sec speedup (FP8/FP16) : 9.42/8.72 = 1.08
Inp len:    32 | Out len:     2 | req/sec speedup (FP8/FP16) : 737.90/546.38 = 1.35
Inp len:    32 | Out len:     8 | req/sec speedup (FP8/FP16) : 426.23/347.88 = 1.23
Inp len:    32 | Out len:    32 | req/sec speedup (FP8/FP16) : 165.10/141.87 = 1.16
Inp len:    32 | Out len:   128 | req/sec speedup (FP8/FP16) : 44.67/39.64 = 1.13
Inp len:    32 | Out len:   512 | req/sec speedup (FP8/FP16) : 9.27/8.55 = 1.08
Inp len:   128 | Out len:     2 | req/sec speedup (FP8/FP16) : 229.69/161.12 = 1.43
Inp len:   128 | Out len:     8 | req/sec speedup (FP8/FP16) : 186.49/134.62 = 1.39
Inp len:   128 | Out len:    32 | req/sec speedup (FP8/FP16) : 107.46/85.82 = 1.25
Inp len:   128 | Out len:   128 | req/sec speedup (FP8/FP16) : 37.75/32.54 = 1.16
Inp len:   128 | Out len:   512 | req/sec speedup (FP8/FP16) : 8.59/7.93 = 1.08