Precise decoding progress reporting

[vladfaust]

A llama_decode call may take considerable amount of time, and I'd like to report decoding progress in UI.

What I'm doing now:

1. Set ggml_backend_sched_set_eval_callback.
2. Call llama_decode.
3. Upon receiving a callback with ask = false, I increase some cur_token counter, as I was naively thinking that the number of calls would be equivalent to the decoded prompt's token count.

However, I'm getting the ratio of calls/prompt length to be anywhere from 2 to 3, presumably depending on if the KV cache/prompt size, yet the ratio seems to be consistent in the same circumstances. I tried reading ggml_backend_sched_compute_splits implementation in ggml-backend.c, and I found multiple callback_eval calls. Unfortunately, my knowledge is not enough to determine the relation precisely, which leads to this imperfect approximation.

It'd be great if there was a way to report the decoding progress precisely. What's the math behind these calculations? Or maybe there is a way to filter the callback using the t argument somehow?

Any help is appreciated!

[vladfaust]

FYI, when I changed my logic to always return false from the callback after reading #6576, it fires exactly 2 * token count times (keys + values, I assume?), given that the logits are only calculated for the latest token in the batch. LGTM.