New hierarchical+parallel speculative decoding method, badly named "Lookahead"

[bullno1]

Paper: https://arxiv.org/abs/2312.12728v2

So instead of only drafting a single sequence, draft multiple sequences.
Then the model can validate more than just one sequence.

They also use a trie to efficiently utilizes the KV cache.
This works because draft sequences will most likely share prefixes, just like in beam search.

The actual draft method is not explicitly mentioned and it can be generic.
Although it seems it's mostly just n-gram lookup because the output is grounded (aka copied from the prompt/data).

I think it is doable with the current API.

[ggerganov]

This is already implemented in the speculative example (#3624)

[bullno1]

I see. Somehow I missed that.

So it didn't improve speed? Is it because of low acceptance rate?