Sampling settings can dramatically change generation throughput

[kalomaze]

It looks like there's still some low hanging fruit when it comes to unoptimized sampler logic in llama.cpp.

30% of the time is spent sampling for q8_0 Mistral 7b when generating 1024 tokens! (that is, if you use topk=0 or topk=32000 to avoid restricting the initial candidate set)

[cmp-nct]

update:

I ran a test and I have a sampling time of 1.6ms/tk on top-k 32000 and 0.1ms on top-k 32
So that's 10 times slower, but you were 100 times slower.

Give it a try with this PR, and use a top-k of 10k for the test (at vocab size it won't help)
#5085

[ggerganov]

What is the purpose of using top-k with anything more than say ~40?
The idea of top-k is to select the top tokens, not to sort all of them