Using --n-gpu-layers L --seed N yields a different result compared with --seed N only

[filippobistaffa]

I am testing offloading some layers of the vicuna-13b-v1.5-16k.Q4_K_M.gguf model on the GPU and I noticed that enabling the --n-gpu-layers option changes the result of the model when using the same seed (even if it's still deterministic).

Without GPU offloading:
./main --model "models/vicuna-13b-v1.5-16k.Q4_K_M.gguf" --prompt "The Answer to the Ultimate Question of Life, the Universe, and Everything is" --escape --repeat_penalty 1.1 --ctx-size 4096 --n-predict -1 --temp 0.7 --seed 0
Result:
The Answer to the Ultimate Question of Life, the Universe, and Everything is a reference to the science fiction series The Hitchhiker's Guide to the Galaxy. In the story, a supercomputer named Deep Thought is tasked with finding the answer to this question, which it reveals as 42.

With GPU offloading:
./main --model "models/vicuna-13b-v1.5-16k.Q4_K_M.gguf" --prompt "The Answer to the Ultimate Question of Life, the Universe, and Everything is" --escape --repeat_penalty 1.1 --ctx-size 4096 --n-predict -1 --temp 0.7 --seed 0 --n-gpu-layers 40
Result:
The Answer to the Ultimate Question of Life, the Universe, and Everything is a joke invented by British author Douglas Adams in his science fiction series The Hitchhiker's Guide to the Galaxy. In the book, a supercomputer called Deep Thought calculates the answer as 42, but no one knows what the actual question is. In the story, Deep Thought was built to find the answer to the ultimate question of life, the universe and everything. The character Marvin the Paranoid Android says that the answer is "42" (the number that appears on his badge), but he is often dismissed as being depressed. The concept has since become a meme and a popular culture reference to represent an unknowable or elusive answer, often used in a humorous way.

Does it depend on the model?
Is there a way to obtain the same result, with and without GPU offloading?

[KerfuffleV2]

There are small differences in stuff like rounding that add up. You might also notice difference results using a different type of GPU as well.

Is there a way to obtain the same result, with and without GPU offloading?

It's probably not impossible but there isn't currently a way. Just a note but you also probably can't really depend on results staying exactly the same long term, at least not unless you just never upgrade software. Other changes, such as to sampling could also cause the same seed to produce different results.

[filippobistaffa]

I see, thanks for the explanation.
In any case, it's not so crucial, my goal was to show the performance improvement thanks to GPU offloading (which is not so meaningful if the result is not the same).

[filippobistaffa]

What if I'm using --n-predict -1? The number of generated tokens is not necessarily the same, or am I missing something?

[KerfuffleV2]

Why would you do such a thing if you're trying to benchmark? :) Set a fixed size and use --ignore-eos and the model will generate exactly the number of tokens you requested.

[filippobistaffa]

Well, I'm not exactly benchmarking :) I'm showing how to use llama.cpp to my students, so I would like to show a "nice" response by the model, if I set a limit the response would likely be truncated. At the same time, I would like to show the benefits of GPU offloading.

[KerfuffleV2]

If you use interactive mode and interrupt it (hit ^C twice to exit) you should see some statistics (I'm assuming you're using main). While the absolute time may differ, stuff like the average time to generate a token, tokens per sec, etc should still be meaningful.

Other than that, I'm not sure what to tell you. You might have to choose between benchmarking and the best possible output in a single generation.

[filippobistaffa]

I will go with the second option, it's perfectly fine :)