Lack of Documentation

[ThioJoe]

Maybe I'm dumb and missed it, but there seems to be effectively zero documentation on using the windows binaries. I was able to piece together enough to get it running but there's still some things I don't get. For example, it would have been nice to have had a descriptions of which binary to use and what each version means.

And I still haven't found if there's a list of parameters are for launching using main.exe. I found another post where someone used the following: main.exe -i --interactive-first -r "### Human:" --temp 0 -c 2048 -n -1 --ignore-eos --repeat_penalty 1.2 --instruct -m whateverModelFileName

Which works but I want to be able to use my GPU, yet it says offloading 0 layers to GPU, total VRAM used: 0 MB and I can't find anything talking about how to change that.

[trollkotze]

Run "main --help" to see parameter usage explained.
It would definitely be nice to find some more in-depth explanations for many of these parameters. I have a hard time figuring out what effect different sampling options may have. But I guess that's just general machine-learning knowledge that one can read up somewhere and then make sense of.

[skidd-level-100]

Yea its currently very much missing documentation, for instance It says that we have CLblast support (we do) but it is not in the readme on how to compile with CLblast, now this was pretty easy to figure out but still there is a lot of this kind of thing.

also, nice videos they are pretty good!

[klosax]

You can find the documentation for the main binary here:

https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md

[ThioJoe]

Good to know, it looks like it hasn't been updated with anything GPU related yet.

But using --help it shows:
-ngl N, --n-gpu-layers N
number of layers to store in VRAM

To find the number of layers for a particular model, run the program normally using that model and look for something like:
llama_model_load_internal: n_layer = 32

So then in this case I added the parameter --n-gpu-layers 32 and that made it load it into RAM. And it's WAY faster! - On a 4090 it feels like 10x the speed.

Not sure if there's any way to use multiple GPUs like you can on oogabooga

[Green-Sky]

I will let you in on a secret. If you use 33, the output layer is also put into vram. :)

yes, this is underdocumented...