How to load llama-3.1?

[opendeluxe]

The README says

Typically finetunes of the base models below are supported as well.

LLaMA 🦙
LLaMA 2 🦙🦙
LLaMA 3 🦙🦙🦙

So they are supported, nice. How can I apply these models to use with llama.cpp?

All other models in that list have a link to download their GGUF, but not for LLaMa. Dows this mean it is somehow integrated without further download being required?

[dspasyuk]

@opendeluxe you can get it from https://huggingface.co/QuantFactory/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main and run it like so in conversation mode:

../llama.cpp/llama-cli --model ./models/Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf --n-gpu-layers 33 --ctx_size 512 --temp 0.3 -fa --top_k 10 --cnv --chat-template llama3

you can also use llama-server or llama.cui https://github.com/dspasyuk/llama.cui

[opendeluxe]

../llama.cpp/llama-cli --model ./models/Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf --n-gpu-layers 33 --ctx_size 512 --temp 0.3 -fa --top_k 10 --cnv --chat-template llama3

Thank you @dspasyuk! How have you figured out a proper value for "--n-gpu-layers"? Is there a formular like maximum VRAM divide by something or so?

[wooooyeahhhh]

@opendeluxe try to look at your dedicated gpu's memory usage and increase the number of layers until you get near the maximum. If you have enough vram for everything then you can just max out layers to something like 999 and it will just offload everything automatically

[dspasyuk]

../llama.cpp/llama-cli --model ./models/Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf --n-gpu-layers 33 --ctx_size 512 --temp 0.3 -fa --top_k 10 --cnv --chat-template llama3

Thank you @dspasyuk! How have you figured out a proper value for "--n-gpu-layers"? Is there a formular like maximum VRAM divide by something or so?`

When llama-cli boots you should see a log line something like this: llm_load_tensors: offloaded 29/29 layers to GPU that means only 29 layers exist in your model and all ot them are loaded to one GPU. The GPU memory usage, however, will depend on quantization, the context windows that is available/used in the model, flash attention and other parameters. You can of course use what @wooooyeahhhh suggested if you have ~8 Gb++ for 8B-llama3.1 model but if you change --ctx_size to 0 you will need much more VRAM to accommodate larger context window