difference in memory requirement for ollama 3.1-8B and same model quantized using Q4_K_M

[amygbAI]

i quantized the 3.1-8B instruct model and the size is as expected at ~5GB ( used Q4_K_M )
1 ubuntu ubuntu 4920738944 Jul 31 11:17 ../Meta-Llama-3.1-8B-Instruct/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

but when i run this using
./llama-cli -m ../Meta-Llama-3.1-8B-Instruct/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -p "how many medals has simone biles won so far" -n 128

its trying to reserve 16 GB on the server !!
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 17179869216
llama_kv_cache_init: failed to allocate buffer for kv cache
llama_new_context_with_model: llama_kv_cache_init() failed for self-attention cache
llama_init_from_gpt_params: error: failed to create context with model '../Meta-Llama-3.1-8B-Instruct/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf'

i used ollama's container and it ran perfectly fine ( and quite accurately ) on the same 8GB RAM .. can someone please lmk whats going on here ?

[dspasyuk]

@amygbAI since you use --ctx_size:0 llama.cpp will try to set context window to 128k you need to set the limit to something like 8000 tokens and enable flash attention -fa
so:
./llama-cli -m ../Meta-Llama-3.1-8B-Instruct/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --ctx_size 8000 -fa -p "how many medals has simone biles won so far" -n 128

[amygbAI]

thanks so much ..works like a charm .. if i am seeing hallucinations, would u suggest i try different quantization flags ?

[dspasyuk]

@amygbAI I would suggest to dial down temperature and --top_k to less than 0.3 and 10 respectively. You will need to use proper prompt template or use --in-prefix --in-suffix to set chat template keywords and I would ask model to be what you want it to be for example, this should result in the proper answer and no garbage:

../llama.cpp/llama-cli --model ../../models/Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf --n-gpu-layers 33 -b 2048 --ctx_size 512 --temp 0.3 -fa --top_k 10 -p "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a pirate large language model. Your purpose is to assist users by providing information, answering questions based on the data you were trained on. Only answer if you know the answer, otherwise say that you do not know. Answer as a pirate.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHow many medals has Simone Biles won so far?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n" -n 128

[amygbAI]

good sir .. you rock 👍 .. this is a great start for me to start playing around .. cant thank you enough