How to load simple GGUF models?

[paulocoutinhox]

Hi,

Im trying load two GGUF models, but im getting error:

// prompt and model
const std::string prompt = "Write a simple Python code";
const std::string modelPath = "codellama-7b.Q2_K.3.gguf";

// params
gpt_params params;
params.model = modelPath;
params.prompt = "### Instruction:" + prompt + "\n\n### Response:";
params.numa = 32;

// init LLM
llama_backend_init(params.numa);

// initialize the model
llama_model_params modelParams = llama_model_default_params();
llama_model *model = llama_load_model_from_file(params.model.c_str(), modelParams);

Error:

gguf_init_from_file: tensor 'token_embd.weight' number of elements (131137536) is not a multiple of block size (0)
error loading model: llama_model_loader: failed to load model from /Users/paulo/Downloads/codellama-7b.Q2_K.3.gguf

In their website (HF) they use this line of command:

/main -ngl 32 -m codellama-7b.q4_K_M.gguf --color -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "{prompt}"

How to convert that params to C++ code with llama.cpp?

Models:

• https://huggingface.co/TheBloke/CodeLlama-7B-GGUF
• https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF

Thanks.

[harryaskham]

I'm having the same issue on an Oracle aarch64 vm with the same mistral gguf. I had to set LLAMA_NO_K_QUANTS=1 to build successfully but I get the same runtime error, whether I use a Q4_K_M or a Q5_0 variant.

[paulocoutinhox]

Hi guys,

Any news to help us run gguf models with llama.cpp?

Thanks.

[ianscrivener]

So just confirming,

1. if you dowload the latest appropriate release - https://github.com/ggerganov/llama.cpp/releases
2. and get a GGUF v2 from TheBloke here https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/tree/main
3. and you run ./main -m xxxx.gguf -ngl 32 -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e

it fails?

if so, what is the error message?
what OS, CPU & GPU are you using?

[paulocoutinhox]

No, when i compile from master/release i get this warning above when compile, but when i run it from cmd line it works:

/Users/paulo/Developer/workspaces/cpp/llama.cpp/ggml-metal.m:127:25: warning: unused variable 'msl_library_source' [-Wunused-const-variable]
static NSString * const msl_library_source = @"see metal.metal";

Building a website can be done in 10 simple steps:
Step 1: Find out what you want to say to your visitors and write it down. This is going to become your web content.

Step 2: Find out how to get the best from that content. Think about how you will lay it out on your page, what design suits you the best or how much of it will fit nicely on one page. And try out different ideas and make your changes in real time. This is what the web-designer allows you to do with his online editor.

Step 3: Write a short text that tells visitors who are visiting your site for the first time, where your interests lie. Tell them why they should visit you again and maybe recommend your website to their friends. You want to give them an incentive to become a regular.

Step 4: Go to Google and see what is being said about your topic. There you will find out who else in the world cares about the same things that interest you, and how many other people are interested in the same issues on the web. You will also find out where they hang out online. This way you can make sure not to be alone!

Step 5: Choose the keywords (search words) for your site by reading again through step 4. Use them all over your site and write down a list. They are what your visitors type into Google search when looking for information about what you care about. The more often your website appears in this list the higher is the chance that you will get more visitors on your website!

Step 6: Get your website online. You can either pay for it or make it for free. If you want to be taken seriously, you should not pick a free web address like xxx.tripod.com, but buy a domain name, which is the easy part of it. And remember that you need a good webhoster, who guarantees you a fast loading time and plenty of storage space
llama_print_timings:        load time =    7646.74 ms
llama_print_timings:      sample time =      35.20 ms /   400 runs   (    0.09 ms per token, 11362.99 tokens per second)
llama_print_timings: prompt eval time =     177.90 ms /    20 tokens (    8.90 ms per token,   112.42 tokens per second)
llama_print_timings:        eval time =   15416.50 ms /   399 runs   (   38.64 ms per token,    25.88 tokens per second)
llama_print_timings:       total time =   15692.71 ms
ggml_metal_free: deallocating
Log end

As i sad, the problem is when i try run the basic sample trying to load the model from the simple code from example in this repository.

Example:

// prompt and model
const std::string prompt = "Write a simple Python code";
const std::string modelPath = "codellama-7b.Q2_K.3.gguf";

// params
gpt_params params;
params.model = modelPath;
params.prompt = "### Instruction:" + prompt + "\n\n### Response:";
params.numa = 32;

// init LLM
llama_backend_init(params.numa);

// initialize the model
llama_model_params modelParams = llama_model_default_params();
llama_model *model = llama_load_model_from_file(params.model.c_str(), modelParams);

Extracted from:
https://github.com/ggerganov/llama.cpp/blob/master/examples/simple/simple.cpp

What i need do, or what params need be set to llama.cpp understand my model like in command line?

Thanks.

[harryaskham]

I should say at this stage I was +1ing the error in the original message:

gguf_init_from_file: tensor 'token_embd.weight' number of elements (131137536) is not a multiple of block size (0)
error loading model: llama_model_loader: failed to load model from /Users/paulo/Downloads/codellama-7b.Q2_K.3.gguf

I'll start a new thread on this issue, it seems to be unrelated to the rest of the discussion.

[ianscrivener]

That build error can be ignored...

Is that model path correct? So the model is in the same directory as the binary...

[paulocoutinhox]

Yes, sure o.o

ls -lah ~/Downloads/codellama-7b.Q2_K.3.gguf
-rw-r--r--@ 1 paulo  staff   2.6G Oct 22 05:06 /Users/paulo/Downloads/codellama-7b.Q2_K.3.gguf