Can't use model after I convert it to gguf

[flitzcore]

I converted this model to gguf :https://huggingface.co/indonlp/cendol-llama2-13b-merged-chat
This model based on llama 13b so I thought that it should work with llama.cpp

I use this code to convert it:

from huggingface_hub import snapshot_download
model_id="indonlp/cendol-llama2-13b-merged-chat"indonlp/cendol-llama2-13b-merged-chat
snapshot_download(repo_id=model_id, local_dir="",
                  local_dir_use_symlinks=False, revision="main")
!python llama.cpp/convert.py indonlp/cendol-llama2-13b-merged-chat \
  --outfile cendol-llama2-13b-merged-chat.gguf \
  --outtype q8_0 \
  --vocab-type hfft --pad-vocab

it run successfully and then I push it to hf repo. Later I want to use it with LangChain, so I try:

from llama_cpp import Llama
from langchain.llms import LlamaCpp
llm = LlamaCpp(model_path=model_path,
            temperature=0.2,
            repeat_penalty=1.2,
            max_tokens=2000,
            top_p=0.5,
            n_gpu_layers=-1,
               n_ctx=2048)

but it return

llama_model_load: error loading model: unable to allocate backend buffer
llama_load_model_from_file: failed to load model
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
[<ipython-input-23-340494cdcc46>](https://localhost:8080/#) in <cell line: 3>()
     1 from llama_cpp import Llama
     2 from langchain.llms import LlamaCpp
----> 3 llm = LlamaCpp(model_path='/content/cendol-llama2-13b-merged-chat.gguf',
     4             temperature=0.2,
     5             repeat_penalty=1.2,

[/usr/local/lib/python3.10/dist-packages/pydantic/v1/main.py](https://localhost:8080/#) in __init__(__pydantic_self__, **data)
   339         values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
   340         if validation_error:
--> 341             raise validation_error
   342         try:
   343             object_setattr(__pydantic_self__, '__dict__', values)

ValidationError: 1 validation error for LlamaCpp
__root__
 Could not load Llama model from path: /content/cendol-llama2-13b-merged-chat.gguf. Received error Failed to load model from file: /content/cendol-llama2-13b-merged-chat.gguf (type=value_error)

I have tried with some other model but all give the same error. Does anyone know what caused this error?

[flitzcore]

Edit:
At first I thought that this caused by q_8 quantization. So I tried with f32 and still the same issue. This is the full error code:

llama_model_loader: loaded meta data with 21 key-value pairs and 483 tensors from /content/Bahasa-4b-chat-f32.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = Bahasalab
llama_model_loader: - kv   2:                           llama.vocab_size u32              = 151936
llama_model_loader: - kv   3:                       llama.context_length u32              = 32768
llama_model_loader: - kv   4:                     llama.embedding_length u32              = 2560
llama_model_loader: - kv   5:                          llama.block_count u32              = 40
llama_model_loader: - kv   6:                  llama.feed_forward_length u32              = 6912
llama_model_loader: - kv   7:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   8:                 llama.attention.head_count u32              = 20
llama_model_loader: - kv   9:              llama.attention.head_count_kv u32              = 20
llama_model_loader: - kv  10:     llama.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  11:                       llama.rope.freq_base f32              = 5000000.000000
llama_model_loader: - kv  12:                          general.file_type u32              = 0
llama_model_loader: - kv  13:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  14:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  15:                      tokenizer.ggml.scores arr[f32,151936]  = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  16:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  17:                      tokenizer.ggml.merges arr[str,151291]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 151643
llama_model_loader: - kv  20:                    tokenizer.chat_template str              = {% set system_message = 'You are a he...
llama_model_loader: - type  f32:  483 tensors
llm_load_vocab: missing pre-tokenizer type, using: 'default'
llm_load_vocab:                                             
llm_load_vocab: ************************************        
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!        
llm_load_vocab: CONSIDER REGENERATING THE MODEL             
llm_load_vocab: ************************************        
llm_load_vocab:                                             
llm_load_vocab: special tokens definition check successful ( 293/151936 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 151936
llm_load_print_meta: n_merges         = 151291
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 2560
llm_load_print_meta: n_head           = 20
llm_load_print_meta: n_head_kv        = 20
llm_load_print_meta: n_layer          = 40
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 2560
llm_load_print_meta: n_embd_v_gqa     = 2560
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 6912
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 5000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 13B
llm_load_print_meta: model ftype      = all F32
llm_load_print_meta: model params     = 3.95 B
llm_load_print_meta: model size       = 14.72 GiB (32.00 BPW) 
llm_load_print_meta: general.name     = Bahasalab
llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token        = 151643 '<|endoftext|>'
llm_load_print_meta: LF token         = 148848 'ÄĬ'
llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
llm_load_tensors: ggml ctx size =    0.46 MiB
llama_model_load: error loading model: unable to allocate backend buffer
llama_load_model_from_file: failed to load model
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
[<ipython-input-9-99ee3faf99c1>](https://localhost:8080/#) in <cell line: 3>()
      1 from llama_cpp import Llama
      2 from langchain.llms import LlamaCpp
----> 3 llm = LlamaCpp(model_path=model_path,
      4             temperature=0.2,
      5             repeat_penalty=1.2,

[/usr/local/lib/python3.10/dist-packages/pydantic/v1/main.py](https://localhost:8080/#) in __init__(__pydantic_self__, **data)
    339         values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
    340         if validation_error:
--> 341             raise validation_error
    342         try:
    343             object_setattr(__pydantic_self__, '__dict__', values)

ValidationError: 1 validation error for LlamaCpp
__root__
  Could not load Llama model from path: /content/Bahasa-4b-chat-f32.gguf. Received error Failed to load model from file: /content/Bahasa-4b-chat-f32.gguf (type=value_error)

[compilade]

llama_model_load: error loading model: unable to allocate backend buffer

Possibly not enough RAM?

llm_load_print_meta: model params = 3.95 B

This is VERY weird. Some tensors are definitely missing from the converted model file. (EDIT: nevermind, that's a 4B model (Bahasa-4b-chat), not the 13B model mentioned before)

Can you try converting the model with convert-hf-to-gguf.py instead of convert.py to see if the problem persists?

Using something like

$ python3 llama.cpp/convert-hf-to-gguf.py --outfile cendol-llama2-13b-merged-chat.{ftype}.gguf --outtype q8_0 indonlp/cendol-llama2-13b-merged-chat