How to convert the Llama-3.1 & 3.2 models into gguf compatible format?

[ShaileshSardaTTL]

Any Idea?

[pseelam02]

Have you tried using the one's that are already on huggingface?

https://huggingface.co/QuantFactory/Llama-3.2-3B-Instruct-GGUF

[SteelPh0enix]

One way is downloading the model from official HuggingFace repository, and using the convert_hf_to_gguf.py script, converting it to GGUF. Then, you can quantize it with llama-quantize

I've made a simple bash function for easier usage of convert_hf_to_gguf.py:

function llama-quantize-raw-model() {
  local base_model_dir=$1
  local output_quantization=${2:-auto}
  local output_gguf_dir=${3:-.}

  # base_model_dir should point to a repository, so dir's name should be model's name
  local model_name=$(basename $base_model_dir)

  if [ ! -d "$base_model_dir" ]; then
    echo "Error: Model directory '$base_model_dir' does not exist."
    return 1
  fi

  # Run the conversion command
  python $LLAMA_CPP_PATH/convert_hf_to_gguf.py --outtype $output_quantization --outfile $output_gguf_dir/$model_name.$output_quantization.gguf $base_model_dir

  # Check if the conversion was successful
  if [ $? -eq 0 ]; then
    echo "Model '$model_name' successfully quantized to $output_quantization format and saved as $output_gguf_dir/$model_name.$output_quantization.gguf"
  else
    echo "Error: Failed to quantize model '$base_model_dir'."
  fi
}