Long load time. How to reduce it?

[dpieve]

I'd like to know if I can reduce the load time to be < 1000ms with my setup.
It is taking too long for it to start giving me an answer.

Any help would be greatly appreciated

Commands:

make clean
make -j LLAMA_CUDA=1
./main -m models/llama-2-7b/llama-2-7b.Q2_K.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e -t 8

No erros in the output.
I tried without the -t 8.
I am using the main sample in the README as a reference.
https://github.com/ggerganov/llama.cpp/blob/10ceba354a3b152ff425e9fa97f9caaef99a46b1/README.md?plain=1#L221

Results:

llama_print_timings:        load time =  113368.34 ms
llama_print_timings:      sample time =      23.66 ms /   400 runs   (    0.06 ms per token, 16905.46 tokens per second)
llama_print_timings: prompt eval time =     903.48 ms /    19 tokens (   47.55 ms per token,    21.03 tokens per second)
llama_print_timings:        eval time =   54160.17 ms /   399 runs   (  135.74 ms per token,     7.37 tokens per second)
llama_print_timings:       total time =   55589.12 ms /   418 tokens

Currently, the load time = 113368.34 ms and I would like it to be < 1000ms. Like in the example:
https://github.com/ggerganov/llama.cpp/blob/10ceba354a3b152ff425e9fa97f9caaef99a46b1/README.md?plain=1#L291

Setup:

• Model TheBloke/Llama-2-7B-GGUF, downloaded hf TheBloke as suggested in the discussions here.

• I'm running it on Ubuntu-22.04 (Windows 11 - WSL)

• RTX3070, 16gb RAM, AMD Ryzen 7 5800H

Output:

make output

``` I llama.cpp build info: I UNAME_S: Linux I UNAME_P: x86_64 I UNAME_M: x86_64 I CFLAGS: -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_OPENMP -DGGML_USE_LLAMAFILE -DGGML_USE_CUDA -I/usr/local/cuda/include -I/usr/local/cuda/targets/x86_64-linux/include -DGGML_CUDA_USE_GRAPHS -std=c11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -fopenmp -Wdouble-promotion I CXXFLAGS: -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread -fopenmp -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_OPENMP -DGGML_USE_LLAMAFILE -DGGML_USE_CUDA -I/usr/local/cuda/include -I/usr/local/cuda/targets/x86_64-linux/include -DGGML_CUDA_USE_GRAPHS I NVCCFLAGS: -std=c++11 -O3 -use_fast_math --forward-unknown-to-host-compiler -arch=native -DGGML_CUDA_DMMV_X=32 -DGGML_CUDA_MMV_Y=1 -DK_QUANTS_PER_ITERATION=2 -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 I LDFLAGS: -lcuda -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/usr/lib64 -L/usr/local/cuda/targets/x86_64-linux/lib -L/usr/lib/wsl/lib I CC: cc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0 I CXX: c++ (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0 I NVCC: Build cuda_12.5.r12.5/compiler.34177558_0 ```

main output

./main -m models/llama-2-7b/llama-2-7b.Q2_K.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e -t 8

Log start
main: build = 3109 (7a16ce7d)
main: built with cc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0 for x86_64-linux-gnu
main: seed  = 1718007210
llama_model_loader: loaded meta data with 19 key-value pairs and 291 tensors from models/llama-2-7b/llama-2-7b.Q2_K.gguf (version GGUF V2)
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = LLaMA v2
llama_model_loader: - kv   2:                       llama.context_length u32              = 4096
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 10
llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  17:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  18:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q2_K:   65 tensors
llama_model_loader: - type q3_K:  160 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 259
llm_load_vocab: token to piece cache size = 0.1684 MB
llm_load_print_meta: format           = GGUF V2
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 4096
llm_load_print_meta: n_embd_v_gqa     = 4096
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q2_K - Medium
llm_load_print_meta: model params     = 6.74 B
llm_load_print_meta: model size       = 2.63 GiB (3.35 BPW) 
llm_load_print_meta: general.name     = LLaMA v2
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3070 Laptop GPU, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.15 MiB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/33 layers to GPU
llm_load_tensors:        CPU buffer size =  2694.32 MiB
.................................................................................................
llama_new_context_with_model: n_ctx      = 4096
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  CUDA_Host KV buffer size =  2048.00 MiB
llama_new_context_with_model: KV self size  = 2048.00 MiB, K (f16): 1024.00 MiB, V (f16): 1024.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.12 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   365.25 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    24.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 356

system_info: n_threads = 8 / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 
sampling: 
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order: 
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature 
generate: n_ctx = 4096, n_batch = 2048, n_predict = 400, n_keep = 1


 Building a website can be done in 10 simple steps:
Step 1: Choose a Domain Name
The first step in creating a website is choosing a domain name. The domain name is the address of your website.
To choose a domain name, first, decide on a name for your website. You can either choose a name that is unique to your business or use a generic name. Once you decide on the name, check to see if the name is available. If the name is not available, you can choose a different name.
When choosing a domain name, make sure the domain name is easy to remember. You can also choose a domain name that is easy to spell.
Once you have chosen a domain name, you will need to register the domain name. To register the domain name, you will need to pay a registration fee. The registration fee is usually around $10 to $20.
Once you have registered the domain name, you will need to create a website for the domain name. You can create a website by using a content management system or a website builder.
Step 2: Decide on Your Website’s Purpose
The next step is to decide on your website’s purpose. There are many different purposes for websites, but some of the most common include:
-Selling products or services
-Providing information
-Offering a service
-Providing entertainment
-Offering a community
-Providing an experience
When deciding on your website’s purpose, it’s important to keep in mind who your target audience is. What are they looking for? How can you provide them with what they need?
Once you’ve decided on your website’s purpose, it’s time to start thinking about the content you’ll need to create. This content can include anything from text and images to videos and audio files.
Step 3: Decide on a Name for Your Business
The third step is to decide on
llama_print_timings:        load time =  113368.34 ms
llama_print_timings:      sample time =      23.66 ms /   400 runs   (    0.06 ms per token, 16905.46 tokens per second)
llama_print_timings: prompt eval time =     903.48 ms /    19 tokens (   47.55 ms per token,    21.03 tokens per second)
llama_print_timings:        eval time =   54160.17 ms /   399 runs   (  135.74 ms per token,     7.37 tokens per second)
llama_print_timings:       total time =   55589.12 ms /   418 tokens
Log end

Additional:

I tried using different models such as Phi-3-mini. However, it took even longer.
I tested ollama run llama3 and phi3, and my computer runs it just fine. Basically, I am trying to achieve similar results but using llama.cpp.