Failed Vulkan

[okamototk]

I got this error during vk::createInstance:

ggml_backend_vk_reg() -> Error: System error: vk::createInstance: ErrorLayerNotPresent

How can I resolve this problem?

Environment:

• commit 7b891bd (HEAD, tag: b4686)
• MacBook Apple M2 RAM 24G + podman krunkit(+MotenVK)
• Vulkan Backend

Result

During run llama-cli, I got error at vk::createInstance:

bash-5.2# build/bin/llama-cli  -m DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf -p "what is GPU" -no-cnv
ggml_vk_instance_init()
ggml_vulkan: Validation layers enabled
ggml_backend_vk_reg() -> Error: System error: vk::createInstance: ErrorLayerNotPresent

Information for Vulkan Device(GPU0):

bash-5.2# vulkaninfo --summary                                                                                                                              15:42:11 [40/1828]
'DISPLAY' environment variable not set... skipping surface info
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
ERROR: [../src/panfrost/vulkan/panvk_physical_device.c:608] Code 0 : WARNING: panvk is not a conformant vulkan implementation, pass PAN_I_WANT_A_BROKEN_VULKAN_DRIVER=1 if you
 know what you're doing. (VK_ERROR_INCOMPATIBLE_DRIVER)
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
error: XDG_RUNTIME_DIR is invalid or not set in the environment.
==========
VULKANINFO
==========

Vulkan Instance Version: 1.3.296

Instance Extensions: count = 24
-------------------------------                                                                                                                             15:42:11 [25/1828]
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_headless_surface                : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 4
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_wayland_surface                 : extension revision 6
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1

Instance Layers: count = 1
--------------------------
VK_LAYER_MESA_device_select Linux device selection layer 1.3.211  version 1

Devices:
========
GPU0:
        apiVersion         = 1.2.0
        driverVersion      = 24.1.5
        vendorID           = 0x106b
        deviceID           = 0xf010208
        deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
        deviceName         = Virtio-GPU Venus (Apple M2)
        driverID           = DRIVER_ID_MESA_VENUS
        driverName         = venus
        driverInfo         = Mesa 24.1.5
        conformanceVersion = 1.3.0.0
        deviceUUID         = 26b49897-c8c9-cdea-542a-91890ff14f78
        driverUUID         = 05a3cceb-1991-3628-23bd-764bdc583f20
GPU1:
        apiVersion         = 1.3.278
        driverVersion      = 0.0.1
        vendorID           = 0x10005
        deviceID           = 0x0000
        deviceType         = PHYSICAL_DEVICE_TYPE_CPU
        deviceName         = llvmpipe (LLVM 18.1.6, 128 bits)
        driverID           = DRIVER_ID_MESA_LLVMPIPE
        driverName         = llvmpipe
        driverInfo         = Mesa 24.1.5 (LLVM 18.1.6)
        conformanceVersion = 1.3.1.1
        deviceUUID         = 6d657361-3234-2e31-2e35-000000000000
        driverUUID         = 6c6c766d-7069-7065-5555-494400000000

[0cc4m]

You enabled the Validation layer from within the backend during compilation, with GGML_VULKAN_VALIDATE, but you don't have the validation layer installed. If you actually want to use the layer, you need to install it. Otherwise, set the compile flag to OFF.

[slp]

Looks like this binary was built with -DGGML_VULKAN_RUN_TESTS=1. That's also broken for me as of 2eea03d. For doing normal inferencing, building just with -DGGML_VULKAN=1 (on a clean build directory) should work just fine (well, the output may be garbage, but that's a molten-vk problem).

[okamototk]

@slp

Looks like this binary was built with -DGGML_VULKAN_RUN_TESTS=1. That's also broken for me as of 2eea03d. For doing normal inferencing, building just with -DGGML_VULKAN=1 (on a clean build directory) should work just fine (well, the output may be garbage, but that's a molten-vk problem).

---

Thanks, I tried without -DGGML_VULKAN_RUN_TESTS but same error as follow. Could you share

• git commit id for llama.cpp
• MoltenVK version you success to run
• Apple Silicon(M1,M2,M3,M4...)
  ?

Here is failed log just use -DGGML_VULKAN=1

# cmake --build build --target clean
# cmake -DGGML_VULKAN=ON
CMake Warning:
  No source or binary directory provided.  Both will be assumed to be the
  same as the current working directory, but note that this warning will
  become a fatal error in future CMake releases.


-- ccache found, compilation results will be cached. Disable with GGML_CCACHE=OFF.
-- CMAKE_SYSTEM_PROCESSOR: aarch64
-- Including CPU backend
-- ARM detected
-- ARM -mcpu not found, -mcpu=native will be used
cc1: error: unknown value 'native+nodotprod+noi8mm+nosve' for '-mcpu'
cc1: note: valid arguments are: cortex-a34 cortex-a35 cortex-a53 cortex-a57 cortex-a72 cortex-a73 thunderx thunderxt88p1 thunderxt88 octeontx octeontx81 octeontx83 thunderxt81 thunderxt83 ampere1 ampere1a ampere1b emag xgene1 falkor qdf24xx exynos-m1 phecda thunderx2t99p1 vulcan thunderx2t99 cortex-a55 cortex-a75 cortex-a76 cortex-a76ae cortex-a77 cortex-a78 cortex-a78ae cortex-a78c cortex-a65 cortex-a65ae cortex-x1 cortex-x1c neoverse-n1 ares neoverse-e1 octeontx2 octeontx2t98 octeontx2t96 octeontx2t93 octeontx2f95 octeontx2f95n octeontx2f95mm a64fx tsv110 thunderx3t110 neoverse-v1 zeus neoverse-512tvb saphira cortex-a57.cortex-a53 cortex-a72.cortex-a53 cortex-a73.cortex-a35 cortex-a73.cortex-a53 cortex-a75.cortex-a55 cortex-a76.cortex-a55 cortex-r82 cortex-a510 cortex-a520 cortex-a710 cortex-a715 cortex-a720 cortex-x2 cortex-x3 cortex-x4 neoverse-n2 cobalt-100 neoverse-v2 grace demeter generic generic-armv8-a generic-armv9-a
CMake Warning at ggml/src/ggml-cpu/CMakeLists.txt:151 (message):
  Failed to get ARM features
Call Stack (most recent call first):
  ggml/src/CMakeLists.txt:318 (ggml_add_cpu_backend_variant_impl)

It's error and seems need -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=native options for ARM:

# cmake -DGGML_VULKAN=ON -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=native

Then build:

# cmake --build build --config Debug

Run inference but same errror:

# build/bin/llama-cli  -m examples/server/tests/stories15M_MOE-F16.gguf  -p "what is GPU" -no-cnv -ngl 99
...
llama_init_from_model: KV self size  =   27.00 MiB, K (f16):   13.50 MiB, V (f16):   13.50 MiB
llama_init_from_model: Vulkan_Host  output buffer size =     0.12 MiB
llama_init_from_model:    Vulkan0 compute buffer size =    63.06 MiB
llama_init_from_model: Vulkan_Host compute buffer size =     8.57 MiB
llama_init_from_model: graph nodes  = 288
llama_init_from_model: graph splits = 2
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
terminate called after throwing an instance of 'std::out_of_range'
  what():  unordered_map::at
Aborted (core dumped)

[slp]

I suspect cmake didn't actually update the build config. I would try this:

1. Install clang (if this is Fedora, dnf install clang should be enough).
2. Remove the build directory.
3. CC=clang CXX=clang++ cmake -B build -DGGML_VULKAN=1
4. cmake --build build

[okamototk]

@slp Sorry not sooner reply.
After update code to HEAD and build from clean. Abort was stopped. But inference result looks invalid:

$ build/bin/llama-cli  -m examples/server/tests/stories15M_MOE-F16.gguf  -p "hello" -no-cnv -ngl 99 
...
 hello even m g ever lived liveeshesh oppositeakakakakakakakakakakakakakakak corner corner even v careobfullyumpakakakakakakingakineakakakingakakak corner cornerunun...

Here is complete procedure and log:

I update head:

commit 5fa07c2f93c73161bf09ef0b23b5d2686f9a073e (HEAD -> master, origin/master, origin/HEAD)
Author: Johannes G__ler <johannesg@5d6.de>
Date:   Sat Feb 22 12:20:17 2025 +0100

related vulkan package:

# rpm -qa |grep vulkan
vulkan-loader-1.3.275.0-1.fc39.aarch64
mesa-vulkan-drivers-23.3.5-102.aarch64
vulkan-tools-1.3.275.0-1.fc39.aarch64
vulkan-headers-1.3.275.0-1.fc39.noarch
vulkan-loader-devel-1.3.275.0-1.fc39.aarch64
vulkan-utility-libraries-devel-1.3.275.0-1.fc39.aarch64

Build from clean:

$ rm -fr build
$ CC=clang CXX=clang++ cmake -B build -DGGML_VULKAN=1
$ cmake --build build

Running inference

$ build/bin/llama-cli  -m examples/server/tests/stories15M_MOE-F16.gguf  -p "hello" -no-cnv -ngl 99 
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Virtio-GPU Venus (Apple M2) (venus) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 32768 | matrix cores: none
build: 4758 (5fa07c2f) with clang version 17.0.6 (Fedora 17.0.6-2.fc39) for aarch64-redhat-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device Vulkan0 (Virtio-GPU Venus (Apple M2)) - 24576 MiB free
llama_model_loader: loaded meta data with 26 key-value pairs and 63 tensors from examples/server/tests/stories15M_MOE-F16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                         general.size_label str              = 4x24M
llama_model_loader: - kv   3:                            general.license str              = mit
llama_model_loader: - kv   4:                          llama.block_count u32              = 6
llama_model_loader: - kv   5:                       llama.context_length u32              = 256
llama_model_loader: - kv   6:                     llama.embedding_length u32              = 288
llama_model_loader: - kv   7:                  llama.feed_forward_length u32              = 768
llama_model_loader: - kv   8:                 llama.attention.head_count u32              = 6
llama_model_loader: - kv   9:              llama.attention.head_count_kv u32              = 6
llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  11:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  12:                         llama.expert_count u32              = 4
llama_model_loader: - kv  13:                    llama.expert_used_count u32              = 2
llama_model_loader: - kv  14:                          general.file_type u32              = 1
llama_model_loader: - kv  15:                           llama.vocab_size u32              = 32000
llama_model_loader: - kv  16:                 llama.rope.dimension_count u32              = 48
llama_model_loader: - kv  17:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  18:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  19:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  20:                      tokenizer.ggml.scores arr[f32,32000]   = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  21:                  tokenizer.ggml.token_type arr[i32,32000]   = [3, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  22:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  23:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  24:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   19 tensors
llama_model_loader: - type  f16:   44 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = F16
print_info: file size   = 69.37 MiB (16.00 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 3
load: token to piece cache size = 0.1684 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 256
print_info: n_embd           = 288
print_info: n_layer          = 6
print_info: n_head           = 6
print_info: n_head_kv        = 6
print_info: n_rot            = 48
print_info: n_swa            = 0
print_info: n_embd_head_k    = 48
print_info: n_embd_head_v    = 48
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 288
print_info: n_embd_v_gqa     = 288
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: n_ff             = 768
print_info: n_expert         = 4
print_info: n_expert_used    = 2
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 256
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = ?B
print_info: model params     = 36.36 M
print_info: general.name     = n/a
print_info: vocab type       = SPM
print_info: n_vocab          = 32000
print_info: n_merges         = 0
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: LF token         = 13 '<0x0A>'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 48
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 6 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 7/7 layers to GPU
load_tensors:      Vulkan0 model buffer size =    51.79 MiB
load_tensors:   CPU_Mapped model buffer size =    17.58 MiB
.........................
llama_init_from_model: n_seq_max     = 1
llama_init_from_model: n_ctx         = 4096
llama_init_from_model: n_ctx_per_seq = 4096
llama_init_from_model: n_batch       = 2048
llama_init_from_model: n_ubatch      = 512
llama_init_from_model: flash_attn    = 0
llama_init_from_model: freq_base     = 1000000.0
llama_init_from_model: freq_scale    = 1
llama_init_from_model: n_ctx_pre_seq (4096) > n_ctx_train (256) -- possible training context overflow
llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 6, can_shift = 1
llama_kv_cache_init:    Vulkan0 KV buffer size =    27.00 MiB
llama_init_from_model: KV self size  =   27.00 MiB, K (f16):   13.50 MiB, V (f16):   13.50 MiB
llama_init_from_model: Vulkan_Host  output buffer size =     0.12 MiB
llama_init_from_model:    Vulkan0 compute buffer size =    63.06 MiB
llama_init_from_model: Vulkan_Host compute buffer size =     8.57 MiB
llama_init_from_model: graph nodes  = 288
llama_init_from_model: graph splits = 2
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 8
main: model was trained on only 256 context tokens (4096 specified)

system_info: n_threads = 8 (n_threads_batch = 8) / 8 | CPU : NEON = 1 | ARM_FMA = 1 | DOTPROD = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

sampler seed: 3879293893
sampler params:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
        top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
generate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 1

 hello even m g ever lived liveeshesh oppositeakakakakakakakakakakakakakakak corner corner even v careobfullyumpakakakakakakingakineakakakingakakak corner cornerunun exchangeinunesheshishmm d deepeshesharealedeseunumeeshousakakakakakak corner cornerlap even end littleinyakakak corner cornerinyeshesh rot edgeakeeralakakaking cornerfully far Unter never evenirir lost partakakakingakakakakakakakakak corner corneruninakeshakeraleshun matterseralcingakakakakakakak corner cornerinyakakakakakingakak cornerakinakunfullyin hisun hopeineshunesheshredake sprinunineshuneraleral mattersarpakakakakakakakakakingakakakakak cornerak corner cornerinyarpakakakakakakakingakeshakakakakakakakakakakakakakakakakakakhing handin unknownak care ju housearp showingakak corner cornerinyakakakakakakakakakakakakakakingakakakakakakakak cornerak corner cornerununoodineununineshuneshuneraleral encoueshakakakak corner cornerarpinyakakakakingak corner cornerarpredese introdueshesheshous teakakakakakakakakakakakakeshakakakakakakak corner cornerfularpakakeshakakakingak corner cornerinyinyakakakakakakakakakakakakakakakak cornerinun exakakakakakakakakakakakakingakWillakeinyakeshakeshousaledeshunununeraleraleralir

[slp]

Yes, that's expected. You need a patched version of MoltenVK. If you have installed krunkit from brew, you can try uninstalling and reinstalling again, which should get you molten-vk-krunkit (the patches version) instead of molten-vk.

If you have installed it with podman-machine, you'll need to wait for the next podman release.