Using Metal with an Intel chip on macOS

[authcompanion]

Hello 👋 ,
Having some issues getting going for the first time and could use some assistance, please see below:

Reproduction:

• installed node-llama-cpp with: npm install --save node-llama-cpp
• copied the example script and named as inference.js + updated the model path with my gguf model.
• run node inference.js

Error:
llama_new_context_with_model: ggml_metal_init() failed

npx node-llama-cpp chat -m models/wizardlm-1.0-uncensored-llama2-13b.Q5_K_M.gguf
fails with the same error too

Model Name: MacBook Pro
ProductName: macOS
ProductVersion: 13.3.1
node: v18.16.0
node-llama-cpp: 2.4.0

inference.js:

import {fileURLToPath} from "url";
import path from "path";
import {LlamaModel, LlamaContext, LlamaChatSession} from "node-llama-cpp";

const __dirname = path.dirname(fileURLToPath(import.meta.url));

const model = new LlamaModel({
    modelPath: path.join(__dirname, "models", "wizardlm-1.0-uncensored-llama2-13b.Q5_K_M.gguf")
});
const context = new LlamaContext({model});
const session = new LlamaChatSession({context});


const q1 = "Hi there, how are you?";
console.log("User: " + q1);

const a1 = await session.prompt(q1);
console.log("AI: " + a1);


const q2 = "Summerize what you said";
console.log("User: " + q2);

const a2 = await session.prompt(q2);
console.log("AI: " + a2);``` 

[giladgd]

@authcompanion Do you have an Apple silicone chip or an Intel chip?

[authcompanion]

Intel!

[giladgd]

I have a Mac with Apple silicone and Metal works well on it, so I suspect it may be possible that llama.cpp doesn't properly support Metal on Macs with an Intel chip.

I found this issue on the llama.cpp repo: ggml-org/llama.cpp#3129
So I think the best solution for you may be to use llama.cpp with Metal support disabled.

You can do this by building node-llama-cpp without Metal support.
To do this, run this command inside of your project directory:

npx --no node-llama-cpp download --no-metal

Then use node-llama-cpp normally in your project.

[authcompanion]

Thanks @giladgd works great now - appreciate the push in the right direction.