@@ -277,6 +277,82 @@ cmake --build build
277
277
cmake --install build
278
278
```
279
279
280
+ ### Windows
281
+
282
+ #### Install Dependencies
283
+
284
+ 1 . (Optional, skip this deps if you don't need to use GPU) Download and install CUDA toolkit
285
+ - We use CUDA Toolkit 12 for the release assets
286
+ - Link: https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64&target_version=11&target_type=exe_local
287
+
288
+ 2 . Download and install Visual Studio 2022 Community Edition
289
+ - Link: https://visualstudio.microsoft.com/vs/community/
290
+ - Select the following components in the installer:
291
+ - msvc v143 - vs 2022 c++ x64/x86 build tools (latest)
292
+ - windows 11 sdk (10.0.22621.0)
293
+ - C++ ATL for v143 build tools (x86 & x64)
294
+
295
+ 3 . Download and install cmake
296
+ - We use cmake 3.29.3 for the release assets
297
+ - Link: https://github.com/Kitware/CMake/releases/download/v3.29.3/cmake-3.29.3-windows-x86_64.msi
298
+
299
+ 5 . Download and install git
300
+ - We use git 2.45.1
301
+ - Link: https://github.com/git-for-windows/git/releases/download/v2.45.1.windows.1/Git-2.45.1-64-bit.exe
302
+
303
+ 6 . Download and install ninja-build
304
+ - We use ninja-build 1.12.1
305
+ - Link: https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-win.zip
306
+ - Installation: just unzip it to a custom folder
307
+
308
+ #### Build
309
+
310
+ 1 . Open Developer PowerShell for VS 2022
311
+ - Start -> Visual Studio 2022 -> Visual Studio Tools -> Developer PowerShell for VS 2022
312
+
313
+ 2 . Inside the PowerShell, use git to download wasmedge repo
314
+
315
+ ``` console
316
+ cd $HOME
317
+ git clone https://github.com/WasmEdge/WasmEdge.git
318
+ cd WasmEdge
319
+ ```
320
+
321
+ 3 . Compile wasmedge with enabling the ` wasi_nn_ggml ` related options, please use the following commands. To build the plugin, you don't need to enable AOT/LLVM related features, so set them to OFF.
322
+
323
+ ##### CUDA Enable
324
+
325
+ ``` console
326
+ # CUDA ENABLE:
327
+ & "C:\Program files\CMake\bin\cmake.exe" -Bbuild -GNinja -DCMAKE_BUILD_TYPE=Release -DWASMEDGE_PLUGIN_WASI_NN_BACKEND=ggml -DWASMEDGE_PLUGIN_WASI_NN_GGML_LLAMA_CUBLAS=ON -DWASMEDGE_USE_LLVM=OFF .
328
+ & "<the ninja-build folder>\ninja.exe" -C build
329
+ ```
330
+
331
+ ##### CUDA Disable
332
+
333
+ ``` console
334
+ # CUDA DISABLE:
335
+ & "C:\Program files\CMake\bin\cmake.exe" -Bbuild -GNinja -DCMAKE_BUILD_TYPE=Release -DWASMEDGE_PLUGIN_WASI_NN_BACKEND=ggml -DWASMEDGE_USE_LLVM=OFF .
336
+ & "<the ninja-build folder>\ninja.exe" -C build
337
+ ```
338
+
339
+ #### Execute the WASI-NN plugin with the llama example
340
+
341
+ 1 . Set the environment variables
342
+
343
+ ``` console
344
+ $env:PATH += ";$pwd\build\lib\api"
345
+ $env:WASMEDGE_PLUGIN_PATH = "$pwd\build\plugins"
346
+ ```
347
+
348
+ 2 . Download the wasm and run
349
+
350
+ ``` console
351
+ wget https://github.com/second-state/WasmEdge-WASINN-examples/raw/master/wasmedge-ggml/llama/wasmedge-ggml-llama.wasm
352
+ wget https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF/blob/main/Meta-Llama-3-8B-Instruct.Q5_K_M.gguf
353
+ wasmedge --dir .:. --env llama3=true --env n_gpu_layers=100 --nn-preload default:GGML:AUTO:Meta-Llama-3-8B-Instruct.Q5_K_M.gguf wasmedge-ggml-llama.wasm default
354
+ ```
355
+
280
356
### Appendix
281
357
282
358
<!-- prettier-ignore -->
0 commit comments