Skip to content

Commit 4312ae6

Browse files
hydaijuntao
authored andcommitted
Add windows build instructions for the ggml plugin
Signed-off-by: hydai <z54981220@gmail.com>
1 parent f373a49 commit 4312ae6

File tree

1 file changed

+76
-0
lines changed

1 file changed

+76
-0
lines changed

docs/contribute/source/plugin/wasi_nn.md

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -277,6 +277,82 @@ cmake --build build
277277
cmake --install build
278278
```
279279

280+
### Windows
281+
282+
#### Install Dependencies
283+
284+
1. (Optional, skip this deps if you don't need to use GPU) Download and install CUDA toolkit
285+
- We use CUDA Toolkit 12 for the release assets
286+
- Link: https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64&target_version=11&target_type=exe_local
287+
288+
2. Download and install Visual Studio 2022 Community Edition
289+
- Link: https://visualstudio.microsoft.com/vs/community/
290+
- Select the following components in the installer:
291+
- msvc v143 - vs 2022 c++ x64/x86 build tools (latest)
292+
- windows 11 sdk (10.0.22621.0)
293+
- C++ ATL for v143 build tools (x86 & x64)
294+
295+
3. Download and install cmake
296+
- We use cmake 3.29.3 for the release assets
297+
- Link: https://github.com/Kitware/CMake/releases/download/v3.29.3/cmake-3.29.3-windows-x86_64.msi
298+
299+
5. Download and install git
300+
- We use git 2.45.1
301+
- Link: https://github.com/git-for-windows/git/releases/download/v2.45.1.windows.1/Git-2.45.1-64-bit.exe
302+
303+
6. Download and install ninja-build
304+
- We use ninja-build 1.12.1
305+
- Link: https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-win.zip
306+
- Installation: just unzip it to a custom folder
307+
308+
#### Build
309+
310+
1. Open Developer PowerShell for VS 2022
311+
- Start -> Visual Studio 2022 -> Visual Studio Tools -> Developer PowerShell for VS 2022
312+
313+
2. Inside the PowerShell, use git to download wasmedge repo
314+
315+
```console
316+
cd $HOME
317+
git clone https://github.com/WasmEdge/WasmEdge.git
318+
cd WasmEdge
319+
```
320+
321+
3. Compile wasmedge with enabling the `wasi_nn_ggml` related options, please use the following commands. To build the plugin, you don't need to enable AOT/LLVM related features, so set them to OFF.
322+
323+
##### CUDA Enable
324+
325+
```console
326+
# CUDA ENABLE:
327+
& "C:\Program files\CMake\bin\cmake.exe" -Bbuild -GNinja -DCMAKE_BUILD_TYPE=Release -DWASMEDGE_PLUGIN_WASI_NN_BACKEND=ggml -DWASMEDGE_PLUGIN_WASI_NN_GGML_LLAMA_CUBLAS=ON -DWASMEDGE_USE_LLVM=OFF .
328+
& "<the ninja-build folder>\ninja.exe" -C build
329+
```
330+
331+
##### CUDA Disable
332+
333+
```console
334+
# CUDA DISABLE:
335+
& "C:\Program files\CMake\bin\cmake.exe" -Bbuild -GNinja -DCMAKE_BUILD_TYPE=Release -DWASMEDGE_PLUGIN_WASI_NN_BACKEND=ggml -DWASMEDGE_USE_LLVM=OFF .
336+
& "<the ninja-build folder>\ninja.exe" -C build
337+
```
338+
339+
#### Execute the WASI-NN plugin with the llama example
340+
341+
1. Set the environment variables
342+
343+
```console
344+
$env:PATH += ";$pwd\build\lib\api"
345+
$env:WASMEDGE_PLUGIN_PATH = "$pwd\build\plugins"
346+
```
347+
348+
2. Download the wasm and run
349+
350+
```console
351+
wget https://github.com/second-state/WasmEdge-WASINN-examples/raw/master/wasmedge-ggml/llama/wasmedge-ggml-llama.wasm
352+
wget https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF/blob/main/Meta-Llama-3-8B-Instruct.Q5_K_M.gguf
353+
wasmedge --dir .:. --env llama3=true --env n_gpu_layers=100 --nn-preload default:GGML:AUTO:Meta-Llama-3-8B-Instruct.Q5_K_M.gguf wasmedge-ggml-llama.wasm default
354+
```
355+
280356
### Appendix
281357

282358
<!-- prettier-ignore -->

0 commit comments

Comments
 (0)