You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Besides the [regular WasmEdge and Rust requirements](../../rust/setup.md), please make sure that you have the [Wasi-NN plugin with ggml installed](../../../start/install.md#wasi-nn-plug-in-with-ggml-backend).
14
14
15
15
## Quick start
16
+
16
17
Because the example already includes a compiled WASM file from the Rust code, we could use WasmEdge CLI to execute the example directly. First, git clone the `llama-utils` repo.
Next, let's get the model. In this example, we are going to use the llama2 7b chat model in GGUF format. You can also use other kinds of llama2 models, check out [here](https://github.com/second-state/llama-utils/blob/main/chat/README.md#get-model).
After executing the command, you may need to wait a moment for the input prompt to appear. You can enter your question once you see the `[USER]:` prompt:
37
36
38
37
```bash
39
38
[USER]:
40
39
I have two apples, each costing 5 dollars. What is the total cost of these apple
41
-
*** [prompt begin] ***
42
-
<s>[INST] <<SYS>>
43
-
You are a helpful, respectful and honest assistant. Always answer as short as possible, while being safe. <</SYS>>
44
-
45
-
I have two apples, each costing 5 dollars. What is the total cost of these apple [/INST]
46
-
*** [prompt end] ***
47
40
[ASSISTANT]:
48
41
The total cost of the two apples is 10 dollars.
49
42
[USER]:
50
43
How about four apples?
51
-
*** [prompt begin] ***
52
-
<s>[INST] <<SYS>>
53
-
You are a helpful, respectful and honest assistant. Always answer as short as possible, while being safe. <</SYS>>
54
-
55
-
I have two apples, each costing 5 dollars. What is the total cost of these apple [/INST] The total cost of the two apples is 10 dollars. </s><s>[INST] How about four apples? [/INST]
After executing the command, you may need to wait a moment for the input prompt to appear. You can enter your question once you see the `[USER]:` prompt:
92
78
93
79
```bash
94
80
[USER]:
95
81
Who is Robert Oppenheimer?
96
-
*** [prompt begin] ***
97
-
<s>[INST] <<SYS>>
98
-
You are a helpful, respectful and honest assistant. Always answer as short as possible, while being safe. <</SYS>>
99
-
100
-
Who is Robert Oppenheimer? [/INST]
101
-
*** [prompt end] ***
102
82
[ASSISTANT]:
103
83
Robert Oppenheimer was an American theoretical physicist and director of the Manhattan Project, which developed the atomic bomb during World War II. He is widely regarded as one of the most important physicists of the 20th century and is known forhis contributions to the development of quantum mechanics and the theory of the atomic nucleus. Oppenheimer was also a prominent figurein the post-war nuclear weapons debate and was a strong advocate for international cooperation on nuclear weapons control.
104
84
```
105
85
106
-
## Optional: run the model with different CLI
86
+
## Options
107
87
108
-
We also have CLI options for more information.
88
+
You can configure the chat inference application through CLI options.
109
89
110
90
```bash
111
91
-m, --model-alias <ALIAS>
@@ -128,19 +108,33 @@ We also have CLI options for more information.
128
108
Print prompt strings to stdout
129
109
--log-stat
130
110
Print statistics to stdout
131
-
--log-enable
111
+
--log-all
132
112
Print all log information to stdout
133
113
--stream-stdout
134
114
Print the output to stdout in the streaming way
135
115
-h, --help
136
116
Print help
137
117
```
138
118
139
-
For example, the following command tells WasmEdge to print out logs and statistics of the model at runtime.
119
+
The `--prompt-template` option is perhaps the most interesting. It allows the application to support different open source LLM models beyond llama2.
@@ -158,27 +152,35 @@ llama_print_timings: total time = 25104.57 ms
158
152
Ah, a fellow Peanuts enthusiast! Snoopy is Charlie Brown's lovable and imaginative beagle, known for his wild and wacky adventures in the comic strip and television specials. He's a loyal companion to Charlie Brown and the rest of the Peanuts gang, and his antics often provide comic relief in the series. Is there anything else you'd like to know about Snoopy? 🐶
159
153
```
160
154
161
-
## Improve performance
155
+
## Improving performance
162
156
163
157
You can make the inference program run faster by AOT compiling the wasm file first.
The [main.rs](https://github.com/second-state/llama-utils/blob/main/chat/src/main.rs
175
-
) is the full Rust code to create an interactive chatbot using a LLM. The Rust program manages the user input, tracks the conversation history, transforms the text into the llama2 and other model’s chat templates, and runs the inference operations using the WASI NN standard API.
166
+
The [main.rs](https://github.com/second-state/llama-utils/blob/main/chat/src/main.rs) is the full Rust code to create an interactive chatbot using a LLM. The Rust program manages the user input, tracks the conversation history, transforms the text into the llama2 and other model’s chat templates, and runs the inference operations using the WASI NN standard API. The code logic for the chat interaction is somewhat complex. In this section, we will use the [simple example](https://github.com/second-state/llama-utils/tree/main/simple) to explain how to set up and perform one inference round trip. Here is how you use the simple example.
--prompt 'Robert Oppenheimer most important achievement is ' --ctx-size 4096
175
+
176
+
output: in 1942, when he led the team that developed the first atomic bomb, which was dropped on Hiroshima, Japan in 1945.
177
+
```
176
178
177
179
First, let's parse command line arguments to customize the chatbot's behavior using `Command` struct. It extracts the following parameters: `prompt` (a prompt that guides the conversation), `model_alias` (a list for the loaded model), and `ctx_size` (the size of the chat context).
178
180
179
181
```rust
180
182
fn main() -> Result<(), String> {
181
-
let matches = Command::new("Llama API Server")
183
+
let matches = Command::new("Simple LLM inference")
The code explanation above is simple [one time chat with llama 2 model](https://github.com/second-state/llama-utils/tree/main/simple). But we have more!
281
+
## Resources
282
+
283
+
* If you're looking for multi-turn conversations with llama 2 models, please check out the above mentioned chat example source code [here](https://github.com/second-state/llama-utils/tree/main/chat).
284
+
* If you want to construct OpenAI-compatible APIs specifically for your llama2 model, or the Llama2 model itself, please check out the source code [for the API server](https://github.com/second-state/llama-utils/tree/main/api-server).
285
+
* To learn more, please check out [this article](https://medium.com/stackademic/fast-and-portable-llama2-inference-on-the-heterogeneous-edge-a62508e82359).
280
286
281
-
* If you're looking for continuous conversations with llama 2 models, please check out the source code [here](https://github.com/second-state/llama-utils/tree/main/chat).
282
-
* If you want to construct OpenAI-compatible APIs specifically for your llama2 model, or the Llama2 model itself, please check out the surce code [here](https://github.com/second-state/llama-utils/tree/main/api-server).
283
-
* For the reason why we need to run LLama2 model with WasmEdge, please check out [this article](https://medium.com/stackademic/fast-and-portable-llama2-inference-on-the-heterogeneous-edge-a62508e82359).
0 commit comments