Skip to content

Commit 10a8c12

Browse files
authored
Add speculative decoding example (#55)
Also fix the structured response example docstring, and mark all example scripts as executable.
1 parent 788d076 commit 10a8c12

File tree

3 files changed

+20
-1
lines changed

3 files changed

+20
-1
lines changed

examples/speculative-decoding.py

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/usr/bin/env python
2+
"""Example script demonstrating speculative decoding."""
3+
4+
import lmstudio as lms
5+
6+
main_model_key = "qwen2.5-7b-instruct-1m"
7+
draft_model_key = "qwen2.5-0.5b-instruct"
8+
9+
model = lms.llm(main_model_key)
10+
result = model.respond(
11+
"What are the prime numbers between 0 and 100?",
12+
config={
13+
"draftModel": draft_model_key,
14+
}
15+
)
16+
17+
print(result)
18+
stats = result.stats
19+
print(f"Accepted {stats.accepted_draft_tokens_count}/{stats.predicted_tokens_count} tokens")

examples/structured-response.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env python
2-
"""Example script demonstrating an interactive LLM chatbot."""
2+
"""Example script demonstrating structured responses."""
33

44
import json
55

examples/tool-use-multiple.py

100644100755
File mode changed.

0 commit comments

Comments
 (0)