Skip to content

Commit f974980

Browse files
Amnah199Amnah199
authored
Merge pull request #244 from deepset-ai/add-llama-stack-cookbook
Add a simple notebook for llama stack
2 parents e652881 + 85ba5b2 commit f974980

File tree

2 files changed

+230
-0
lines changed

2 files changed

+230
-0
lines changed

‎index.toml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,11 @@ title = "Build with Gemma and Haystack"
7171
notebook = "gemma_chat_rag.ipynb"
7272
topics = ["RAG"]
7373

74+
[[cookbook]]
75+
title = "Build with Llama Stack and Haystack Agent"
76+
notebook = "llama_stack_with_agent.ipynb"
77+
topics = ["Function Calling", "Agents"]
78+
7479
[[cookbook]]
7580
title = "Hacker News Summaries with Custom Components"
7681
notebook = "hackernews-custom-component-rag.ipynb"

‎notebooks/llama_stack_with_agent.ipynb

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# 🛠️🦙 Build with Llama Stack and Haystack Agent\n",
8+
"\n",
9+
"\n",
10+
"This notebook demonstrates how to use the `LlamaStackChatGenerator` component with Haystack `Agent` to enable function calling capabilities. We'll create a simple weather tool that the `Agent` can call to provide dynamic, up-to-date information.\n",
11+
"\n",
12+
"We start with installing integration package."
13+
]
14+
},
15+
{
16+
"cell_type": "code",
17+
"execution_count": null,
18+
"metadata": {},
19+
"outputs": [],
20+
"source": [
21+
"%%bash\n",
22+
"\n",
23+
"pip install llama-stack-haystack"
24+
]
25+
},
26+
{
27+
"cell_type": "markdown",
28+
"metadata": {},
29+
"source": [
30+
"## Setup\n",
31+
"\n",
32+
"Before running this example, you need to:\n",
33+
"\n",
34+
"1. Set up Llama Stack Server through an inference provider\n",
35+
"2. Have a model available (e.g., `llama3.2:3b`)\n",
36+
"\n",
37+
"For a quick start on how to setup server with Ollama, see the [Llama Stack documentation](https://llama-stack.readthedocs.io/en/latest/getting_started/index.html).\n",
38+
"\n",
39+
"Once you have the server running, it will typically be available at `http://localhost:8321/v1/openai/v1`.\n"
40+
]
41+
},
42+
{
43+
"cell_type": "markdown",
44+
"metadata": {},
45+
"source": [
46+
"## Defining a Tool\n",
47+
"\n",
48+
"Tools in Haystack allow models to call functions to get real-time information or perform actions. Let's create a simple weather tool that the model can use to provide weather information.\n"
49+
]
50+
},
51+
{
52+
"cell_type": "code",
53+
"execution_count": 1,
54+
"metadata": {},
55+
"outputs": [],
56+
"source": [
57+
"from haystack.dataclasses import ChatMessage\n",
58+
"from haystack.tools import Tool\n",
59+
"\n",
60+
"# Define a tool that models can call\n",
61+
"def weather(city: str):\n",
62+
" \"\"\"Return mock weather info for the given city.\"\"\"\n",
63+
" return f\"The weather in {city} is sunny and 32°C\"\n",
64+
"\n",
65+
"# Define the tool parameters schema\n",
66+
"tool_parameters = {\n",
67+
" \"type\": \"object\", \n",
68+
" \"properties\": {\n",
69+
" \"city\": {\"type\": \"string\"}\n",
70+
" }, \n",
71+
" \"required\": [\"city\"]\n",
72+
"}\n",
73+
"\n",
74+
"# Create the weather tool\n",
75+
"weather_tool = Tool(\n",
76+
" name=\"weather\",\n",
77+
" description=\"Useful for getting the weather in a specific city\",\n",
78+
" parameters=tool_parameters,\n",
79+
" function=weather,\n",
80+
")\n"
81+
]
82+
},
83+
{
84+
"cell_type": "markdown",
85+
"metadata": {},
86+
"source": [
87+
"## Setting Up Agent\n",
88+
"\n",
89+
"Now let's create a `LlamaStackChatGenerator` and pass it to the `Agent`.\n"
90+
]
91+
},
92+
{
93+
"cell_type": "code",
94+
"execution_count": 4,
95+
"metadata": {},
96+
"outputs": [],
97+
"source": [
98+
"from haystack.components.agents import Agent\n",
99+
"from haystack_integrations.components.generators.llama_stack import LlamaStackChatGenerator\n",
100+
"from haystack.components.generators.utils import print_streaming_chunk\n",
101+
"\n",
102+
"# Create the LlamaStackChatGenerator\n",
103+
"chat_generator = LlamaStackChatGenerator(\n",
104+
" model=\"ollama/llama3.2:3b\", # model name varies depending on the inference provider used for the Llama Stack Server\n",
105+
" api_base_url=\"http://localhost:8321/v1/openai/v1\",\n",
106+
")\n",
107+
"# Agent Setup\n",
108+
"agent = Agent(\n",
109+
" chat_generator=chat_generator,\n",
110+
" tools=[weather_tool],\n",
111+
")\n",
112+
"\n",
113+
"# Run the Agent\n",
114+
"agent.warm_up()\n"
115+
]
116+
},
117+
{
118+
"cell_type": "markdown",
119+
"metadata": {},
120+
"source": [
121+
"## Using Tools with the Agent\n",
122+
"\n",
123+
"Now, when we ask questions, the `Agent` will utilize both the provided `tool` and the `LlamaStackChatGenerator` to generate answers. We enable the streaming in Agent, so that you can observe the tool calls and the tool results in real time.\n"
124+
]
125+
},
126+
{
127+
"cell_type": "code",
128+
"execution_count": 7,
129+
"metadata": {},
130+
"outputs": [
131+
{
132+
"name": "stdout",
133+
"output_type": "stream",
134+
"text": [
135+
"[TOOL CALL]\n",
136+
"Tool: weather \n",
137+
"Arguments: {\"city\":\"Tokyo\"}\n",
138+
"\n",
139+
"[TOOL RESULT]\n",
140+
"The weather in Tokyo is sunny and 32°C\n",
141+
"\n",
142+
"In[ASSISTANT]\n",
143+
" Tokyo, the current weather conditions are mostly sunny with a temperature of 32°C. Would you like to know more about Tokyo's climate or weather forecast for a specific date?\n",
144+
"\n"
145+
]
146+
}
147+
],
148+
"source": [
149+
"# Create a message asking about the weather\n",
150+
"messages = [ChatMessage.from_user(\"What's the weather in Tokyo?\")]\n",
151+
"\n",
152+
"# Generate a response from the model with access to tools\n",
153+
"response = agent.run(messages=messages, tools=[weather_tool], streaming_callback=print_streaming_chunk,\n",
154+
")\n",
155+
"\n"
156+
]
157+
},
158+
{
159+
"cell_type": "markdown",
160+
"metadata": {},
161+
"source": [
162+
"## Simple Chat with ChatGenerator\n",
163+
"For a simpler use case, you can also create a lightweight mechanism to chat directly with `LlamaStackChatGenerator`."
164+
]
165+
},
166+
{
167+
"cell_type": "code",
168+
"execution_count": 15,
169+
"metadata": {},
170+
"outputs": [
171+
{
172+
"name": "stdout",
173+
"output_type": "stream",
174+
"text": [
175+
"🤖 The main character in The Witcher series, also known as the eponymous figure, is Geralt of Rivia, a monster hunter with supernatural abilities and mutations that allow him to control the elements. He was created by Polish author_and_polish_video_game_development_company_(CD Projekt).\n",
176+
"🤖 One of the most fascinating aspects of dolphin behavior is their ability to produce complex, context-dependent vocalizations that are unique to each individual, similar to human language. They also exhibit advanced social behaviors, such as cooperation, empathy, and self-awareness.\n"
177+
]
178+
}
179+
],
180+
"source": [
181+
"messages = []\n",
182+
"\n",
183+
"while True:\n",
184+
" msg = input(\"Enter your message or Q to exit\\n🧑 \")\n",
185+
" if msg==\"Q\":\n",
186+
" break\n",
187+
" messages.append(ChatMessage.from_user(msg))\n",
188+
" response = chat_generator.run(messages=messages)\n",
189+
" assistant_resp = response['replies'][0]\n",
190+
" print(\"🤖 \"+assistant_resp.text)\n",
191+
" messages.append(assistant_resp)"
192+
]
193+
},
194+
{
195+
"cell_type": "markdown",
196+
"metadata": {},
197+
"source": [
198+
"If you want to switch your model provider, you can reuse the same `LlamaStackChatGenerator` code with different providers. Simply run the desired inference provider on the Llama Stack Server and update the model name during the initialization of `LlamaStackChatGenerator`.\n",
199+
"\n",
200+
"For more details on available inference providers, see (Llama Stack docs)[https://llama-stack.readthedocs.io/en/latest/providers/inference/index.html]."
201+
]
202+
}
203+
],
204+
"metadata": {
205+
"kernelspec": {
206+
"display_name": ".venv",
207+
"language": "python",
208+
"name": "python3"
209+
},
210+
"language_info": {
211+
"codemirror_mode": {
212+
"name": "ipython",
213+
"version": 3
214+
},
215+
"file_extension": ".py",
216+
"mimetype": "text/x-python",
217+
"name": "python",
218+
"nbconvert_exporter": "python",
219+
"pygments_lexer": "ipython3",
220+
"version": "3.13.5"
221+
}
222+
},
223+
"nbformat": 4,
224+
"nbformat_minor": 2
225+
}

0 commit comments

Comments
 (0)