Skip to content

Commit 497f4f7

Browse files
github-actions[bot]github-actions[bot]
committed
🎨 Cleaned
1 parent 879c993 commit 497f4f7

File tree

1 file changed

+55
-144
lines changed

1 file changed

+55
-144
lines changed

‎examples/project_configuration/model_chat_evaluation_project.ipynb

Lines changed: 55 additions & 144 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,18 @@
11
{
2+
"nbformat": 4,
3+
"nbformat_minor": 2,
4+
"metadata": {},
25
"cells": [
36
{
4-
"cell_type": "markdown",
57
"metadata": {},
68
"source": [
7-
"<td>\n",
8-
" <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>\n",
9+
"<td>",
10+
" <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>",
911
"</td>\n"
10-
]
12+
],
13+
"cell_type": "markdown"
1114
},
1215
{
13-
"cell_type": "markdown",
1416
"metadata": {},
1517
"source": [
1618
"<td>\n",
@@ -22,10 +24,10 @@
2224
"<a href=\"https://github.com/Labelbox/labelbox-python/tree/develop/examples/project_configuration/model_chat_evaluation_project.ipynb\" target=\"_blank\"><img\n",
2325
"src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n",
2426
"</td>"
25-
]
27+
],
28+
"cell_type": "markdown"
2629
},
2730
{
28-
"cell_type": "markdown",
2931
"metadata": {},
3032
"source": [
3133
"# Model Chat Evaluation Project Setup\n",
@@ -38,149 +40,95 @@
3840
"- `client.create_ontology`: Methods used to create Labelbox ontologies for MCE project this requires a `ontology_kind` parameter set to `lb.OntologyKind.ModelEvaluation`\n",
3941
"\n",
4042
"- `client.create_ontology_from_feature_schemas`: Similar to `client.create_ontology` but from a list of `feature schema ids` which is designed to allow you to use existing features instead of creating new features. This also requires a `ontology_kind` set to `lb.OntologyKind.ModelEvaluation`."
41-
]
43+
],
44+
"cell_type": "markdown"
4245
},
4346
{
44-
"cell_type": "markdown",
4547
"metadata": {},
4648
"source": [
4749
"## Set Up"
48-
]
50+
],
51+
"cell_type": "markdown"
4952
},
5053
{
51-
"cell_type": "code",
52-
"execution_count": null,
5354
"metadata": {},
55+
"source": "%pip install -q \"labelbox[data]\"",
56+
"cell_type": "code",
5457
"outputs": [],
55-
"source": [
56-
"%pip install -q \"labelbox[data]\""
57-
]
58+
"execution_count": null
5859
},
5960
{
60-
"cell_type": "code",
61-
"execution_count": null,
6261
"metadata": {},
62+
"source": "import labelbox as lb",
63+
"cell_type": "code",
6364
"outputs": [],
64-
"source": [
65-
"import labelbox as lb"
66-
]
65+
"execution_count": null
6766
},
6867
{
69-
"cell_type": "markdown",
7068
"metadata": {},
7169
"source": [
7270
"## API Key and Client\n",
7371
"Provide a valid API key below in order to properly connect to the Labelbox client. Please review [Create API key guide](https://docs.labelbox.com/reference/create-api-key) for more information."
74-
]
72+
],
73+
"cell_type": "markdown"
7574
},
7675
{
77-
"cell_type": "code",
78-
"execution_count": null,
7976
"metadata": {},
77+
"source": "API_KEY = None\nclient = lb.Client(api_key=API_KEY)",
78+
"cell_type": "code",
8079
"outputs": [],
81-
"source": [
82-
"API_KEY = None\n",
83-
"client = lb.Client(api_key=API_KEY)"
84-
]
80+
"execution_count": null
8581
},
8682
{
87-
"cell_type": "markdown",
8883
"metadata": {},
8984
"source": [
9085
"## Example: Create Model Chat Evaluation Project\n",
9186
"\n",
9287
"The steps to creating a Model Chat Evaluation Project through the Labelbox-Python SDK are similar to creating a regular project. However, they vary slightly, and we will showcase the different methods in this example workflow."
93-
]
88+
],
89+
"cell_type": "markdown"
9490
},
9591
{
96-
"cell_type": "markdown",
9792
"metadata": {},
9893
"source": [
9994
"### Create a MOE Ontology\n",
10095
"\n",
10196
"You can create ontologies for Model Evaluation projects the same way as creating ontologies for other projects with the only requirement of passing in a `ontology_kind` parameter which needs set to `lb.OntologyKind.ModelEvaluation`. You can create ontologies with two methods: `client.create_ontology` and `client.create_ontology_from_feature_schemas`."
102-
]
97+
],
98+
"cell_type": "markdown"
10399
},
104100
{
105-
"cell_type": "markdown",
106101
"metadata": {},
107102
"source": [
108103
"#### Option A: `client.create_ontology`\n",
109104
"\n",
110105
"Typically, you create ontologies and generate the associated features at the same time. Below is an example of creating an ontology for your model chat evaluation project using supported tools and classifications. For information on supported annotation types visit our [model chat evaluation](https://docs.labelbox.com/docs/model-chat-evaluation#supported-annotation-types) guide."
111-
]
106+
],
107+
"cell_type": "markdown"
112108
},
113109
{
114-
"cell_type": "code",
115-
"execution_count": null,
116110
"metadata": {},
111+
"source": "ontology_builder = lb.OntologyBuilder(\n tools=[\n lb.Tool(\n tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,\n name=\"single select feature\",\n ),\n lb.Tool(\n tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,\n name=\"multi select feature\",\n ),\n lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name=\"ranking feature\"),\n ],\n classifications=[\n lb.Classification(\n class_type=lb.Classification.Type.CHECKLIST,\n name=\"checklist feature\",\n options=[\n lb.Option(value=\"option 1\", label=\"option 1\"),\n lb.Option(value=\"option 2\", label=\"option 2\"),\n ],\n ),\n lb.Classification(\n class_type=lb.Classification.Type.RADIO,\n name=\"radio_question\",\n options=[\n lb.Option(value=\"first_radio_answer\"),\n lb.Option(value=\"second_radio_answer\"),\n ],\n ),\n ],\n)\n\n# Create ontology\nontology = client.create_ontology(\n \"MCE ontology\",\n ontology_builder.asdict(),\n media_type=lb.MediaType.Conversational,\n ontology_kind=lb.OntologyKind.ModelEvaluation,\n)",
112+
"cell_type": "code",
117113
"outputs": [],
118-
"source": [
119-
"ontology_builder = lb.OntologyBuilder(\n",
120-
" tools=[\n",
121-
" lb.Tool(\n",
122-
" tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,\n",
123-
" name=\"single select feature\",\n",
124-
" ),\n",
125-
" lb.Tool(\n",
126-
" tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,\n",
127-
" name=\"multi select feature\",\n",
128-
" ),\n",
129-
" lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name=\"ranking feature\"),\n",
130-
" ],\n",
131-
" classifications=[\n",
132-
" lb.Classification(\n",
133-
" class_type=lb.Classification.Type.CHECKLIST,\n",
134-
" name=\"checklist feature\",\n",
135-
" options=[\n",
136-
" lb.Option(value=\"option 1\", label=\"option 1\"),\n",
137-
" lb.Option(value=\"option 2\", label=\"option 2\"),\n",
138-
" ],\n",
139-
" ),\n",
140-
" lb.Classification(\n",
141-
" class_type=lb.Classification.Type.RADIO,\n",
142-
" name=\"radio_question\",\n",
143-
" options=[\n",
144-
" lb.Option(value=\"first_radio_answer\"),\n",
145-
" lb.Option(value=\"second_radio_answer\"),\n",
146-
" ],\n",
147-
" ),\n",
148-
" ],\n",
149-
")\n",
150-
"\n",
151-
"# Create ontology\n",
152-
"ontology = client.create_ontology(\n",
153-
" \"MCE ontology\",\n",
154-
" ontology_builder.asdict(),\n",
155-
" media_type=lb.MediaType.Conversational,\n",
156-
" ontology_kind=lb.OntologyKind.ModelEvaluation,\n",
157-
")"
158-
]
114+
"execution_count": null
159115
},
160116
{
161-
"cell_type": "markdown",
162117
"metadata": {},
163118
"source": [
164119
"### Option B: `client.create_ontology_from_feature_schemas`\n",
165120
"Ontologies can also be created with feature schema IDs. This makes your ontologies with existing features compared to generating new features. You can get these features by going to the _Schema_ tab inside Labelbox. (uncomment the below code block for this option)"
166-
]
121+
],
122+
"cell_type": "markdown"
167123
},
168124
{
169-
"cell_type": "code",
170-
"execution_count": null,
171125
"metadata": {},
126+
"source": "# ontology = client.create_ontology_from_feature_schemas(\n# \"MCE ontology\",\n# feature_schema_ids=[\"<list of feature schema ids\"],\n# media_type=lb.MediaType.Conversational,\n# ontology_kind=lb.OntologyKind.ModelEvaluation,\n# )",
127+
"cell_type": "code",
172128
"outputs": [],
173-
"source": [
174-
"# ontology = client.create_ontology_from_feature_schemas(\n",
175-
"# \"MCE ontology\",\n",
176-
"# feature_schema_ids=[\"<list of feature schema ids\"],\n",
177-
"# media_type=lb.MediaType.Conversational,\n",
178-
"# ontology_kind=lb.OntologyKind.ModelEvaluation,\n",
179-
"# )"
180-
]
129+
"execution_count": null
181130
},
182131
{
183-
"cell_type": "markdown",
184132
"metadata": {},
185133
"source": [
186134
"### Set Up Model Chat Evaluation Project\n",
@@ -204,83 +152,46 @@
204152
" - `dataset_id`: A dataset ID of an existing Labelbox dataset. Include this parameter if you are wanting to append to an existing MOE dataset.\n",
205153
"\n",
206154
" - `data_row_count`: The number of data row assets that will be generated and used with your project.\n"
207-
]
155+
],
156+
"cell_type": "markdown"
208157
},
209158
{
210-
"cell_type": "code",
211-
"execution_count": null,
212159
"metadata": {},
160+
"source": "project = client.create_model_evaluation_project(\n name=\"Demo MCE Project\",\n media_type=lb.MediaType.Conversational,\n dataset_name=\"Demo MCE dataset\",\n data_row_count=100,\n)\n\n# Setup project with ontology created above\nproject.setup_editor(ontology)",
161+
"cell_type": "code",
213162
"outputs": [],
214-
"source": [
215-
"project = client.create_model_evaluation_project(\n",
216-
" name=\"Demo MCE Project\",\n",
217-
" media_type=lb.MediaType.Conversational,\n",
218-
" dataset_name=\"Demo MCE dataset\",\n",
219-
" data_row_count=100,\n",
220-
")\n",
221-
"\n",
222-
"# Setup project with ontology created above\n",
223-
"project.setup_editor(ontology)"
224-
]
163+
"execution_count": null
225164
},
226165
{
227-
"cell_type": "markdown",
228166
"metadata": {},
229167
"source": [
230168
"## Exporting Model Chat Evaluation Project\n",
231169
"Exporting from a Model Chat Evaluation project works the same as exporting from other projects. In this example, unless you have created labels inside the Labelbox platform your exported will be shown as empty. Please review our [Model Chat Evaluation Export](https://docs.labelbox.com/reference/export-model-chat-evaluation-annotations) guide for a sample export."
232-
]
170+
],
171+
"cell_type": "markdown"
233172
},
234173
{
235-
"cell_type": "code",
236-
"execution_count": null,
237174
"metadata": {},
175+
"source": "# Start export from project\nexport_task = project.export()\nexport_task.wait_till_done()\n\n# Conditional if task has errors\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n # Start export stream\n stream = export_task.get_buffered_stream()\n\n # Iterate through data rows\n for data_row in stream:\n print(data_row.json)",
176+
"cell_type": "code",
238177
"outputs": [],
239-
"source": [
240-
"# Start export from project\n",
241-
"export_task = project.export()\n",
242-
"export_task.wait_till_done()\n",
243-
"\n",
244-
"# Conditional if task has errors\n",
245-
"if export_task.has_errors():\n",
246-
" export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n",
247-
" stream_handler=lambda error: print(error))\n",
248-
"\n",
249-
"if export_task.has_result():\n",
250-
" # Start export stream\n",
251-
" stream = export_task.get_buffered_stream()\n",
252-
"\n",
253-
" # Iterate through data rows\n",
254-
" for data_row in stream:\n",
255-
" print(data_row.json)"
256-
]
178+
"execution_count": null
257179
},
258180
{
259-
"cell_type": "markdown",
260181
"metadata": {},
261182
"source": [
262183
"## Clean Up\n",
263184
"\n",
264185
"This section serves as an optional clean-up step to delete the Labelbox assets created within this guide. You will need to uncomment the delete methods shown."
265-
]
186+
],
187+
"cell_type": "markdown"
266188
},
267189
{
268-
"cell_type": "code",
269-
"execution_count": null,
270190
"metadata": {},
191+
"source": "# project.delete()\n# client.delete_unused_ontology(ontology.uid)\n# dataset.delete()",
192+
"cell_type": "code",
271193
"outputs": [],
272-
"source": [
273-
"# project.delete()\n",
274-
"# client.delete_unused_ontology(ontology.uid)\n",
275-
"# dataset.delete()"
276-
]
194+
"execution_count": null
277195
}
278-
],
279-
"metadata": {
280-
"language_info": {
281-
"name": "python"
282-
}
283-
},
284-
"nbformat": 4,
285-
"nbformat_minor": 2
286-
}
196+
]
197+
}

0 commit comments

Comments
 (0)