"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": [
+ "# Prompt and response projects with MAL and Ground Truth\n",
+ "\n",
+ "This notebook is meant to showcase how to generate prompts and responses to fine-tune large language models (LLMs) using MAL and Ground truth"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": [
+ "## Annotation payload types\n",
+ "\n",
+ "Labelbox supports two formats for the annotations payload:\n",
+ "\n",
+ "- Python annotation types (recommended)\n",
+ " - Provides a seamless transition between third-party platforms, machine learning pipelines, and Labelbox.\n",
+ " - Allows you to build annotations locally with local file paths, numpy arrays, or URLs.\n",
+ " - Supports easy conversion to NDJSON format to quickly import annotations to Labelbox.\n",
+ " - Supports one-level nested classification (radio, checklist, or free-form text) under a tool or classification annotation.\n",
+ "- JSON\n",
+ " - Skips formatting annotation payload in the Labelbox Python annotation type.\n",
+ " - Supports any levels of nested classification (radio, checklist, or free-form text) under a tool or classification annotation."
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": [
+ "## Label Import Types\n",
+ "\n",
+ "Labelbox supports two types of label imports:\n",
+ "\n",
+ "- [Model-assisted labeling (MAL)](https://docs.labelbox.com/docs/model-assisted-labeling) allows you to import computer-generated predictions and simple annotations created outside of Labelbox as pre-labels on an asset.\n",
+ "- [Ground truth](hhttps://docs.labelbox.com/docs/import-ground-truth) allows you to bulk import ground truth annotations from an external or third-party labeling system into Labelbox _Annotate_. Using the label import API to import external data can consolidate and migrate all annotations into Labelbox as a single source of truth."
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": [
+ "## Set up "
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "%pip install -q \"labelbox[data]\"",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": "import labelbox as lb\nimport labelbox.types as lb_types\nimport uuid",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "### Replace with your API key\n",
+ "\n",
+ "Replace the value of `API_KEY` with a valid [API key]([ref:create-api-key](https://docs.labelbox.com/reference/create-api-key)) to connect to the Labelbox client."
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "API_KEY = None\nclient = lb.Client(api_key=API_KEY)",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "## Supported Annotations\n",
+ "\n",
+ "Prompt and response generated projects support the following annotations:\n",
+ "\n",
+ "- Prompt and response creation projects\n",
+ " - Prompt text\n",
+ " - Radio\n",
+ " - Checklist\n",
+ " - Response text\n",
+ "\n",
+ "- Prompt creation projects\n",
+ " - Prompt text\n",
+ "\n",
+ "- Response creation projects\n",
+ " - Radio\n",
+ " - Checklist"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": [
+ "### Prompt"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": [
+ "#### Prompt text"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "prompt_annotation = lb_types.PromptClassificationAnnotation(\n name=\"Follow the prompt and select answers\",\n value=lb_types.PromptText(answer=\"This is an example of a prompt\"),\n)\n\nprompt_annotation_ndjson = {\n \"name\": \"Follow the prompt and select answers\",\n \"answer\": \"This is an example of a prompt\",\n}",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "### Responses"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": [
+ "#### Radio (single-choice)"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "response_radio_annotation = lb_types.ClassificationAnnotation(\n name=\"response radio feature\",\n value=lb_types.Radio(answer=lb_types.ClassificationAnswer(\n name=\"first_radio_answer\")),\n)\n\nresponse_radio_annotation_ndjson = {\n \"name\": \"response radio feature\",\n \"answer\": {\n \"name\": \"first_radio_answer\"\n },\n}",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "#### Checklist (multi-choice)"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "response_checklist_annotation = lb_types.ClassificationAnnotation(\n name=\"response checklist feature\",\n value=lb_types.Checklist(answer=[\n lb_types.ClassificationAnswer(name=\"option_1\"),\n lb_types.ClassificationAnswer(name=\"option_2\"),\n ]),\n)\n\nresponse_checklist_annotation_ndjson = {\n \"name\": \"response checklist feature\",\n \"answer\": [{\n \"name\": \"option_1\"\n }, {\n \"name\": \"option_2\"\n }],\n}",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "#### Response text"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "response_text_annotation = lb_types.ClassificationAnnotation(\n name=\"Provide a reason for your choice\",\n value=lb_types.Text(answer=\"This is an example of a response text\"),\n)\n\nresponse_text_annotation_ndjson = {\n \"name\": \"Provide a reason for your choice\",\n \"answer\": \"This is an example of a response text\",\n}",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "#### Nested classifications"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "nested_response_radio_annotation = lb_types.ClassificationAnnotation(\n name=\"nested_response_radio_question\",\n value=lb_types.Radio(answer=lb_types.ClassificationAnswer(\n name=\"first_radio_answer\",\n classifications=[\n lb_types.ClassificationAnnotation(\n name=\"sub_radio_question\",\n value=lb_types.Radio(answer=lb_types.ClassificationAnswer(\n name=\"first_sub_radio_answer\")),\n )\n ],\n )),\n)\n\nnested_response_checklist_annotation = lb_types.ClassificationAnnotation(\n name=\"nested_response_checklist_question\",\n value=lb_types.Checklist(answer=[\n lb_types.ClassificationAnswer(\n name=\"first_checklist_answer\",\n classifications=[\n lb_types.ClassificationAnnotation(\n name=\"sub_checklist_question\",\n value=lb_types.Checklist(answer=[\n lb_types.ClassificationAnswer(\n name=\"first_sub_checklist_answer\")\n ]),\n )\n ],\n )\n ]),\n)\n\nnested_response_radio_annotation_ndjson = {\n \"name\":\n \"nested_radio_question\",\n \"answer\": [{\n \"name\":\n \"first_radio_answer\",\n \"classifications\": [{\n \"name\": \"sub_radio_question\",\n \"answer\": {\n \"name\": \"first_sub_radio_answer\"\n },\n }],\n }],\n}\n\nnested_response_checklist_annotation_ndjson = {\n \"name\":\n \"nested_checklist_question\",\n \"answer\": [{\n \"name\":\n \"first_checklist_answer\",\n \"classifications\": [{\n \"name\": \"sub_checklist_question\",\n \"answer\": {\n \"name\": \"first_sub_checklist_answer\"\n },\n }],\n }],\n}",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "## Step 1: Create a project and data rows using the Labelbox UI\n",
+ "\n",
+ "Each type of the prompt and response generation project requires different setup. See [prompt and response project](https://docs.labelbox.com/reference/prompt-and-response-projects) for more details on the differences.\n",
+ "\n",
+ "In this tutorial, we will show how to import annotations for a prompt and response creation (humans generate prompts and responses) project. The process is also similar for prompt creation (humans generate prompts) and response creation (humans generate responses to uploaded prompts) projects. See [import prompt and response annotations](https://docs.labelbox.com/reference/import-prompt-and-response-annotations) for a tutorial and more examples on other project types."
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": [
+ "### Prompt response and prompt creation\n",
+ "\n",
+ "A prompts and responses creation project automatically generates empty data rows upon creation. You will then need to obtain either the `global_keys` or `data_row_ids` attached to the generated data rows by exporting them from the created project."
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "prompt_response_project = client.create_model_evaluation_project(\n name=\"Demo prompt response project\",\n media_type=lb.MediaType.LLMPromptResponseCreation,\n dataset_name=\"Demo prompt response dataset\",\n data_row_count=1,\n)\n\nexport_task = prompt_response_project.export()\nexport_task.wait_till_done()\n\n# Check export for any errors\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nstream = export_task.get_buffered_stream()\n\n# Obtain global keys to be used later on\nglobal_keys = [dr.json[\"data_row\"][\"global_key\"] for dr in stream]",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "## Step 2: Set up ontology\n",
+ "\n",
+ "Your project ontology needs to support the classifications required by your annotations. To ensure accurate schema feature mapping, the value used as the `name` parameter needs to match the value of the `name` field in your annotation. \n",
+ "\n",
+ "For example, if you provide a name `annotation_name` for your created annotation, you need to name the bounding box tool as `anotations_name` when setting up your ontology. The same alignment must hold true for the other tools and classifications that you create in the ontology.\n",
+ "\n",
+ "This example shows how to create an ontology containing all supported by prompt and response projects [annotation types](#supported-annotations)."
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "ontology_builder = lb.OntologyBuilder(\n tools=[],\n classifications=[\n lb.PromptResponseClassification(\n class_type=lb.PromptResponseClassification.Type.PROMPT,\n name=\"prompt text\",\n character_min=1, # Minimum character count of prompt field (optional)\n character_max=\n 20, # Maximum character count of prompt field (optional)\n ),\n lb.PromptResponseClassification(\n class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,\n name=\"response checklist feature\",\n options=[\n lb.ResponseOption(value=\"option_1\", label=\"option_1\"),\n lb.ResponseOption(value=\"option_2\", label=\"option_2\"),\n ],\n ),\n lb.PromptResponseClassification(\n class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,\n name=\"response radio feature\",\n options=[\n lb.ResponseOption(value=\"first_radio_answer\"),\n lb.ResponseOption(value=\"second_radio_answer\"),\n ],\n ),\n lb.PromptResponseClassification(\n class_type=lb.PromptResponseClassification.Type.RESPONSE_TEXT,\n name=\"response text\",\n character_min=\n 1, # Minimum character count of response text field (optional)\n character_max=\n 20, # Maximum character count of response text field (optional)\n ),\n lb.PromptResponseClassification(\n class_type=lb.PromptResponseClassification.Type.RESPONSE_RADIO,\n name=\"nested_response_radio_question\",\n options=[\n lb.ResponseOption(\n \"first_radio_answer\",\n options=[\n lb.PromptResponseClassification(\n class_type=lb.PromptResponseClassification.\n RESPONSE_RADIO,\n name=\"sub_radio_question\",\n options=[\n lb.ResponseOption(\"first_sub_radio_answer\")\n ],\n )\n ],\n )\n ],\n ),\n lb.PromptResponseClassification(\n class_type=lb.PromptResponseClassification.Type.RESPONSE_CHECKLIST,\n name=\"nested_response_checklist_question\",\n options=[\n lb.ResponseOption(\n \"first_checklist_answer\",\n options=[\n lb.PromptResponseClassification(\n class_type=lb.PromptResponseClassification.\n RESPONSE_CHECKLIST,\n name=\"sub_checklist_question\",\n options=[\n lb.ResponseOption(\"first_sub_checklist_answer\")\n ],\n )\n ],\n )\n ],\n ),\n ],\n)\n\n# Create ontology\nontology = client.create_ontology(\n \"Prompt and response ontology\",\n ontology_builder.asdict(),\n media_type=lb.MediaType.LLMPromptResponseCreation,\n)",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "## Step 3: Create the annotations payload\n",
+ "\n",
+ "For prelabeled (model-assisted labeling) scenarios, pass your payload as the value of the `predictions` parameter. For ground truths, pass the payload to the `labels` parameter."
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "# Python annotation objects\nlabel = []\nannotations = [\n prompt_annotation,\n response_radio_annotation,\n response_checklist_annotation,\n response_text_annotation,\n nested_response_radio_annotation,\n nested_response_checklist_annotation,\n]\nlabel.append(\n lb_types.Label(data={\"global_key\": global_keys[0]},\n annotations=annotations))\n\n# NDJSON\nlabel_ndjson = []\nannotations = [\n prompt_annotation_ndjson,\n response_radio_annotation_ndjson,\n response_checklist_annotation_ndjson,\n response_text_annotation_ndjson,\n nested_response_radio_annotation_ndjson,\n nested_response_checklist_annotation_ndjson,\n]\nfor annotation in annotations:\n annotation.update({\n \"dataRow\": {\n \"globalKey\": global_keys[0]\n },\n })\n label_ndjson.append(annotation)",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "#### Option A: Upload as [prelabels (model assisted labeling)](doc:model-assisted-labeling)\n",
+ "\n",
+ "This option is helpful for speeding up the initial labeling process and reducing the manual labeling workload for high-volume datasets."
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "upload_job = lb.MALPredictionImport.create_from_objects(\n client=client,\n project_id=prompt_response_project.uid,\n name=f\"mal_job-{str(uuid.uuid4())}\",\n predictions=label,\n)\n\nupload_job.wait_until_done()\nprint(\"Errors:\", upload_job.errors)\nprint(\"Status of uploads: \", upload_job.statuses)",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "#### Option B: Upload to a labeling project as [ground truth](doc:import-ground-truth)\n",
+ "\n",
+ "This option is helpful for loading high-confidence labels from another platform or previous projects that just need review rather than manual labeling effort."
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "upload_job = lb.LabelImport.create_from_objects(\n client=client,\n project_id=prompt_response_project.uid,\n name=\"label_import_job\" + str(uuid.uuid4()),\n labels=label_ndjson,\n)\n\nupload_job.wait_until_done()\nprint(\"Errors:\", upload_job.errors)\nprint(\"Status of uploads: \", upload_job.statuses)",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "metadata": {},
+ "source": [
+ "## Clean up\n",
+ "\n",
+ "Uncomment and run the cell below to optionally delete Labelbox objects created"
+ ],
+ "cell_type": "markdown"
+ },
+ {
+ "metadata": {},
+ "source": "# project.delete()\n# client.delete_unused_ontology(ontology.uid)",
+ "cell_type": "code",
+ "outputs": [],
+ "execution_count": null
+ }
+ ]
+}
\ No newline at end of file
diff --git a/scripts/generate_readme.py b/scripts/generate_readme.py
index 7542cb6..db02293 100644
--- a/scripts/generate_readme.py
+++ b/scripts/generate_readme.py
@@ -45,21 +45,23 @@
"""
COLAB_TEMPLATE = "https://colab.research.google.com/github/Labelbox/labelbox-notebooks/blob/main/{filename}"
-GITHUB_TEMPLATE = "https://github.com/Labelbox/labelbox-notebooks/tree/main/{filename}"
+GITHUB_TEMPLATE = (
+ "https://github.com/Labelbox/labelbox-notebooks/tree/main/{filename}"
+)
+
def special_order(link_dict: Dict[str, list]) -> Dict:
- """This is used to add a special order to certain sections. It makes a copy of the link dict provided then loops through items inside the link dict to create a specified order. (Not random) anything not found in the global variable for the section is just tacked on to the end.
- """
+ """This is used to add a special order to certain sections. It makes a copy of the link dict provided then loops through items inside the link dict to create a specified order. (Not random) anything not found in the global variable for the section is just tacked on to the end."""
modified_link_dict = copy.deepcopy(link_dict)
for section, links in link_dict.items():
-
+
if section == "basics":
basic_order = BASICS_ORDER
for link_name in links:
if link_name not in BASICS_ORDER:
basic_order.append(link_name)
modified_link_dict[section] = basic_order
-
+
return modified_link_dict
![](\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\"){kind=logo}
",
+ "