[SN-150] Model Evaluation SDK Method Notebook #1645
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,197 @@ | ||
| { | ||
|
There was a problem hiding this comment.
"Optional description of your project." -> For consistency "The optional description of your project"
Suggestion: "A dataset ID of an existing Labelbox dataset." -> "An optional dataset ID of an existing Labelbox dataset."
Reply via ReviewNB |
||
| "nbformat": 4, | ||
| "nbformat_minor": 2, | ||
| "metadata": {}, | ||
| "cells": [ | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "<td>", | ||
| " <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>", | ||
| "</td>\n" | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "<td>\n", | ||
| "<a href=\"https://colab.research.google.com/github/Labelbox/labelbox-python/blob/develop/examples/project_configuration/model_chat_evaluation_project.ipynb\" target=\"_blank\"><img\n", | ||
| "src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n", | ||
| "</td>\n", | ||
| "\n", | ||
| "<td>\n", | ||
| "<a href=\"https://github.com/Labelbox/labelbox-python/tree/develop/examples/project_configuration/model_chat_evaluation_project.ipynb\" target=\"_blank\"><img\n", | ||
| "src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n", | ||
| "</td>" | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Model Chat Evaluation Project Setup\n", | ||
| "\n", | ||
| "This notebook will provide an example workflow of setting up a Model Chat Evaluation (MCE) Project with the Labelbox-Python SDK.\n", | ||
| "Model Chat Evaluation Projects are set up differently than other projects with its own unique method and modifications to existing methods:\n", | ||
| "\n", | ||
| "- `client.create_model_evaluation_project`: The main method used to create a model chat evaluation project\n", | ||
| "\n", | ||
| "- `client.create_ontology`: Methods used to create Labelbox ontologies for MCE project this requires an `ontology_kind` parameter set to `lb.OntologyKind.ModelEvaluation`\n", | ||
| "\n", | ||
| "- `client.create_ontology_from_feature_schemas`: Similar to `client.create_ontology` but from a list of `feature schema ids` designed to allow you to use existing features instead of creating new features. This also requires an `ontology_kind` set to `lb.OntologyKind.ModelEvaluation`." | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Set Up" | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": "%pip install -q \"labelbox[data]\"", | ||
| "cell_type": "code", | ||
| "outputs": [], | ||
| "execution_count": null | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": "import labelbox as lb", | ||
| "cell_type": "code", | ||
| "outputs": [], | ||
| "execution_count": null | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## API Key and Client\n", | ||
| "Provide a valid API key below in order to properly connect to the Labelbox client. Please review [Create API key guide](https://docs.labelbox.com/reference/create-api-key) for more information." | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": "API_KEY = None\nclient = lb.Client(api_key=API_KEY)", | ||
| "cell_type": "code", | ||
| "outputs": [], | ||
| "execution_count": null | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Example: Create Model Chat Evaluation Project\n", | ||
| "\n", | ||
| "The steps to creating a Model Chat Evaluation Project through the Labelbox-Python SDK are similar to creating a regular project. However, they vary slightly, and we will showcase the different methods in this example workflow." | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Create a Model Chat Evaluation Ontology\n", | ||
| "\n", | ||
| "You can create ontologies for Model Evaluation projects the same way as creating ontologies for other projects with the only requirement of passing in a `ontology_kind` parameter which needs set to `lb.OntologyKind.ModelEvaluation`. You can create ontologies with two methods: `client.create_ontology` and `client.create_ontology_from_feature_schemas`." | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "#### Option A: `client.create_ontology`\n", | ||
| "\n", | ||
| "Typically, you create ontologies and generate the associated features at the same time. Below is an example of creating an ontology for your model chat evaluation project using supported tools and classifications. For information on supported annotation types visit our [model chat evaluation](https://docs.labelbox.com/docs/model-chat-evaluation#supported-annotation-types) guide." | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": "ontology_builder = lb.OntologyBuilder(\n tools=[\n lb.Tool(\n tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,\n name=\"single select feature\",\n ),\n lb.Tool(\n tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,\n name=\"multi select feature\",\n ),\n lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name=\"ranking feature\"),\n ],\n classifications=[\n lb.Classification(\n class_type=lb.Classification.Type.CHECKLIST,\n name=\"checklist feature\",\n options=[\n lb.Option(value=\"option 1\", label=\"option 1\"),\n lb.Option(value=\"option 2\", label=\"option 2\"),\n ],\n ),\n lb.Classification(\n class_type=lb.Classification.Type.RADIO,\n name=\"radio_question\",\n options=[\n lb.Option(value=\"first_radio_answer\"),\n lb.Option(value=\"second_radio_answer\"),\n ],\n ),\n ],\n)\n\n# Create ontology\nontology = client.create_ontology(\n \"MCE ontology\",\n ontology_builder.asdict(),\n media_type=lb.MediaType.Conversational,\n ontology_kind=lb.OntologyKind.ModelEvaluation,\n)", | ||
| "cell_type": "code", | ||
| "outputs": [], | ||
| "execution_count": null | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Option B: `client.create_ontology_from_feature_schemas`\n", | ||
| "Ontologies can also be created with feature schema IDs. This makes your ontologies with existing features compared to generating new features. You can get these features by going to the _Schema_ tab inside Labelbox. (uncomment the below code block for this option)" | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": "# ontology = client.create_ontology_from_feature_schemas(\n# \"MCE ontology\",\n# feature_schema_ids=[\"<list of feature schema ids\"],\n# media_type=lb.MediaType.Conversational,\n# ontology_kind=lb.OntologyKind.ModelEvaluation,\n# )", | ||
| "cell_type": "code", | ||
| "outputs": [], | ||
| "execution_count": null | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Set Up Model Chat Evaluation Project\n", | ||
| "You do not have to create data rows with a model evaluation project; instead, they are generated for you when you create the project. The method you use to create your project is `client.create_model_evaluation_project`, which takes the same parameters as the traditional `client.create_project` but with a few specific additional parameters. \n", | ||
| "\n", | ||
| "__Setting up different models that you are evaluating can only be done inside the Labelbox platform__\n", | ||
| "\n", | ||
| "#### Parameters\n", | ||
| "When using `client.create_model_evaluation_project` the following parameters are needed:\n", | ||
| "\n", | ||
| "- `create_model_evaluation_project` parameters:\n", | ||
| "\n", | ||
| " - `name`: The name of your new project.\n", | ||
| "\n", | ||
| " - `description`: An optional description of your project.\n", | ||
| "\n", | ||
| " - `media_type`: The type of assets that this project will accept. This should be set to lb.MediaType.Conversational\n", | ||
| "\n", | ||
| " - `dataset_name`: The name of the dataset where the generated data rows will be located. Include this parameter only needed if wanting to create a new dataset.\n", | ||
| "\n", | ||
| " - `dataset_id`: An optional dataset ID of an existing Labelbox dataset. Include this parameter if you are wanting to append to an existing MCE dataset.\n", | ||
| "\n", | ||
| " - `data_row_count`: The number of data row assets that will be generated and used with your project.\n" | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": "project = client.create_model_evaluation_project(\n name=\"Demo MCE Project\",\n media_type=lb.MediaType.Conversational,\n dataset_name=\"Demo MCE dataset\",\n data_row_count=100,\n)\n\n# Setup project with ontology created above\nproject.setup_editor(ontology)", | ||
| "cell_type": "code", | ||
| "outputs": [], | ||
| "execution_count": null | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Exporting Model Chat Evaluation Project\n", | ||
| "Exporting from a Model Chat Evaluation project works the same as exporting from other projects. In this example, your export will be shown as empty unless you have created labels inside the Labelbox platform. Please review our [Model Chat Evaluation Export](https://docs.labelbox.com/reference/export-model-chat-evaluation-annotations) guide for a sample export." | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": "# Start export from project\nexport_task = project.export()\nexport_task.wait_till_done()\n\n# Conditional if task has errors\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n # Start export stream\n stream = export_task.get_buffered_stream()\n\n # Iterate through data rows\n for data_row in stream:\n print(data_row.json)", | ||
| "cell_type": "code", | ||
| "outputs": [], | ||
| "execution_count": null | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Clean Up\n", | ||
| "\n", | ||
| "This section serves as an optional clean-up step to delete the Labelbox assets created within this guide. You will need to uncomment the delete methods shown." | ||
| ], | ||
| "cell_type": "markdown" | ||
| }, | ||
| { | ||
| "metadata": {}, | ||
| "source": "# project.delete()\n# client.delete_unused_ontology(ontology.uid)\n# dataset.delete()", | ||
| "cell_type": "code", | ||
| "outputs": [], | ||
| "execution_count": null | ||
| } | ||
| ] | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.