From a3f9fb7346fbeb388181dfa346340cdd47e0b3a4 Mon Sep 17 00:00:00 2001 From: Gabefire <33893811+Gabefire@users.noreply.github.com> Date: Mon, 3 Jun 2024 12:18:15 -0500 Subject: [PATCH 1/4] added methods to modify --- .../model_chat_evaluation_project.ipynb | 335 +++++++++++++++--- 1 file changed, 279 insertions(+), 56 deletions(-) diff --git a/examples/project_configuration/model_chat_evaluation_project.ipynb b/examples/project_configuration/model_chat_evaluation_project.ipynb index ede272b78..1164be5ce 100644 --- a/examples/project_configuration/model_chat_evaluation_project.ipynb +++ b/examples/project_configuration/model_chat_evaluation_project.ipynb @@ -1,18 +1,16 @@ { - "nbformat": 4, - "nbformat_minor": 2, - "metadata": {}, "cells": [ { + "cell_type": "markdown", "metadata": {}, "source": [ - "", - " ", + "\n", + " \n", "\n" - ], - "cell_type": "markdown" + ] }, { + "cell_type": "markdown", "metadata": {}, "source": [ "\n", @@ -24,10 +22,10 @@ "\n", "" - ], - "cell_type": "markdown" + ] }, { + "cell_type": "markdown", "metadata": {}, "source": [ "# Model Chat Evaluation Project Setup\n", @@ -40,102 +38,165 @@ "- `client.create_ontology`: Methods used to create Labelbox ontologies for MCE project this requires an `ontology_kind` parameter set to `lb.OntologyKind.ModelEvaluation`\n", "\n", "- `client.create_ontology_from_feature_schemas`: Similar to `client.create_ontology` but from a list of `feature schema ids` designed to allow you to use existing features instead of creating new features. This also requires an `ontology_kind` set to `lb.OntologyKind.ModelEvaluation`." - ], - "cell_type": "markdown" + ] }, { + "cell_type": "markdown", "metadata": {}, "source": [ "## Set Up" - ], - "cell_type": "markdown" + ] }, { - "metadata": {}, - "source": "%pip install -q \"labelbox[data]\"", "cell_type": "code", - "outputs": [], - "execution_count": null + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install -q --upgrade \"labelbox[data]\"" + ] }, { - "metadata": {}, - "source": "import labelbox as lb", "cell_type": "code", + "execution_count": 22, + "metadata": {}, "outputs": [], - "execution_count": null + "source": [ + "import labelbox as lb" + ] }, { + "cell_type": "markdown", "metadata": {}, "source": [ "## API Key and Client\n", "Provide a valid API key below in order to properly connect to the Labelbox client. Please review [Create API key guide](https://docs.labelbox.com/reference/create-api-key) for more information." - ], - "cell_type": "markdown" + ] }, { - "metadata": {}, - "source": "API_KEY = None\nclient = lb.Client(api_key=API_KEY)", "cell_type": "code", + "execution_count": 23, + "metadata": {}, "outputs": [], - "execution_count": null + "source": [ + "API_KEY = None\n", + "client = lb.Client(api_key=API_KEY)" + ] }, { + "cell_type": "markdown", "metadata": {}, "source": [ "## Example: Create Model Chat Evaluation Project\n", "\n", "The steps to creating a Model Chat Evaluation Project through the Labelbox-Python SDK are similar to creating a regular project. However, they vary slightly, and we will showcase the different methods in this example workflow." - ], - "cell_type": "markdown" + ] }, { + "cell_type": "markdown", "metadata": {}, "source": [ "### Create a Model Chat Evaluation Ontology\n", "\n", "You can create ontologies for Model Evaluation projects the same way as creating ontologies for other projects with the only requirement of passing in a `ontology_kind` parameter which needs set to `lb.OntologyKind.ModelEvaluation`. You can create ontologies with two methods: `client.create_ontology` and `client.create_ontology_from_feature_schemas`." - ], - "cell_type": "markdown" + ] }, { + "cell_type": "markdown", "metadata": {}, "source": [ "#### Option A: `client.create_ontology`\n", "\n", "Typically, you create ontologies and generate the associated features at the same time. Below is an example of creating an ontology for your model chat evaluation project using supported tools and classifications. For information on supported annotation types visit our [model chat evaluation](https://docs.labelbox.com/docs/model-chat-evaluation#supported-annotation-types) guide." - ], - "cell_type": "markdown" + ] }, { - "metadata": {}, - "source": "ontology_builder = lb.OntologyBuilder(\n tools=[\n lb.Tool(\n tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,\n name=\"single select feature\",\n ),\n lb.Tool(\n tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,\n name=\"multi select feature\",\n ),\n lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name=\"ranking feature\"),\n ],\n classifications=[\n lb.Classification(\n class_type=lb.Classification.Type.CHECKLIST,\n name=\"checklist feature\",\n options=[\n lb.Option(value=\"option 1\", label=\"option 1\"),\n lb.Option(value=\"option 2\", label=\"option 2\"),\n ],\n ),\n lb.Classification(\n class_type=lb.Classification.Type.RADIO,\n name=\"radio_question\",\n options=[\n lb.Option(value=\"first_radio_answer\"),\n lb.Option(value=\"second_radio_answer\"),\n ],\n ),\n ],\n)\n\n# Create ontology\nontology = client.create_ontology(\n \"MCE ontology\",\n ontology_builder.asdict(),\n media_type=lb.MediaType.Conversational,\n ontology_kind=lb.OntologyKind.ModelEvaluation,\n)", "cell_type": "code", + "execution_count": 24, + "metadata": {}, "outputs": [], - "execution_count": null + "source": [ + "ontology_builder = lb.OntologyBuilder(\n", + " tools=[\n", + " lb.Tool(\n", + " tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,\n", + " name=\"single select feature\",\n", + " ),\n", + " lb.Tool(\n", + " tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,\n", + " name=\"multi select feature\",\n", + " ),\n", + " lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name=\"ranking feature\"),\n", + " ],\n", + " classifications=[\n", + " lb.Classification(\n", + " class_type=lb.Classification.Type.CHECKLIST,\n", + " name=\"checklist feature\",\n", + " options=[\n", + " lb.Option(value=\"option 1\", label=\"option 1\"),\n", + " lb.Option(value=\"option 2\", label=\"option 2\"),\n", + " ],\n", + " ),\n", + " lb.Classification(\n", + " class_type=lb.Classification.Type.RADIO,\n", + " name=\"radio_question\",\n", + " options=[\n", + " lb.Option(value=\"first_radio_answer\"),\n", + " lb.Option(value=\"second_radio_answer\"),\n", + " ],\n", + " ),\n", + " ],\n", + ")\n", + "\n", + "# Create ontology\n", + "ontology = client.create_ontology(\n", + " \"MCE ontology\",\n", + " ontology_builder.asdict(),\n", + " media_type=lb.MediaType.Conversational,\n", + " ontology_kind=lb.OntologyKind.ModelEvaluation,\n", + ")" + ] }, { + "cell_type": "markdown", "metadata": {}, "source": [ "### Option B: `client.create_ontology_from_feature_schemas`\n", "Ontologies can also be created with feature schema IDs. This makes your ontologies with existing features compared to generating new features. You can get these features by going to the _Schema_ tab inside Labelbox. (uncomment the below code block for this option)" - ], - "cell_type": "markdown" + ] }, { - "metadata": {}, - "source": "# ontology = client.create_ontology_from_feature_schemas(\n# \"MCE ontology\",\n# feature_schema_ids=[\" Date: Mon, 3 Jun 2024 17:21:13 +0000 Subject: [PATCH 2/4] :art: Cleaned --- .../model_chat_evaluation_project.ipynb | 311 +++++------------- 1 file changed, 77 insertions(+), 234 deletions(-) diff --git a/examples/project_configuration/model_chat_evaluation_project.ipynb b/examples/project_configuration/model_chat_evaluation_project.ipynb index 1164be5ce..f2882d617 100644 --- a/examples/project_configuration/model_chat_evaluation_project.ipynb +++ b/examples/project_configuration/model_chat_evaluation_project.ipynb @@ -1,16 +1,18 @@ { + "nbformat": 4, + "nbformat_minor": 2, + "metadata": {}, "cells": [ { - "cell_type": "markdown", "metadata": {}, "source": [ - "\n", - " \n", + "", + " ", "\n" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", "metadata": {}, "source": [ "\n", @@ -22,10 +24,10 @@ "\n", "" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", "metadata": {}, "source": [ "# Model Chat Evaluation Project Setup\n", @@ -38,160 +40,95 @@ "- `client.create_ontology`: Methods used to create Labelbox ontologies for MCE project this requires an `ontology_kind` parameter set to `lb.OntologyKind.ModelEvaluation`\n", "\n", "- `client.create_ontology_from_feature_schemas`: Similar to `client.create_ontology` but from a list of `feature schema ids` designed to allow you to use existing features instead of creating new features. This also requires an `ontology_kind` set to `lb.OntologyKind.ModelEvaluation`." - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", "metadata": {}, "source": [ "## Set Up" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "code", - "execution_count": 21, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", - "Note: you may need to restart the kernel to use updated packages.\n" - ] - } - ], - "source": [ - "%pip install -q --upgrade \"labelbox[data]\"" - ] + "source": "%pip install -q --upgrade \"labelbox[data]\"", + "cell_type": "code", + "outputs": [], + "execution_count": null }, { - "cell_type": "code", - "execution_count": 22, "metadata": {}, + "source": "import labelbox as lb", + "cell_type": "code", "outputs": [], - "source": [ - "import labelbox as lb" - ] + "execution_count": null }, { - "cell_type": "markdown", "metadata": {}, "source": [ "## API Key and Client\n", "Provide a valid API key below in order to properly connect to the Labelbox client. Please review [Create API key guide](https://docs.labelbox.com/reference/create-api-key) for more information." - ] + ], + "cell_type": "markdown" }, { - "cell_type": "code", - "execution_count": 23, "metadata": {}, + "source": "API_KEY = None\nclient = lb.Client(api_key=API_KEY)", + "cell_type": "code", "outputs": [], - "source": [ - "API_KEY = None\n", - "client = lb.Client(api_key=API_KEY)" - ] + "execution_count": null }, { - "cell_type": "markdown", "metadata": {}, "source": [ "## Example: Create Model Chat Evaluation Project\n", "\n", "The steps to creating a Model Chat Evaluation Project through the Labelbox-Python SDK are similar to creating a regular project. However, they vary slightly, and we will showcase the different methods in this example workflow." - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", "metadata": {}, "source": [ "### Create a Model Chat Evaluation Ontology\n", "\n", "You can create ontologies for Model Evaluation projects the same way as creating ontologies for other projects with the only requirement of passing in a `ontology_kind` parameter which needs set to `lb.OntologyKind.ModelEvaluation`. You can create ontologies with two methods: `client.create_ontology` and `client.create_ontology_from_feature_schemas`." - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", "metadata": {}, "source": [ "#### Option A: `client.create_ontology`\n", "\n", "Typically, you create ontologies and generate the associated features at the same time. Below is an example of creating an ontology for your model chat evaluation project using supported tools and classifications. For information on supported annotation types visit our [model chat evaluation](https://docs.labelbox.com/docs/model-chat-evaluation#supported-annotation-types) guide." - ] + ], + "cell_type": "markdown" }, { - "cell_type": "code", - "execution_count": 24, "metadata": {}, + "source": "ontology_builder = lb.OntologyBuilder(\n tools=[\n lb.Tool(\n tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,\n name=\"single select feature\",\n ),\n lb.Tool(\n tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,\n name=\"multi select feature\",\n ),\n lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name=\"ranking feature\"),\n ],\n classifications=[\n lb.Classification(\n class_type=lb.Classification.Type.CHECKLIST,\n name=\"checklist feature\",\n options=[\n lb.Option(value=\"option 1\", label=\"option 1\"),\n lb.Option(value=\"option 2\", label=\"option 2\"),\n ],\n ),\n lb.Classification(\n class_type=lb.Classification.Type.RADIO,\n name=\"radio_question\",\n options=[\n lb.Option(value=\"first_radio_answer\"),\n lb.Option(value=\"second_radio_answer\"),\n ],\n ),\n ],\n)\n\n# Create ontology\nontology = client.create_ontology(\n \"MCE ontology\",\n ontology_builder.asdict(),\n media_type=lb.MediaType.Conversational,\n ontology_kind=lb.OntologyKind.ModelEvaluation,\n)", + "cell_type": "code", "outputs": [], - "source": [ - "ontology_builder = lb.OntologyBuilder(\n", - " tools=[\n", - " lb.Tool(\n", - " tool=lb.Tool.Type.MESSAGE_SINGLE_SELECTION,\n", - " name=\"single select feature\",\n", - " ),\n", - " lb.Tool(\n", - " tool=lb.Tool.Type.MESSAGE_MULTI_SELECTION,\n", - " name=\"multi select feature\",\n", - " ),\n", - " lb.Tool(tool=lb.Tool.Type.MESSAGE_RANKING, name=\"ranking feature\"),\n", - " ],\n", - " classifications=[\n", - " lb.Classification(\n", - " class_type=lb.Classification.Type.CHECKLIST,\n", - " name=\"checklist feature\",\n", - " options=[\n", - " lb.Option(value=\"option 1\", label=\"option 1\"),\n", - " lb.Option(value=\"option 2\", label=\"option 2\"),\n", - " ],\n", - " ),\n", - " lb.Classification(\n", - " class_type=lb.Classification.Type.RADIO,\n", - " name=\"radio_question\",\n", - " options=[\n", - " lb.Option(value=\"first_radio_answer\"),\n", - " lb.Option(value=\"second_radio_answer\"),\n", - " ],\n", - " ),\n", - " ],\n", - ")\n", - "\n", - "# Create ontology\n", - "ontology = client.create_ontology(\n", - " \"MCE ontology\",\n", - " ontology_builder.asdict(),\n", - " media_type=lb.MediaType.Conversational,\n", - " ontology_kind=lb.OntologyKind.ModelEvaluation,\n", - ")" - ] + "execution_count": null }, { - "cell_type": "markdown", "metadata": {}, "source": [ "### Option B: `client.create_ontology_from_feature_schemas`\n", "Ontologies can also be created with feature schema IDs. This makes your ontologies with existing features compared to generating new features. You can get these features by going to the _Schema_ tab inside Labelbox. (uncomment the below code block for this option)" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "code", - "execution_count": 25, "metadata": {}, + "source": "# ontology = client.create_ontology_from_feature_schemas(\n# \"MCE ontology\",\n# feature_schema_ids=[\" Date: Mon, 3 Jun 2024 12:22:29 -0500 Subject: [PATCH 3/4] Typos --- .../model_chat_evaluation_project.ipynb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/examples/project_configuration/model_chat_evaluation_project.ipynb b/examples/project_configuration/model_chat_evaluation_project.ipynb index f2882d617..b6ade562f 100644 --- a/examples/project_configuration/model_chat_evaluation_project.ipynb +++ b/examples/project_configuration/model_chat_evaluation_project.ipynb @@ -176,9 +176,9 @@ "\n", "- `name`: Name of the model config.\n", "\n", - "- `model_id`: ID of model to configure. You must obtain this through the UI by navigating to the Model tab, selecting the model you are trying to use and coping the id inside the URL. For supported models visit the [Model chat evaluation page](https://docs.labelbox.com/docs/model-chat-evaluation#supported-annotation-types).\n", + "- `model_id`: The ID of the model to configure. You must obtain this through the UI by navigating to the Model tab, selecting the model you are trying to use, and copying the id inside the URL. For supported models, visit the [Model chat evaluation page](https://docs.labelbox.com/docs/model-chat-evaluation#supported-annotation-types).\n", "\n", - "- `inference_params`: JSON of model configuration parameters. This will vary depending on the model you are trying to set up. It is recommended to first setup a model config inside the UI to learn all the associated parameters.\n", + "- `inference_params`: JSON of model configuration parameters. This will vary depending on the model you are trying to set up. It is recommended to first set up a model config inside the UI to learn all the associated parameters.\n", "\n", "For the example below, we will be setting up a Google Gemini 1.5 Pro model config." ], @@ -195,7 +195,7 @@ "metadata": {}, "source": [ "### Attaching Model Config to Project\n", - "You can attach and remove model configs to your project by using `project.add_model_config` or `project.remove_model_config`. Both methods take just a `model_config` ID." + "You can attach and remove model configs to your project using `project.add_model_config` or `project.remove_model_config`. Both methods take just a `model_config` ID." ], "cell_type": "markdown" }, @@ -210,7 +210,7 @@ "metadata": {}, "source": [ "### Delete Model Config\n", - "You can also delete model configs by using the `client.delete_model_config`. You just need to pass in the `model_config` ID in order to delete your model config. You can obtain this ID from your created model config above or get the model configs directly from your project using `project.project_model_configs` and then iterating through the list of model configs attached to your project. Uncomment the code below to delete your model configs. " + "You can also delete model configs using the `client.delete_model_config`. You just need to pass in the `model_config` ID in order to delete your model config. You can obtain this ID from your created model config above or get the model configs directly from your project using `project.project_model_configs` and then iterating through the list of model configs attached to your project. Uncomment the code below to delete your model configs. " ], "cell_type": "markdown" }, @@ -260,4 +260,4 @@ "execution_count": null } ] -} \ No newline at end of file +} From d375a7c9b4d4d530031c6b14a3045b1fc32e4f54 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Mon, 3 Jun 2024 17:23:22 +0000 Subject: [PATCH 4/4] :art: Cleaned --- .../project_configuration/model_chat_evaluation_project.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/project_configuration/model_chat_evaluation_project.ipynb b/examples/project_configuration/model_chat_evaluation_project.ipynb index b6ade562f..26cd9e6de 100644 --- a/examples/project_configuration/model_chat_evaluation_project.ipynb +++ b/examples/project_configuration/model_chat_evaluation_project.ipynb @@ -260,4 +260,4 @@ "execution_count": null } ] -} +} \ No newline at end of file