From 652a8d5c68ea8248b49fe1df82037a8c12779029 Mon Sep 17 00:00:00 2001 From: Gabe <33893811+Gabefire@users.noreply.github.com> Date: Thu, 18 Jul 2024 13:51:18 -0500 Subject: [PATCH 1/5] Adding back in export migration guide --- .../export_v1_to_v2_migration_support.ipynb | 1220 +++++++++++++++++ 1 file changed, 1220 insertions(+) create mode 100644 examples/exports/export_v1_to_v2_migration_support.ipynb diff --git a/examples/exports/export_v1_to_v2_migration_support.ipynb b/examples/exports/export_v1_to_v2_migration_support.ipynb new file mode 100644 index 000000000..c78a62f02 --- /dev/null +++ b/examples/exports/export_v1_to_v2_migration_support.ipynb @@ -0,0 +1,1220 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "4E7YgJ6CJ6Qc", + "metadata": { + "id": "4E7YgJ6CJ6Qc" + }, + "source": [ + "\n", + " \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "MrE_dQKBJ6Qe", + "metadata": { + "id": "MrE_dQKBJ6Qe" + }, + "source": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "nxOw67b-J6Qe", + "metadata": { + "id": "nxOw67b-J6Qe" + }, + "source": [ + "# Export V1 to V2 migration" + ] + }, + { + "cell_type": "markdown", + "id": "qCY12qC8J6Qf", + "metadata": { + "id": "qCY12qC8J6Qf" + }, + "source": [ + "**Export V1 will no longer be available in any version of the SDK starting on April 2024**. We recommend users to plan accordingly.\n", + "\n", + "This notebook is designed to help users identify alternative V2 export methods that can serve as replacements for V1 methods." + ] + }, + { + "cell_type": "markdown", + "id": "5IBNbHJAJ6Qf", + "metadata": { + "id": "5IBNbHJAJ6Qf" + }, + "source": [ + "### Key changes included in export V2 methods ( ``export()`` and ``export_v2()``):\n", + "1. Added flexibility to only export the data that is needed. The new methods include parameters and filters to give you more granular control over your exports.\n", + "2. Added functionality to stream an **unlimited** number of data rows using ``export()`` (available on SDK >=3.56). Upgrading to `export()` is recommended as it is a more scalable solution.\n", + "\n", + "For complete details on how to use export V2 methods please see the [Export V2 methods](https://docs.labelbox.com/reference/label-export#export-v2-methods) documentation." + ] + }, + { + "cell_type": "markdown", + "id": "L2Xb8V5PJ6Qf", + "metadata": { + "id": "L2Xb8V5PJ6Qf" + }, + "source": [ + "### Export V1 deprecated methods:\n", + "Project methods :\n", + "1. ```project.export_labels()```\n", + "2. ```project.label_generator()```\n", + "3. ```project.export_queued_data_rows()```\n", + "\n", + "Dataset methods:\n", + "1. ```dataset.export_data_rows()```\n", + "\n", + "Batch methods:\n", + "1. ```batch.export_data_rows()```\n", + "\n", + "Model methods :\n", + "1. ```model_run.export_labels()```\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "ZEzKj4TwJ6Qf", + "metadata": { + "id": "ZEzKj4TwJ6Qf" + }, + "source": [ + "# Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8egYmUB1J6Qf", + "metadata": { + "id": "8egYmUB1J6Qf" + }, + "outputs": [], + "source": [ + "!pip install -q \"labelbox[data]\"" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "S8YPoX2zJ6Qg", + "metadata": { + "id": "S8YPoX2zJ6Qg" + }, + "outputs": [], + "source": [ + "import labelbox as lb\n", + "import pprint\n", + "pp = pprint.PrettyPrinter(width=30, compact=True)\n" + ] + }, + { + "cell_type": "markdown", + "id": "ZMLM7LejJ6Qg", + "metadata": { + "id": "ZMLM7LejJ6Qg" + }, + "source": [ + "## API Key and Client\n", + "See the developer guide for [creating an API key](https://docs.labelbox.com/reference/create-api-key)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sZk3-cRqJ6Qg", + "metadata": { + "id": "sZk3-cRqJ6Qg" + }, + "outputs": [], + "source": [ + "API_KEY = \"\"\n", + "client = lb.Client(api_key=API_KEY)" + ] + }, + { + "cell_type": "markdown", + "id": "aCvPcEHIJ6Qg", + "metadata": { + "id": "aCvPcEHIJ6Qg" + }, + "source": [ + "# Exports V1 to V2 guidelines\n", + "The follow sections will demonstrate how to use the export V2 methods to fetch data from your projects, datasets, batches and model runs." + ] + }, + { + "cell_type": "markdown", + "id": "S5g48GJPJ6Qg", + "metadata": { + "id": "S5g48GJPJ6Qg" + }, + "source": [ + "## Export labels from a project\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "WDlK0c-rJ6Qg", + "metadata": { + "id": "WDlK0c-rJ6Qg" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"\"\n", + "project = client.get_project(PROJECT_ID)" + ] + }, + { + "cell_type": "markdown", + "id": "-z3K7jDQJ6Qg", + "metadata": { + "id": "-z3K7jDQJ6Qg" + }, + "source": [ + "##### Export V1 (deprecated) \n", + "1. ```project.export_labels()```\n", + " - Parameters: \n", + " - ```download: bool = False```\n", + " - ```timeout_seconds: int = 1800```\n", + " - Output : (str | List[Dict[Any, Any]] | None)" + ] + }, + { + "cell_type": "markdown", + "id": "mJqPO1GGJ6Qg", + "metadata": { + "id": "mJqPO1GGJ6Qg" + }, + "source": [ + "For a comprehensive example of Export V1 ``project.export_labels()`` output, please refer to our documentation: [Export V1 sample export](https://docs.labelbox.com/reference/export-image-annotations#sample-project-export)" + ] + }, + { + "cell_type": "markdown", + "id": "8UapSOOnJ6Qh", + "metadata": { + "id": "8UapSOOnJ6Qh" + }, + "source": [ + "2. ```project.label_generator()```\n", + " - Parameters:\n", + " - ```timeout_seconds: int = 600```\n", + " - Output: LabelGenerator" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5ALbzsbEJ6Qh", + "metadata": { + "id": "5ALbzsbEJ6Qh" + }, + "outputs": [], + "source": [ + "# Single entry from the output of project.label_generator() (deprecated)\n", + "# Label objects will not be deprecated.\n", + "single_output_from_generator = \"\"\"\n", + "\n", + "Label(\n", + " uid='clrf5csho2ihx07ilffgp2fzj',\n", + " data=ImageData(\n", + " im_bytes=None,\n", + " file_path=None,\n", + " url='https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg',\n", + " arr=None\n", + " ),\n", + " annotations=[\n", + " ObjectAnnotation(\n", + " confidence=None,\n", + " name='bounding_box',\n", + " feature_schema_id='clrf5ck4a0b9b071paa9ncu15',\n", + " extra={\n", + " 'instanceURI': 'https://api.labelbox.com/masks/feature/clrf5csvi6ofm07lsf9pygwvi?token='\n", + " 'color': '#ff0000',\n", + " 'feature_id': 'clrf5csvi6ofm07lsf9pygwvi',\n", + " 'value': 'bounding_box',\n", + " 'page': None,\n", + " 'unit': None\n", + " },\n", + " value=Rectangle(\n", + " extra={},\n", + " start=Point(extra={}, x=2096.0, y=1264.0),\n", + " end=Point(extra={}, x=2240.0, y=1689.0)\n", + " ),\n", + " classifications=[]\n", + " ),\n", + " # Add more annotations as needed\n", + " # ...\n", + " ],\n", + " extra={\n", + " 'Created By': 'aovalle@labelbox.com',\n", + " 'Project Name': 'Image Annotation Import Demo',\n", + " 'Created At': '2024-01-15T16:35:59.000Z',\n", + " 'Updated At': '2024-01-15T16:51:56.000Z',\n", + " 'Seconds to Label': 66.0,\n", + " 'Agreement': -1.0,\n", + " 'Benchmark Agreement': -1.0,\n", + " 'Benchmark ID': None,\n", + " 'Dataset Name': 'image-demo-dataset',\n", + " 'Reviews': [],\n", + " 'View Label': 'https://editor.labelbox.com?project=clrf5ckex09m9070x1te223u5&label=clrf5csho2ihx07ilffgp2fzj',\n", + " 'Has Open Issues': 0.0,\n", + " 'Skipped': False,\n", + " 'media_type': 'image',\n", + " 'Data Split': None,\n", + " 'Global Key': '2560px-Kitano_Street_Kobe01s5s41102.jpeg'\n", + " }\n", + ")\n", + "\n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "id": "7WJT6y4xJ6Qh", + "metadata": { + "id": "7WJT6y4xJ6Qh" + }, + "source": [ + "##### Export V2\n", + "\n", + "For complete details on the supported filters abd parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", + "\n", + "1. ```project.export()``` : Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "\n", + " - Parameters: \n", + " - ```\"label_details\": True```\n", + " - ```\"attachments\": True```\n", + " - ```\"data_row_details\": True```\n", + " - ```\"project_details\": True```\n", + " - ```\"label_details\": True```\n", + " - ```\"performance_details\": True```\n", + " - Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ouzz6t5PJ6Qh", + "metadata": { + "id": "ouzz6t5PJ6Qh" + }, + "outputs": [], + "source": [ + "## Set the export parameters to only export labels\n", + "export_params= {\n", + " \"attachments\": True,\n", + " \"metadata_fields\": True,\n", + " \"data_row_details\": True,\n", + " \"project_details\": True,\n", + " \"label_details\": True,\n", + " \"performance_details\": True\n", + "}\n", + "# You also have the option to include additional filtering to narrow down the list of labels\n", + "filters = {}\n", + "\n", + "export_task = project.export(params=export_params, filters=filters)\n", + "export_task.wait_till_done()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "JpGRBQPpJ6Qh", + "metadata": { + "id": "JpGRBQPpJ6Qh" + }, + "outputs": [], + "source": [ + "# Provide results with JSON converter\n", + "# Returns streamed JSON output strings from export task results/errors, one by one\n", + "\n", + "# Callback used for JSON Converter\n", + "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", + " print(output.json)\n", + "\n", + "if export_task.has_errors():\n", + " export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.ERRORS\n", + " ).start(stream_handler=lambda error: print(error))\n", + "\n", + "if export_task.has_result():\n", + " export_json = export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.RESULT\n", + " ).start(stream_handler=json_stream_handler)\n", + "\n", + "print(\"file size: \", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))\n", + "print(\"line count: \", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))\n" + ] + }, + { + "cell_type": "markdown", + "id": "QZxfvXBHJ6Qh", + "metadata": { + "id": "QZxfvXBHJ6Qh" + }, + "source": [ + "## Export queued (\"To Label\") data rows from a project" + ] + }, + { + "cell_type": "markdown", + "id": "j3n3kPgrJ6Qh", + "metadata": { + "id": "j3n3kPgrJ6Qh" + }, + "source": [ + "##### Export V1 (deprecated): \n", + "1. ``project.export_queued_data_rows()`` :\n", + " - Parameters:\n", + " - ``timeout_seconds: int = 120``\n", + " - ``include_metadata: bool = False``\n", + " - Output: List[Dict[str, str]]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "puNxgxlUJ6Qh", + "metadata": { + "id": "puNxgxlUJ6Qh" + }, + "outputs": [], + "source": [ + "# Single entry from the output of project.export_queued_data_rows() (deprecated)\n", + "single_output_example = \"\"\"\n", + "[\n", + " {'id': 'clpouak6nap2g0783ajd1d6pf',\n", + " 'createdAt': '2023-12-03T02:04:34.062Z',\n", + " 'updatedAt': '2023-12-03T02:05:33.797Z',\n", + " 'externalId': None,\n", + " 'globalKey': 'b57c9ab2-304f-4c17-ba5f-c536f39a6a46',\n", + " 'metadataFields': [],\n", + " 'customMetadata': [],\n", + " 'rowData': 'https://storage.googleapis.com/labelbox-developer-testing-assets/image/data_files/santa.jpeg',\n", + " 'mediaAttributes': {'assetType': 'image',\n", + " 'contentLength': 305973,\n", + " 'height': 1333,\n", + " 'mimeType': 'image/jpeg',\n", + " 'subType': 'jpeg',\n", + " 'superType': 'image',\n", + " 'width': 2000}}\n", + "]\n", + "\n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "id": "vFqPfKcSJ6Qh", + "metadata": { + "id": "vFqPfKcSJ6Qh" + }, + "source": [ + "##### Export V2\n", + "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", + "\n", + "1. ```project.export()``` : Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "\n", + " - Parameters (Minimum required parameters): \n", + " - ```\"data_row_details\": True```\n", + " - ```\"project_details\": True```\n", + " - Required filters:\n", + " - ``` \"workflow_status\": \"ToLabel\"```\n", + " - Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "HSTLb8bXJ6Qh", + "metadata": { + "id": "HSTLb8bXJ6Qh" + }, + "outputs": [], + "source": [ + "export_params= {\n", + " \"attachments\": True, # Set to true if you want to export attachments\n", + " \"metadata_fields\": True, # Set to true if you want to export metadata\n", + " \"data_row_details\": True,\n", + " \"project_details\": True\n", + "}\n", + "filters = {\n", + " \"workflow_status\": \"ToLabel\" ## Using this filter will only export queued data rows\n", + "}\n", + "\n", + "# An ExportTask is returned, this provides additional information about the status of your task, such as\n", + "# any errors encountered and includes additional methods to stream your data\n", + "\n", + "\n", + "export_task = project.export(params=export_params, filters=filters)\n", + "export_task.wait_till_done()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8sxoNt6MJ6Qh", + "metadata": { + "id": "8sxoNt6MJ6Qh" + }, + "outputs": [], + "source": [ + "# Provide results with JSON converter\n", + "# Returns streamed JSON output strings from export task results/errors, one by one\n", + "\n", + "# Callback used for JSON Converter\n", + "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", + " print(output.json)\n", + "\n", + "\n", + "if export_task.has_errors():\n", + " export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.ERRORS\n", + " ).start(stream_handler=lambda error: print(error))\n", + "\n", + "if export_task.has_result():\n", + " export_json = export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.RESULT\n", + " ).start(stream_handler=json_stream_handler)\n", + "\n", + "print(\"file size: \", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))\n", + "print(\"line count: \", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))" + ] + }, + { + "cell_type": "markdown", + "id": "cCnY3j3NJ6Qh", + "metadata": { + "id": "cCnY3j3NJ6Qh" + }, + "source": [ + "## Export data rows from a Dataset" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3VGQZaHPJ6Qh", + "metadata": { + "id": "3VGQZaHPJ6Qh" + }, + "outputs": [], + "source": [ + "DATASET_ID = \"\"\n", + "dataset = client.get_dataset(DATASET_ID)" + ] + }, + { + "cell_type": "markdown", + "id": "Ws7WCa0eJ6Qh", + "metadata": { + "id": "Ws7WCa0eJ6Qh" + }, + "source": [ + "#### Export V1 (deprecated):\n", + "\n", + "1. ```dataset.export_data_rows()```\n", + " - Parameters: \n", + " - ``timeout_seconds=120``\n", + " - ``include_metadata: bool = True``\n", + " - Output:\n", + " - Data row object generator\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c_93zHuKJ6Qh", + "metadata": { + "id": "c_93zHuKJ6Qh" + }, + "outputs": [], + "source": [ + "# Single entry from the output of dataset.export_data_rows() (deprecated)\n", + "# Data row objects will not be deprecated.\n", + "\n", + "single_output_from_data_row_generator = \"\"\"\n", + "\n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "id": "i82Jg4_bJ6Qh", + "metadata": { + "id": "i82Jg4_bJ6Qh" + }, + "source": [ + "#### Export V2\n", + "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", + "\n", + "1. ```project.export()``` : Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "\n", + " - Parameters (minimum required parameters): \n", + " - ``\"data_row_details\": True``\n", + " - Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "vlkTDFDPJ6Qi", + "metadata": { + "id": "vlkTDFDPJ6Qi" + }, + "outputs": [], + "source": [ + "export_params = {\n", + " \"attachments\": True, # Set to true if you want to export attachments\n", + " \"metadata_fields\": True, # Set to true if you want to export metadata\n", + " \"data_row_details\": True,\n", + "}\n", + "filters = {}\n", + "\n", + "# A task is returned, this provides additional information about the status of your task, such as\n", + "# any errors encountered\n", + "export_task = dataset.export(params=export_params, filters=filters)\n", + "export_task.wait_till_done()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "k0JIGqNlJ6Qi", + "metadata": { + "id": "k0JIGqNlJ6Qi" + }, + "outputs": [], + "source": [ + "# Provide results with JSON converter\n", + "# Returns streamed JSON output strings from export task results/errors, one by one\n", + "\n", + "# Callback used for JSON Converter\n", + "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", + " print(output.json)\n", + "\n", + "\n", + "if export_task.has_errors():\n", + " export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.ERRORS\n", + " ).start(stream_handler=lambda error: print(error))\n", + "\n", + "if export_task.has_result():\n", + " export_json = export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.RESULT\n", + " ).start(stream_handler=json_stream_handler)\n", + "\n", + "print(\"file size: \", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))\n", + "print(\"line count: \", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))" + ] + }, + { + "cell_type": "markdown", + "id": "LPv1lghqJ6Qi", + "metadata": { + "id": "LPv1lghqJ6Qi" + }, + "source": [ + "## Export data rows from a batch" + ] + }, + { + "cell_type": "markdown", + "id": "wGO_FSlpJ6Qi", + "metadata": { + "id": "wGO_FSlpJ6Qi" + }, + "source": [ + "#### Export V1 (deprecated):\n", + "1. ```batch.export_data_rows()```\n", + " - Parameters: \n", + " - ``timeout_seconds=120``\n", + " - ``include_metadata: bool = True``\n", + " - Output:\n", + " - Data row object generator" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "TvjQgBjYJ6Qi", + "metadata": { + "id": "TvjQgBjYJ6Qi" + }, + "outputs": [], + "source": [ + "# Single output from batch.export_data_rows() method (deprecated)\n", + "# Data row objects will not be deprecated\n", + "\n", + "single_output_from_data_row_generator = \"\"\"\n", + "\n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "id": "IciDQuDxJ6Qi", + "metadata": { + "id": "IciDQuDxJ6Qi" + }, + "source": [ + "#### Export V2\n", + "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", + "\n", + "1. ```project.export()``` : Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "\n", + " - Required parameters: \n", + " - ```\"data_row_details\": True```,\n", + " - ```\"batch_ids\": [] ```\n", + " - Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ZgjgVg34J6Qi", + "metadata": { + "id": "ZgjgVg34J6Qi" + }, + "outputs": [], + "source": [ + "# Find the batch ID by navigating to \"Batches\" --> \"Manage batches\" --> \"Copy Batch ID\"\n", + "BATCH_ID = \"\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "AGTbvKw7J6Ql", + "metadata": { + "id": "AGTbvKw7J6Ql" + }, + "outputs": [], + "source": [ + "export_params = {\n", + " \"attachments\": True,\n", + " \"metadata_fields\": True,\n", + " \"data_row_details\": True,\n", + " \"project_details\": True,\n", + " \"performance_details\": True,\n", + " \"batch_ids\" : [BATCH_ID] # Include batch ids if you only want to export specific batches, otherwise,\n", + " #you can export all the data without using this parameter\n", + "}\n", + "filters = {}\n", + "\n", + "# A task is returned, this provides additional information about the status of your task, such as\n", + "# any errors encountered\n", + "export_task = project.export(params=export_params, filters=filters)\n", + "export_task.wait_till_done()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "N5ysx8FyJ6Ql", + "metadata": { + "id": "N5ysx8FyJ6Ql" + }, + "outputs": [], + "source": [ + "# Provide results with JSON converter\n", + "# Returns streamed JSON output strings from export task results/errors, one by one\n", + "\n", + "# Callback used for JSON Converter\n", + "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", + " print(output.json)\n", + "\n", + "\n", + "if export_task.has_errors():\n", + " export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.ERRORS\n", + " ).start(stream_handler=lambda error: print(error))\n", + "\n", + "if export_task.has_result():\n", + " export_json = export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.RESULT\n", + " ).start(stream_handler=json_stream_handler)\n", + "\n", + "print(\"file size: \", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))\n", + "print(\"line count: \", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))" + ] + }, + { + "cell_type": "markdown", + "id": "-dTKlTzmJ6Ql", + "metadata": { + "id": "-dTKlTzmJ6Ql" + }, + "source": [ + "## Export data rows from a Model" + ] + }, + { + "cell_type": "markdown", + "id": "ERTdBvxaJ6Ql", + "metadata": { + "id": "ERTdBvxaJ6Ql" + }, + "source": [ + "#### Export V1 (deprecated):\n", + "1. ```model_run.export_labels(downlaod=True)```\n", + " - Parameters: \n", + " - ```download: bool = False```\n", + " - ```timeout_seconds: int = 1800```\n", + " - Output : (str | List[Dict[Any, Any]] | None)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ynE1bihvJ6Ql", + "metadata": { + "id": "ynE1bihvJ6Ql" + }, + "outputs": [], + "source": [ + "# Single output from model_run.export_labels()\n", + "single_output_example = \"\"\"\n", + "[\n", + " {'ID': '1c48a7a0-3016-48e0-b0e3-47430f974869',\n", + " 'Data Split': 'training',\n", + " 'DataRow ID': 'clpqdyf650xd40712pycshy6a',\n", + " 'External ID': './resume/BANKING/99124477.pdf',\n", + " 'Labeled Data': 'https://storage.labelbox.com/cl5bn8qvq1av907xtb3bp8q60%2F8c6afc38-42a4-b2e1-a2e3-1e3b0c2998fc-99124477.pdf?Expires=1706637969726&KeyName=labelbox-assets-key-3&Signature=2nVt3sJ21CbjGS9I64yFquUELRw',\n", + " 'Media Attributes': {'assetType': 'pdf',\n", + " 'contentLength': 42535,\n", + " 'mimeType': 'application/pdf',\n", + " 'pageCount': 3,\n", + " 'subType': 'pdf',\n", + " 'superType': 'application'},\n", + " 'Label': {'objects': [{'featureId': 'b9f3b584-0f45-050a-88d4-39c2a169c8e1',\n", + " 'schemaId': 'clq1ckwbd08jp07z91q9mch5j',\n", + " 'title': 'Test',\n", + " 'value': 'test',\n", + " 'color': '#1CE6FF',\n", + " 'data': {'location': [{'text-bbox': {'page': 1,\n", + " 'top': 158.44,\n", + " 'left': 58.765,\n", + " 'height': 13.691,\n", + " 'width': 78.261}}],\n", + " 'unit': 'POINTS'}}],\n", + " 'classifications': [],\n", + " 'relationships': []}}\n", + " ]\n", + " \"\"\"\n" + ] + }, + { + "cell_type": "markdown", + "id": "Mtr8gaTtJ6Ql", + "metadata": { + "id": "Mtr8gaTtJ6Ql" + }, + "source": [ + "#### Export V2\n", + "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", + "\n", + "1. ```model_run.export()```: Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "\n", + " - Required parameters: \n", + " - ```\"data_row_details\": True```\n", + " - ```\"project_details\": True```\n", + " - ```\"label_details\": True```\n", + " - Required filters:\n", + " - N/A -> Filters not supported\n", + " - Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "iJA1W1NsJ6Ql", + "metadata": { + "id": "iJA1W1NsJ6Ql" + }, + "outputs": [], + "source": [ + "MODEL_RUN_ID = \"\"\n", + "model_run = client.get_model_run(MODEL_RUN_ID)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "wPug5qzJJ6Qm", + "metadata": { + "id": "wPug5qzJJ6Qm" + }, + "outputs": [], + "source": [ + "export_params = {\n", + " \"attachments\": True,\n", + " \"metadata_fields\": True,\n", + " \"data_row_details\": True,\n", + " \"project_details\": True,\n", + " \"performance_details\": True\n", + "}\n", + "\n", + "export_task = model_run.export(params=export_params)\n", + "export_task.wait_till_done()\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ap9fPl6KJ6Qm", + "metadata": { + "id": "ap9fPl6KJ6Qm" + }, + "outputs": [], + "source": [ + "# Provide results with JSON converter\n", + "# Returns streamed JSON output strings from export task results/errors, one by one\n", + "\n", + "# Callback used for JSON Converter\n", + "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", + " print(output.json)\n", + "\n", + "if export_task.has_errors():\n", + " export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.ERRORS\n", + " ).start(stream_handler=lambda error: print(error))\n", + "\n", + "if export_task.has_result():\n", + " export_json = export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.RESULT\n", + " ).start(stream_handler=json_stream_handler)\n", + "\n", + "print(\"file size: \", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))\n", + "print(\"line count: \", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))" + ] + }, + { + "cell_type": "markdown", + "id": "aEgufP_LJ6Qm", + "metadata": { + "id": "aEgufP_LJ6Qm" + }, + "source": [ + "## Export data rows from a video project\n", + "Video projects include additional fields. Please refer to the example below to extract specific fields from video exports.\n" + ] + }, + { + "cell_type": "markdown", + "id": "XuiEWCkqJ6Qm", + "metadata": { + "id": "XuiEWCkqJ6Qm" + }, + "source": [ + "##### Export V1 (deprecated) \n", + "1. ```project.export_labels()```\n", + " - Parameters: \n", + " - ```download: bool = False```\n", + " - ```timeout_seconds: int = 1800```\n", + " - Output : (str | List[Dict[Any, Any]] | None)" + ] + }, + { + "cell_type": "markdown", + "id": "3riNZm2jJ6Qm", + "metadata": { + "id": "3riNZm2jJ6Qm" + }, + "source": [ + "For a comprehensive example of Export V1 ``project.export_labels()`` output, please refer to our documentation: [Export V1 sample export](https://docs.labelbox.com/reference/export-image-annotations#sample-project-export)" + ] + }, + { + "cell_type": "markdown", + "id": "ODXIhOQUJ6Qm", + "metadata": { + "id": "ODXIhOQUJ6Qm" + }, + "source": [ + "2. ```project.video_label_generator()```\n", + " - Parameters:\n", + " - ```timeout_seconds: int = 600```\n", + " - Output: LabelGenerator" + ] + }, + { + "cell_type": "markdown", + "id": "BR0RIC-YJ6Qm", + "metadata": { + "id": "BR0RIC-YJ6Qm" + }, + "source": [ + "##### Export V2\n", + "\n", + "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", + "\n", + "1. ```project.export()```: Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "\n", + " - Required parameters: \n", + " - ```\"attachments\": True```\n", + " - ```\"data_row_details\": True```\n", + " - ```\"project_details\": True```\n", + " - ```\"label_details\": True```\n", + " - ```\"performance_details\": True```\n", + " - Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "xOhfu5ORJ6Qm", + "metadata": { + "id": "xOhfu5ORJ6Qm" + }, + "outputs": [], + "source": [ + "VIDEO_PROJECT_ID = \"\"\n", + "project = client.get_project(VIDEO_PROJECT_ID)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "hNECFiPIJ6Qm", + "metadata": { + "id": "hNECFiPIJ6Qm" + }, + "outputs": [], + "source": [ + "export_params = {\n", + " \"attachments\": True,\n", + " \"metadata_fields\": True,\n", + " \"data_row_details\": True,\n", + " \"project_details\": True,\n", + " \"performance_details\": True,\n", + " \"label_details\": True,\n", + " \"interpolated_frames\": True # For additional information on interpolated frames please visit our documentation https://docs.labelbox.com/docs/video-annotations#video-editor-components\n", + "}\n", + "filters = {}\n", + "\n", + "# A task is returned, this provides additional information about the status of your task, such as\n", + "# any errors encountered\n", + "export_task = project.export(params=export_params, filters=filters)\n", + "export_task.wait_till_done()" + ] + }, + { + "cell_type": "markdown", + "id": "wyR5dBD5J6Qm", + "metadata": { + "id": "wyR5dBD5J6Qm" + }, + "source": [ + "Fetch frame specific objects and frame or global classifications" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "NARj5Qe5J6Qm", + "metadata": { + "id": "NARj5Qe5J6Qm" + }, + "outputs": [], + "source": [ + "import pprint as pp # Assuming pp is imported from pprint module\n", + "\n", + "frames_objects_class_list = []\n", + "global_class_list = []\n", + "\n", + "stream = export_task.get_buffered_stream()\n", + "for output in stream:\n", + " output_json = output.json\n", + " for dr in output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]:\n", + " frames_data = dr[\"annotations\"][\"frames\"]\n", + " for k, v in frames_data.items():\n", + " frames_objects_class_list.append({k: v})\n", + " global_class_list.extend(dr[\"annotations\"][\"classifications\"])\n", + "\n", + " print(\"------- Frame specific classifications and objects -------\")\n", + " pp.pprint(frames_objects_class_list)\n", + "\n", + " print(\"------ Global classifications -------\")\n", + " pp.pprint(global_class_list)\n", + "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "mtrugG1jJ6Qm", + "metadata": { + "id": "mtrugG1jJ6Qm" + }, + "source": [ + "Fetch key frame feature map" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0txVTPYhJ6Qm", + "metadata": { + "id": "0txVTPYhJ6Qm" + }, + "outputs": [], + "source": [ + "keyframe_map = []\n", + "\n", + "stream = export_task.get_buffered_stream()\n", + "for output in stream:\n", + " output_json = output.json\n", + " labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n", + " for label in labels:\n", + " annotations = label[\"annotations\"][\"key_frame_feature_map\"]\n", + " for key, value in annotations.items():\n", + " keyframe_map.append({key: value})\n", + "\n", + "print(\"----- Keyframe Feature Map -----\")\n", + "pp.pprint(keyframe_map)" + ] + }, + { + "cell_type": "markdown", + "id": "BVdBnS0OJ6Qm", + "metadata": { + "id": "BVdBnS0OJ6Qm" + }, + "source": [ + "Fetch segments" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "w6Ktbq86J6Qm", + "metadata": { + "id": "w6Ktbq86J6Qm" + }, + "outputs": [], + "source": [ + "segments_map = []\n", + "stream = export_task.get_buffered_stream()\n", + "for output in stream:\n", + " output_json = output.json\n", + " labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n", + " for label in labels:\n", + " annotations = label[\"annotations\"][\"segments\"]\n", + " for key, value in annotations.items():\n", + " segments_map.append({key: value})\n", + "\n", + "print(\"----- Segments Feature Map -----\")\n", + "pp.pprint(segments_map)" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From dde4fccc9d8cbbe7992f34bb9dc36c550361c1db Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 18 Jul 2024 18:52:28 +0000 Subject: [PATCH 2/5] :art: Cleaned --- .../export_v1_to_v2_migration_support.ipynb | 1015 ++++------------- 1 file changed, 193 insertions(+), 822 deletions(-) diff --git a/examples/exports/export_v1_to_v2_migration_support.ipynb b/examples/exports/export_v1_to_v2_migration_support.ipynb index c78a62f02..7d324f7b5 100644 --- a/examples/exports/export_v1_to_v2_migration_support.ipynb +++ b/examples/exports/export_v1_to_v2_migration_support.ipynb @@ -1,80 +1,61 @@ { + "nbformat": 4, + "nbformat_minor": 5, + "metadata": {}, "cells": [ { - "cell_type": "markdown", - "id": "4E7YgJ6CJ6Qc", - "metadata": { - "id": "4E7YgJ6CJ6Qc" - }, + "metadata": {}, "source": [ - "\n", - " \n", - "" - ] + "", + " ", + "\n" + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "MrE_dQKBJ6Qe", - "metadata": { - "id": "MrE_dQKBJ6Qe" - }, + "metadata": {}, "source": [ - "\n", "\n", - "\n", - "\n", "\n", "\n", "\n", - "\n", - "\n", "" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "nxOw67b-J6Qe", - "metadata": { - "id": "nxOw67b-J6Qe" - }, + "metadata": {}, "source": [ "# Export V1 to V2 migration" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "qCY12qC8J6Qf", - "metadata": { - "id": "qCY12qC8J6Qf" - }, + "metadata": {}, "source": [ "**Export V1 will no longer be available in any version of the SDK starting on April 2024**. We recommend users to plan accordingly.\n", "\n", "This notebook is designed to help users identify alternative V2 export methods that can serve as replacements for V1 methods." - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "5IBNbHJAJ6Qf", - "metadata": { - "id": "5IBNbHJAJ6Qf" - }, + "metadata": {}, "source": [ "### Key changes included in export V2 methods ( ``export()`` and ``export_v2()``):\n", "1. Added flexibility to only export the data that is needed. The new methods include parameters and filters to give you more granular control over your exports.\n", "2. Added functionality to stream an **unlimited** number of data rows using ``export()`` (available on SDK >=3.56). Upgrading to `export()` is recommended as it is a more scalable solution.\n", "\n", "For complete details on how to use export V2 methods please see the [Export V2 methods](https://docs.labelbox.com/reference/label-export#export-v2-methods) documentation." - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "L2Xb8V5PJ6Qf", - "metadata": { - "id": "L2Xb8V5PJ6Qf" - }, + "metadata": {}, "source": [ "### Export V1 deprecated methods:\n", "Project methods :\n", @@ -91,108 +72,69 @@ "Model methods :\n", "1. ```model_run.export_labels()```\n", "\n" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "ZEzKj4TwJ6Qf", - "metadata": { - "id": "ZEzKj4TwJ6Qf" - }, + "metadata": {}, "source": [ "# Imports" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "%pip install -q \"labelbox[data]\"", "cell_type": "code", - "execution_count": null, - "id": "8egYmUB1J6Qf", - "metadata": { - "id": "8egYmUB1J6Qf" - }, "outputs": [], - "source": [ - "!pip install -q \"labelbox[data]\"" - ] + "execution_count": null }, { + "metadata": {}, + "source": "import labelbox as lb\nimport pprint\n\npp = pprint.PrettyPrinter(width=30, compact=True)", "cell_type": "code", - "execution_count": 1, - "id": "S8YPoX2zJ6Qg", - "metadata": { - "id": "S8YPoX2zJ6Qg" - }, "outputs": [], - "source": [ - "import labelbox as lb\n", - "import pprint\n", - "pp = pprint.PrettyPrinter(width=30, compact=True)\n" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "ZMLM7LejJ6Qg", - "metadata": { - "id": "ZMLM7LejJ6Qg" - }, + "metadata": {}, "source": [ "## API Key and Client\n", "See the developer guide for [creating an API key](https://docs.labelbox.com/reference/create-api-key)." - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "API_KEY = \"\"\nclient = lb.Client(api_key=API_KEY)", "cell_type": "code", - "execution_count": null, - "id": "sZk3-cRqJ6Qg", - "metadata": { - "id": "sZk3-cRqJ6Qg" - }, "outputs": [], - "source": [ - "API_KEY = \"\"\n", - "client = lb.Client(api_key=API_KEY)" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "aCvPcEHIJ6Qg", - "metadata": { - "id": "aCvPcEHIJ6Qg" - }, + "metadata": {}, "source": [ "# Exports V1 to V2 guidelines\n", "The follow sections will demonstrate how to use the export V2 methods to fetch data from your projects, datasets, batches and model runs." - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "S5g48GJPJ6Qg", - "metadata": { - "id": "S5g48GJPJ6Qg" - }, + "metadata": {}, "source": [ "## Export labels from a project\n" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "PROJECT_ID = \"\"\nproject = client.get_project(PROJECT_ID)", "cell_type": "code", - "execution_count": null, - "id": "WDlK0c-rJ6Qg", - "metadata": { - "id": "WDlK0c-rJ6Qg" - }, "outputs": [], - "source": [ - "PROJECT_ID = \"\"\n", - "project = client.get_project(PROJECT_ID)" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "-z3K7jDQJ6Qg", - "metadata": { - "id": "-z3K7jDQJ6Qg" - }, + "metadata": {}, "source": [ "##### Export V1 (deprecated) \n", "1. ```project.export_labels()```\n", @@ -200,104 +142,35 @@ " - ```download: bool = False```\n", " - ```timeout_seconds: int = 1800```\n", " - Output : (str | List[Dict[Any, Any]] | None)" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "mJqPO1GGJ6Qg", - "metadata": { - "id": "mJqPO1GGJ6Qg" - }, + "metadata": {}, "source": [ "For a comprehensive example of Export V1 ``project.export_labels()`` output, please refer to our documentation: [Export V1 sample export](https://docs.labelbox.com/reference/export-image-annotations#sample-project-export)" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "8UapSOOnJ6Qh", - "metadata": { - "id": "8UapSOOnJ6Qh" - }, + "metadata": {}, "source": [ "2. ```project.label_generator()```\n", " - Parameters:\n", " - ```timeout_seconds: int = 600```\n", " - Output: LabelGenerator" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "# Single entry from the output of project.label_generator() (deprecated)\n# Label objects will not be deprecated.\nsingle_output_from_generator = \"\"\"\n\nLabel(\n uid='clrf5csho2ihx07ilffgp2fzj',\n data=ImageData(\n im_bytes=None,\n file_path=None,\n url='https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg',\n arr=None\n ),\n annotations=[\n ObjectAnnotation(\n confidence=None,\n name='bounding_box',\n feature_schema_id='clrf5ck4a0b9b071paa9ncu15',\n extra={\n 'instanceURI': 'https://api.labelbox.com/masks/feature/clrf5csvi6ofm07lsf9pygwvi?token='\n 'color': '#ff0000',\n 'feature_id': 'clrf5csvi6ofm07lsf9pygwvi',\n 'value': 'bounding_box',\n 'page': None,\n 'unit': None\n },\n value=Rectangle(\n extra={},\n start=Point(extra={}, x=2096.0, y=1264.0),\n end=Point(extra={}, x=2240.0, y=1689.0)\n ),\n classifications=[]\n ),\n # Add more annotations as needed\n # ...\n ],\n extra={\n 'Created By': 'aovalle@labelbox.com',\n 'Project Name': 'Image Annotation Import Demo',\n 'Created At': '2024-01-15T16:35:59.000Z',\n 'Updated At': '2024-01-15T16:51:56.000Z',\n 'Seconds to Label': 66.0,\n 'Agreement': -1.0,\n 'Benchmark Agreement': -1.0,\n 'Benchmark ID': None,\n 'Dataset Name': 'image-demo-dataset',\n 'Reviews': [],\n 'View Label': 'https://editor.labelbox.com?project=clrf5ckex09m9070x1te223u5&label=clrf5csho2ihx07ilffgp2fzj',\n 'Has Open Issues': 0.0,\n 'Skipped': False,\n 'media_type': 'image',\n 'Data Split': None,\n 'Global Key': '2560px-Kitano_Street_Kobe01s5s41102.jpeg'\n }\n)\n\n\"\"\"", "cell_type": "code", - "execution_count": null, - "id": "5ALbzsbEJ6Qh", - "metadata": { - "id": "5ALbzsbEJ6Qh" - }, "outputs": [], - "source": [ - "# Single entry from the output of project.label_generator() (deprecated)\n", - "# Label objects will not be deprecated.\n", - "single_output_from_generator = \"\"\"\n", - "\n", - "Label(\n", - " uid='clrf5csho2ihx07ilffgp2fzj',\n", - " data=ImageData(\n", - " im_bytes=None,\n", - " file_path=None,\n", - " url='https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg',\n", - " arr=None\n", - " ),\n", - " annotations=[\n", - " ObjectAnnotation(\n", - " confidence=None,\n", - " name='bounding_box',\n", - " feature_schema_id='clrf5ck4a0b9b071paa9ncu15',\n", - " extra={\n", - " 'instanceURI': 'https://api.labelbox.com/masks/feature/clrf5csvi6ofm07lsf9pygwvi?token='\n", - " 'color': '#ff0000',\n", - " 'feature_id': 'clrf5csvi6ofm07lsf9pygwvi',\n", - " 'value': 'bounding_box',\n", - " 'page': None,\n", - " 'unit': None\n", - " },\n", - " value=Rectangle(\n", - " extra={},\n", - " start=Point(extra={}, x=2096.0, y=1264.0),\n", - " end=Point(extra={}, x=2240.0, y=1689.0)\n", - " ),\n", - " classifications=[]\n", - " ),\n", - " # Add more annotations as needed\n", - " # ...\n", - " ],\n", - " extra={\n", - " 'Created By': 'aovalle@labelbox.com',\n", - " 'Project Name': 'Image Annotation Import Demo',\n", - " 'Created At': '2024-01-15T16:35:59.000Z',\n", - " 'Updated At': '2024-01-15T16:51:56.000Z',\n", - " 'Seconds to Label': 66.0,\n", - " 'Agreement': -1.0,\n", - " 'Benchmark Agreement': -1.0,\n", - " 'Benchmark ID': None,\n", - " 'Dataset Name': 'image-demo-dataset',\n", - " 'Reviews': [],\n", - " 'View Label': 'https://editor.labelbox.com?project=clrf5ckex09m9070x1te223u5&label=clrf5csho2ihx07ilffgp2fzj',\n", - " 'Has Open Issues': 0.0,\n", - " 'Skipped': False,\n", - " 'media_type': 'image',\n", - " 'Data Split': None,\n", - " 'Global Key': '2560px-Kitano_Street_Kobe01s5s41102.jpeg'\n", - " }\n", - ")\n", - "\n", - "\"\"\"" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "7WJT6y4xJ6Qh", - "metadata": { - "id": "7WJT6y4xJ6Qh" - }, + "metadata": {}, "source": [ "##### Export V2\n", "\n", @@ -317,79 +190,32 @@ " - `ExportTask.has_result()` return type: bool\n", " - `ExportTask.has_errors()` return type: bool\n", " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "## Set the export parameters to only export labels\nexport_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"label_details\": True,\n \"performance_details\": True,\n}\n# You also have the option to include additional filtering to narrow down the list of labels\nfilters = {}\n\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", "cell_type": "code", - "execution_count": null, - "id": "ouzz6t5PJ6Qh", - "metadata": { - "id": "ouzz6t5PJ6Qh" - }, "outputs": [], - "source": [ - "## Set the export parameters to only export labels\n", - "export_params= {\n", - " \"attachments\": True,\n", - " \"metadata_fields\": True,\n", - " \"data_row_details\": True,\n", - " \"project_details\": True,\n", - " \"label_details\": True,\n", - " \"performance_details\": True\n", - "}\n", - "# You also have the option to include additional filtering to narrow down the list of labels\n", - "filters = {}\n", - "\n", - "export_task = project.export(params=export_params, filters=filters)\n", - "export_task.wait_till_done()" - ] + "execution_count": null }, { + "metadata": {}, + "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", "cell_type": "code", - "execution_count": null, - "id": "JpGRBQPpJ6Qh", - "metadata": { - "id": "JpGRBQPpJ6Qh" - }, "outputs": [], - "source": [ - "# Provide results with JSON converter\n", - "# Returns streamed JSON output strings from export task results/errors, one by one\n", - "\n", - "# Callback used for JSON Converter\n", - "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", - " print(output.json)\n", - "\n", - "if export_task.has_errors():\n", - " export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.ERRORS\n", - " ).start(stream_handler=lambda error: print(error))\n", - "\n", - "if export_task.has_result():\n", - " export_json = export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.RESULT\n", - " ).start(stream_handler=json_stream_handler)\n", - "\n", - "print(\"file size: \", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))\n", - "print(\"line count: \", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))\n" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "QZxfvXBHJ6Qh", - "metadata": { - "id": "QZxfvXBHJ6Qh" - }, + "metadata": {}, "source": [ "## Export queued (\"To Label\") data rows from a project" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "j3n3kPgrJ6Qh", - "metadata": { - "id": "j3n3kPgrJ6Qh" - }, + "metadata": {}, "source": [ "##### Export V1 (deprecated): \n", "1. ``project.export_queued_data_rows()`` :\n", @@ -397,46 +223,18 @@ " - ``timeout_seconds: int = 120``\n", " - ``include_metadata: bool = False``\n", " - Output: List[Dict[str, str]]" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "# Single entry from the output of project.export_queued_data_rows() (deprecated)\nsingle_output_example = \"\"\"\n[\n {'id': 'clpouak6nap2g0783ajd1d6pf',\n 'createdAt': '2023-12-03T02:04:34.062Z',\n 'updatedAt': '2023-12-03T02:05:33.797Z',\n 'externalId': None,\n 'globalKey': 'b57c9ab2-304f-4c17-ba5f-c536f39a6a46',\n 'metadataFields': [],\n 'customMetadata': [],\n 'rowData': 'https://storage.googleapis.com/labelbox-developer-testing-assets/image/data_files/santa.jpeg',\n 'mediaAttributes': {'assetType': 'image',\n 'contentLength': 305973,\n 'height': 1333,\n 'mimeType': 'image/jpeg',\n 'subType': 'jpeg',\n 'superType': 'image',\n 'width': 2000}}\n]\n\n\"\"\"", "cell_type": "code", - "execution_count": null, - "id": "puNxgxlUJ6Qh", - "metadata": { - "id": "puNxgxlUJ6Qh" - }, "outputs": [], - "source": [ - "# Single entry from the output of project.export_queued_data_rows() (deprecated)\n", - "single_output_example = \"\"\"\n", - "[\n", - " {'id': 'clpouak6nap2g0783ajd1d6pf',\n", - " 'createdAt': '2023-12-03T02:04:34.062Z',\n", - " 'updatedAt': '2023-12-03T02:05:33.797Z',\n", - " 'externalId': None,\n", - " 'globalKey': 'b57c9ab2-304f-4c17-ba5f-c536f39a6a46',\n", - " 'metadataFields': [],\n", - " 'customMetadata': [],\n", - " 'rowData': 'https://storage.googleapis.com/labelbox-developer-testing-assets/image/data_files/santa.jpeg',\n", - " 'mediaAttributes': {'assetType': 'image',\n", - " 'contentLength': 305973,\n", - " 'height': 1333,\n", - " 'mimeType': 'image/jpeg',\n", - " 'subType': 'jpeg',\n", - " 'superType': 'image',\n", - " 'width': 2000}}\n", - "]\n", - "\n", - "\"\"\"" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "vFqPfKcSJ6Qh", - "metadata": { - "id": "vFqPfKcSJ6Qh" - }, + "metadata": {}, "source": [ "##### Export V2\n", "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", @@ -453,95 +251,39 @@ " - `ExportTask.has_result()` return type: bool\n", " - `ExportTask.has_errors()` return type: bool\n", " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "export_params = {\n \"attachments\": True, # Set to true if you want to export attachments\n \"metadata_fields\": True, # Set to true if you want to export metadata\n \"data_row_details\": True,\n \"project_details\": True,\n}\nfilters = {\n \"workflow_status\":\n \"ToLabel\" ## Using this filter will only export queued data rows\n}\n\n# An ExportTask is returned, this provides additional information about the status of your task, such as\n# any errors encountered and includes additional methods to stream your data\n\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", "cell_type": "code", - "execution_count": null, - "id": "HSTLb8bXJ6Qh", - "metadata": { - "id": "HSTLb8bXJ6Qh" - }, "outputs": [], - "source": [ - "export_params= {\n", - " \"attachments\": True, # Set to true if you want to export attachments\n", - " \"metadata_fields\": True, # Set to true if you want to export metadata\n", - " \"data_row_details\": True,\n", - " \"project_details\": True\n", - "}\n", - "filters = {\n", - " \"workflow_status\": \"ToLabel\" ## Using this filter will only export queued data rows\n", - "}\n", - "\n", - "# An ExportTask is returned, this provides additional information about the status of your task, such as\n", - "# any errors encountered and includes additional methods to stream your data\n", - "\n", - "\n", - "export_task = project.export(params=export_params, filters=filters)\n", - "export_task.wait_till_done()\n" - ] + "execution_count": null }, { + "metadata": {}, + "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", "cell_type": "code", - "execution_count": null, - "id": "8sxoNt6MJ6Qh", - "metadata": { - "id": "8sxoNt6MJ6Qh" - }, "outputs": [], - "source": [ - "# Provide results with JSON converter\n", - "# Returns streamed JSON output strings from export task results/errors, one by one\n", - "\n", - "# Callback used for JSON Converter\n", - "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", - " print(output.json)\n", - "\n", - "\n", - "if export_task.has_errors():\n", - " export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.ERRORS\n", - " ).start(stream_handler=lambda error: print(error))\n", - "\n", - "if export_task.has_result():\n", - " export_json = export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.RESULT\n", - " ).start(stream_handler=json_stream_handler)\n", - "\n", - "print(\"file size: \", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))\n", - "print(\"line count: \", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "cCnY3j3NJ6Qh", - "metadata": { - "id": "cCnY3j3NJ6Qh" - }, + "metadata": {}, "source": [ "## Export data rows from a Dataset" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "DATASET_ID = \"\"\ndataset = client.get_dataset(DATASET_ID)", "cell_type": "code", - "execution_count": null, - "id": "3VGQZaHPJ6Qh", - "metadata": { - "id": "3VGQZaHPJ6Qh" - }, "outputs": [], - "source": [ - "DATASET_ID = \"\"\n", - "dataset = client.get_dataset(DATASET_ID)" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "Ws7WCa0eJ6Qh", - "metadata": { - "id": "Ws7WCa0eJ6Qh" - }, + "metadata": {}, "source": [ "#### Export V1 (deprecated):\n", "\n", @@ -551,49 +293,18 @@ " - ``include_metadata: bool = True``\n", " - Output:\n", " - Data row object generator\n" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "# Single entry from the output of dataset.export_data_rows() (deprecated)\n# Data row objects will not be deprecated.\n\nsingle_output_from_data_row_generator = \"\"\"\n\n\"\"\"", "cell_type": "code", - "execution_count": null, - "id": "c_93zHuKJ6Qh", - "metadata": { - "id": "c_93zHuKJ6Qh" - }, "outputs": [], - "source": [ - "# Single entry from the output of dataset.export_data_rows() (deprecated)\n", - "# Data row objects will not be deprecated.\n", - "\n", - "single_output_from_data_row_generator = \"\"\"\n", - "\n", - "\"\"\"" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "i82Jg4_bJ6Qh", - "metadata": { - "id": "i82Jg4_bJ6Qh" - }, + "metadata": {}, "source": [ "#### Export V2\n", "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", @@ -607,77 +318,32 @@ " - `ExportTask.has_result()` return type: bool\n", " - `ExportTask.has_errors()` return type: bool\n", " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "export_params = {\n \"attachments\": True, # Set to true if you want to export attachments\n \"metadata_fields\": True, # Set to true if you want to export metadata\n \"data_row_details\": True,\n}\nfilters = {}\n\n# A task is returned, this provides additional information about the status of your task, such as\n# any errors encountered\nexport_task = dataset.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", "cell_type": "code", - "execution_count": null, - "id": "vlkTDFDPJ6Qi", - "metadata": { - "id": "vlkTDFDPJ6Qi" - }, "outputs": [], - "source": [ - "export_params = {\n", - " \"attachments\": True, # Set to true if you want to export attachments\n", - " \"metadata_fields\": True, # Set to true if you want to export metadata\n", - " \"data_row_details\": True,\n", - "}\n", - "filters = {}\n", - "\n", - "# A task is returned, this provides additional information about the status of your task, such as\n", - "# any errors encountered\n", - "export_task = dataset.export(params=export_params, filters=filters)\n", - "export_task.wait_till_done()" - ] + "execution_count": null }, { + "metadata": {}, + "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", "cell_type": "code", - "execution_count": null, - "id": "k0JIGqNlJ6Qi", - "metadata": { - "id": "k0JIGqNlJ6Qi" - }, "outputs": [], - "source": [ - "# Provide results with JSON converter\n", - "# Returns streamed JSON output strings from export task results/errors, one by one\n", - "\n", - "# Callback used for JSON Converter\n", - "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", - " print(output.json)\n", - "\n", - "\n", - "if export_task.has_errors():\n", - " export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.ERRORS\n", - " ).start(stream_handler=lambda error: print(error))\n", - "\n", - "if export_task.has_result():\n", - " export_json = export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.RESULT\n", - " ).start(stream_handler=json_stream_handler)\n", - "\n", - "print(\"file size: \", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))\n", - "print(\"line count: \", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "LPv1lghqJ6Qi", - "metadata": { - "id": "LPv1lghqJ6Qi" - }, + "metadata": {}, "source": [ "## Export data rows from a batch" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "wGO_FSlpJ6Qi", - "metadata": { - "id": "wGO_FSlpJ6Qi" - }, + "metadata": {}, "source": [ "#### Export V1 (deprecated):\n", "1. ```batch.export_data_rows()```\n", @@ -686,49 +352,18 @@ " - ``include_metadata: bool = True``\n", " - Output:\n", " - Data row object generator" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "# Single output from batch.export_data_rows() method (deprecated)\n# Data row objects will not be deprecated\n\nsingle_output_from_data_row_generator = \"\"\"\n\n\"\"\"", "cell_type": "code", - "execution_count": null, - "id": "TvjQgBjYJ6Qi", - "metadata": { - "id": "TvjQgBjYJ6Qi" - }, "outputs": [], - "source": [ - "# Single output from batch.export_data_rows() method (deprecated)\n", - "# Data row objects will not be deprecated\n", - "\n", - "single_output_from_data_row_generator = \"\"\"\n", - "\n", - "\"\"\"" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "IciDQuDxJ6Qi", - "metadata": { - "id": "IciDQuDxJ6Qi" - }, + "metadata": {}, "source": [ "#### Export V2\n", "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", @@ -743,94 +378,39 @@ " - `ExportTask.has_result()` return type: bool\n", " - `ExportTask.has_errors()` return type: bool\n", " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "# Find the batch ID by navigating to \"Batches\" --> \"Manage batches\" --> \"Copy Batch ID\"\nBATCH_ID = \"\"", "cell_type": "code", - "execution_count": null, - "id": "ZgjgVg34J6Qi", - "metadata": { - "id": "ZgjgVg34J6Qi" - }, "outputs": [], - "source": [ - "# Find the batch ID by navigating to \"Batches\" --> \"Manage batches\" --> \"Copy Batch ID\"\n", - "BATCH_ID = \"\"" - ] + "execution_count": null }, { + "metadata": {}, + "source": "export_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"performance_details\": True,\n \"batch_ids\": [\n BATCH_ID\n ], # Include batch ids if you only want to export specific batches, otherwise,\n # you can export all the data without using this parameter\n}\nfilters = {}\n\n# A task is returned, this provides additional information about the status of your task, such as\n# any errors encountered\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", "cell_type": "code", - "execution_count": null, - "id": "AGTbvKw7J6Ql", - "metadata": { - "id": "AGTbvKw7J6Ql" - }, "outputs": [], - "source": [ - "export_params = {\n", - " \"attachments\": True,\n", - " \"metadata_fields\": True,\n", - " \"data_row_details\": True,\n", - " \"project_details\": True,\n", - " \"performance_details\": True,\n", - " \"batch_ids\" : [BATCH_ID] # Include batch ids if you only want to export specific batches, otherwise,\n", - " #you can export all the data without using this parameter\n", - "}\n", - "filters = {}\n", - "\n", - "# A task is returned, this provides additional information about the status of your task, such as\n", - "# any errors encountered\n", - "export_task = project.export(params=export_params, filters=filters)\n", - "export_task.wait_till_done()" - ] + "execution_count": null }, { + "metadata": {}, + "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", "cell_type": "code", - "execution_count": null, - "id": "N5ysx8FyJ6Ql", - "metadata": { - "id": "N5ysx8FyJ6Ql" - }, "outputs": [], - "source": [ - "# Provide results with JSON converter\n", - "# Returns streamed JSON output strings from export task results/errors, one by one\n", - "\n", - "# Callback used for JSON Converter\n", - "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", - " print(output.json)\n", - "\n", - "\n", - "if export_task.has_errors():\n", - " export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.ERRORS\n", - " ).start(stream_handler=lambda error: print(error))\n", - "\n", - "if export_task.has_result():\n", - " export_json = export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.RESULT\n", - " ).start(stream_handler=json_stream_handler)\n", - "\n", - "print(\"file size: \", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))\n", - "print(\"line count: \", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "-dTKlTzmJ6Ql", - "metadata": { - "id": "-dTKlTzmJ6Ql" - }, + "metadata": {}, "source": [ "## Export data rows from a Model" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "ERTdBvxaJ6Ql", - "metadata": { - "id": "ERTdBvxaJ6Ql" - }, + "metadata": {}, "source": [ "#### Export V1 (deprecated):\n", "1. ```model_run.export_labels(downlaod=True)```\n", @@ -838,54 +418,18 @@ " - ```download: bool = False```\n", " - ```timeout_seconds: int = 1800```\n", " - Output : (str | List[Dict[Any, Any]] | None)" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "# Single output from model_run.export_labels()\nsingle_output_example = \"\"\"\n[\n {'ID': '1c48a7a0-3016-48e0-b0e3-47430f974869',\n 'Data Split': 'training',\n 'DataRow ID': 'clpqdyf650xd40712pycshy6a',\n 'External ID': './resume/BANKING/99124477.pdf',\n 'Labeled Data': 'https://storage.labelbox.com/cl5bn8qvq1av907xtb3bp8q60%2F8c6afc38-42a4-b2e1-a2e3-1e3b0c2998fc-99124477.pdf?Expires=1706637969726&KeyName=labelbox-assets-key-3&Signature=2nVt3sJ21CbjGS9I64yFquUELRw',\n 'Media Attributes': {'assetType': 'pdf',\n 'contentLength': 42535,\n 'mimeType': 'application/pdf',\n 'pageCount': 3,\n 'subType': 'pdf',\n 'superType': 'application'},\n 'Label': {'objects': [{'featureId': 'b9f3b584-0f45-050a-88d4-39c2a169c8e1',\n 'schemaId': 'clq1ckwbd08jp07z91q9mch5j',\n 'title': 'Test',\n 'value': 'test',\n 'color': '#1CE6FF',\n 'data': {'location': [{'text-bbox': {'page': 1,\n 'top': 158.44,\n 'left': 58.765,\n 'height': 13.691,\n 'width': 78.261}}],\n 'unit': 'POINTS'}}],\n 'classifications': [],\n 'relationships': []}}\n ]\n \"\"\"", "cell_type": "code", - "execution_count": null, - "id": "ynE1bihvJ6Ql", - "metadata": { - "id": "ynE1bihvJ6Ql" - }, "outputs": [], - "source": [ - "# Single output from model_run.export_labels()\n", - "single_output_example = \"\"\"\n", - "[\n", - " {'ID': '1c48a7a0-3016-48e0-b0e3-47430f974869',\n", - " 'Data Split': 'training',\n", - " 'DataRow ID': 'clpqdyf650xd40712pycshy6a',\n", - " 'External ID': './resume/BANKING/99124477.pdf',\n", - " 'Labeled Data': 'https://storage.labelbox.com/cl5bn8qvq1av907xtb3bp8q60%2F8c6afc38-42a4-b2e1-a2e3-1e3b0c2998fc-99124477.pdf?Expires=1706637969726&KeyName=labelbox-assets-key-3&Signature=2nVt3sJ21CbjGS9I64yFquUELRw',\n", - " 'Media Attributes': {'assetType': 'pdf',\n", - " 'contentLength': 42535,\n", - " 'mimeType': 'application/pdf',\n", - " 'pageCount': 3,\n", - " 'subType': 'pdf',\n", - " 'superType': 'application'},\n", - " 'Label': {'objects': [{'featureId': 'b9f3b584-0f45-050a-88d4-39c2a169c8e1',\n", - " 'schemaId': 'clq1ckwbd08jp07z91q9mch5j',\n", - " 'title': 'Test',\n", - " 'value': 'test',\n", - " 'color': '#1CE6FF',\n", - " 'data': {'location': [{'text-bbox': {'page': 1,\n", - " 'top': 158.44,\n", - " 'left': 58.765,\n", - " 'height': 13.691,\n", - " 'width': 78.261}}],\n", - " 'unit': 'POINTS'}}],\n", - " 'classifications': [],\n", - " 'relationships': []}}\n", - " ]\n", - " \"\"\"\n" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "Mtr8gaTtJ6Ql", - "metadata": { - "id": "Mtr8gaTtJ6Ql" - }, + "metadata": {}, "source": [ "#### Export V2\n", "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", @@ -903,89 +447,40 @@ " - `ExportTask.has_result()` return type: bool\n", " - `ExportTask.has_errors()` return type: bool\n", " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "MODEL_RUN_ID = \"\"\nmodel_run = client.get_model_run(MODEL_RUN_ID)", "cell_type": "code", - "execution_count": null, - "id": "iJA1W1NsJ6Ql", - "metadata": { - "id": "iJA1W1NsJ6Ql" - }, "outputs": [], - "source": [ - "MODEL_RUN_ID = \"\"\n", - "model_run = client.get_model_run(MODEL_RUN_ID)" - ] + "execution_count": null }, { + "metadata": {}, + "source": "export_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"performance_details\": True,\n}\n\nexport_task = model_run.export(params=export_params)\nexport_task.wait_till_done()", "cell_type": "code", - "execution_count": null, - "id": "wPug5qzJJ6Qm", - "metadata": { - "id": "wPug5qzJJ6Qm" - }, "outputs": [], - "source": [ - "export_params = {\n", - " \"attachments\": True,\n", - " \"metadata_fields\": True,\n", - " \"data_row_details\": True,\n", - " \"project_details\": True,\n", - " \"performance_details\": True\n", - "}\n", - "\n", - "export_task = model_run.export(params=export_params)\n", - "export_task.wait_till_done()\n" - ] + "execution_count": null }, { + "metadata": {}, + "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", "cell_type": "code", - "execution_count": null, - "id": "ap9fPl6KJ6Qm", - "metadata": { - "id": "ap9fPl6KJ6Qm" - }, "outputs": [], - "source": [ - "# Provide results with JSON converter\n", - "# Returns streamed JSON output strings from export task results/errors, one by one\n", - "\n", - "# Callback used for JSON Converter\n", - "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", - " print(output.json)\n", - "\n", - "if export_task.has_errors():\n", - " export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.ERRORS\n", - " ).start(stream_handler=lambda error: print(error))\n", - "\n", - "if export_task.has_result():\n", - " export_json = export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.RESULT\n", - " ).start(stream_handler=json_stream_handler)\n", - "\n", - "print(\"file size: \", export_task.get_total_file_size(stream_type=lb.StreamType.RESULT))\n", - "print(\"line count: \", export_task.get_total_lines(stream_type=lb.StreamType.RESULT))" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "aEgufP_LJ6Qm", - "metadata": { - "id": "aEgufP_LJ6Qm" - }, + "metadata": {}, "source": [ "## Export data rows from a video project\n", "Video projects include additional fields. Please refer to the example below to extract specific fields from video exports.\n" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "XuiEWCkqJ6Qm", - "metadata": { - "id": "XuiEWCkqJ6Qm" - }, + "metadata": {}, "source": [ "##### Export V1 (deprecated) \n", "1. ```project.export_labels()```\n", @@ -993,37 +488,28 @@ " - ```download: bool = False```\n", " - ```timeout_seconds: int = 1800```\n", " - Output : (str | List[Dict[Any, Any]] | None)" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "3riNZm2jJ6Qm", - "metadata": { - "id": "3riNZm2jJ6Qm" - }, + "metadata": {}, "source": [ "For a comprehensive example of Export V1 ``project.export_labels()`` output, please refer to our documentation: [Export V1 sample export](https://docs.labelbox.com/reference/export-image-annotations#sample-project-export)" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "ODXIhOQUJ6Qm", - "metadata": { - "id": "ODXIhOQUJ6Qm" - }, + "metadata": {}, "source": [ "2. ```project.video_label_generator()```\n", " - Parameters:\n", " - ```timeout_seconds: int = 600```\n", " - Output: LabelGenerator" - ] + ], + "cell_type": "markdown" }, { - "cell_type": "markdown", - "id": "BR0RIC-YJ6Qm", - "metadata": { - "id": "BR0RIC-YJ6Qm" - }, + "metadata": {}, "source": [ "##### Export V2\n", "\n", @@ -1042,179 +528,64 @@ " - `ExportTask.has_result()` return type: bool\n", " - `ExportTask.has_errors()` return type: bool\n", " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "VIDEO_PROJECT_ID = \"\"\nproject = client.get_project(VIDEO_PROJECT_ID)", "cell_type": "code", - "execution_count": null, - "id": "xOhfu5ORJ6Qm", - "metadata": { - "id": "xOhfu5ORJ6Qm" - }, "outputs": [], - "source": [ - "VIDEO_PROJECT_ID = \"\"\n", - "project = client.get_project(VIDEO_PROJECT_ID)" - ] + "execution_count": null }, { + "metadata": {}, + "source": "export_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"performance_details\": True,\n \"label_details\": True,\n \"interpolated_frames\":\n True, # For additional information on interpolated frames please visit our documentation https://docs.labelbox.com/docs/video-annotations#video-editor-components\n}\nfilters = {}\n\n# A task is returned, this provides additional information about the status of your task, such as\n# any errors encountered\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", "cell_type": "code", - "execution_count": null, - "id": "hNECFiPIJ6Qm", - "metadata": { - "id": "hNECFiPIJ6Qm" - }, "outputs": [], - "source": [ - "export_params = {\n", - " \"attachments\": True,\n", - " \"metadata_fields\": True,\n", - " \"data_row_details\": True,\n", - " \"project_details\": True,\n", - " \"performance_details\": True,\n", - " \"label_details\": True,\n", - " \"interpolated_frames\": True # For additional information on interpolated frames please visit our documentation https://docs.labelbox.com/docs/video-annotations#video-editor-components\n", - "}\n", - "filters = {}\n", - "\n", - "# A task is returned, this provides additional information about the status of your task, such as\n", - "# any errors encountered\n", - "export_task = project.export(params=export_params, filters=filters)\n", - "export_task.wait_till_done()" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "wyR5dBD5J6Qm", - "metadata": { - "id": "wyR5dBD5J6Qm" - }, + "metadata": {}, "source": [ "Fetch frame specific objects and frame or global classifications" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "import pprint as pp # Assuming pp is imported from pprint module\n\nframes_objects_class_list = []\nglobal_class_list = []\n\nstream = export_task.get_buffered_stream()\nfor output in stream:\n output_json = output.json\n for dr in output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]:\n frames_data = dr[\"annotations\"][\"frames\"]\n for k, v in frames_data.items():\n frames_objects_class_list.append({k: v})\n global_class_list.extend(dr[\"annotations\"][\"classifications\"])\n\n print(\"------- Frame specific classifications and objects -------\")\n pp.pprint(frames_objects_class_list)\n\n print(\"------ Global classifications -------\")\n pp.pprint(global_class_list)", "cell_type": "code", - "execution_count": null, - "id": "NARj5Qe5J6Qm", - "metadata": { - "id": "NARj5Qe5J6Qm" - }, "outputs": [], - "source": [ - "import pprint as pp # Assuming pp is imported from pprint module\n", - "\n", - "frames_objects_class_list = []\n", - "global_class_list = []\n", - "\n", - "stream = export_task.get_buffered_stream()\n", - "for output in stream:\n", - " output_json = output.json\n", - " for dr in output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]:\n", - " frames_data = dr[\"annotations\"][\"frames\"]\n", - " for k, v in frames_data.items():\n", - " frames_objects_class_list.append({k: v})\n", - " global_class_list.extend(dr[\"annotations\"][\"classifications\"])\n", - "\n", - " print(\"------- Frame specific classifications and objects -------\")\n", - " pp.pprint(frames_objects_class_list)\n", - "\n", - " print(\"------ Global classifications -------\")\n", - " pp.pprint(global_class_list)\n", - "\n", - "\n" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "mtrugG1jJ6Qm", - "metadata": { - "id": "mtrugG1jJ6Qm" - }, + "metadata": {}, "source": [ "Fetch key frame feature map" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "keyframe_map = []\n\nstream = export_task.get_buffered_stream()\nfor output in stream:\n output_json = output.json\n labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n for label in labels:\n annotations = label[\"annotations\"][\"key_frame_feature_map\"]\n for key, value in annotations.items():\n keyframe_map.append({key: value})\n\nprint(\"----- Keyframe Feature Map -----\")\npp.pprint(keyframe_map)", "cell_type": "code", - "execution_count": null, - "id": "0txVTPYhJ6Qm", - "metadata": { - "id": "0txVTPYhJ6Qm" - }, "outputs": [], - "source": [ - "keyframe_map = []\n", - "\n", - "stream = export_task.get_buffered_stream()\n", - "for output in stream:\n", - " output_json = output.json\n", - " labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n", - " for label in labels:\n", - " annotations = label[\"annotations\"][\"key_frame_feature_map\"]\n", - " for key, value in annotations.items():\n", - " keyframe_map.append({key: value})\n", - "\n", - "print(\"----- Keyframe Feature Map -----\")\n", - "pp.pprint(keyframe_map)" - ] + "execution_count": null }, { - "cell_type": "markdown", - "id": "BVdBnS0OJ6Qm", - "metadata": { - "id": "BVdBnS0OJ6Qm" - }, + "metadata": {}, "source": [ "Fetch segments" - ] + ], + "cell_type": "markdown" }, { + "metadata": {}, + "source": "segments_map = []\nstream = export_task.get_buffered_stream()\nfor output in stream:\n output_json = output.json\n labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n for label in labels:\n annotations = label[\"annotations\"][\"segments\"]\n for key, value in annotations.items():\n segments_map.append({key: value})\n\nprint(\"----- Segments Feature Map -----\")\npp.pprint(segments_map)", "cell_type": "code", - "execution_count": null, - "id": "w6Ktbq86J6Qm", - "metadata": { - "id": "w6Ktbq86J6Qm" - }, "outputs": [], - "source": [ - "segments_map = []\n", - "stream = export_task.get_buffered_stream()\n", - "for output in stream:\n", - " output_json = output.json\n", - " labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n", - " for label in labels:\n", - " annotations = label[\"annotations\"][\"segments\"]\n", - " for key, value in annotations.items():\n", - " segments_map.append({key: value})\n", - "\n", - "print(\"----- Segments Feature Map -----\")\n", - "pp.pprint(segments_map)" - ] + "execution_count": null } - ], - "metadata": { - "colab": { - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.13" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} + ] +} \ No newline at end of file From 2f1db0b710a5b4777ad2e52fe3b02cf591645ae1 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 18 Jul 2024 18:53:10 +0000 Subject: [PATCH 3/5] :memo: README updated --- examples/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/examples/README.md b/examples/README.md index 387840707..1523e1737 100644 --- a/examples/README.md +++ b/examples/README.md @@ -90,6 +90,11 @@ Open In Github Open In Colab + + Export V1 to V2 Migration Support + Open In Github + Open In Colab + From df3e74fa402a57cf63403c880eae17f0944f089b Mon Sep 17 00:00:00 2001 From: x-eun Date: Thu, 18 Jul 2024 15:43:36 -0700 Subject: [PATCH 4/5] Update export_v1_to_v2_migration_support.ipynb --- .../export_v1_to_v2_migration_support.ipynb | 1141 +++++++++++++---- 1 file changed, 877 insertions(+), 264 deletions(-) diff --git a/examples/exports/export_v1_to_v2_migration_support.ipynb b/examples/exports/export_v1_to_v2_migration_support.ipynb index 7d324f7b5..48d17f4d4 100644 --- a/examples/exports/export_v1_to_v2_migration_support.ipynb +++ b/examples/exports/export_v1_to_v2_migration_support.ipynb @@ -1,19 +1,26 @@ { "nbformat": 4, "nbformat_minor": 5, - "metadata": {}, + "metadata": { + "colab": { + "provenance": [] + } + }, "cells": [ { - "metadata": {}, + "metadata": { + "id": "ZvandVSm2lAW" + }, "source": [ - "", - " ", - "\n" + " \n" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "ZvandVSm2lAW" }, { - "metadata": {}, + "metadata": { + "id": "XvojFdVV2lAX" + }, "source": [ "\n", "\n", "" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "XvojFdVV2lAX" }, { - "metadata": {}, + "metadata": { + "id": "YUi47Rww2lAX" + }, "source": [ - "# Export V1 to V2 migration" + "# Export V1 migration" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "YUi47Rww2lAX" }, { - "metadata": {}, + "metadata": { + "id": "V1YDn3Gd2lAX" + }, "source": [ - "**Export V1 will no longer be available in any version of the SDK starting on April 2024**. We recommend users to plan accordingly.\n", - "\n", - "This notebook is designed to help users identify alternative V2 export methods that can serve as replacements for V1 methods." + "Export V1 is no longer available in any version of the SDK in favor of the latest `export()` method, which allows you to export data with granular control. This notebook provide guidelines and comparisons on migrating from Export V1 to `export()` to ensure successful data export." ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "V1YDn3Gd2lAX" }, { - "metadata": {}, + "metadata": { + "id": "gzHbqq8r2lAX" + }, "source": [ - "### Key changes included in export V2 methods ( ``export()`` and ``export_v2()``):\n", - "1. Added flexibility to only export the data that is needed. The new methods include parameters and filters to give you more granular control over your exports.\n", - "2. Added functionality to stream an **unlimited** number of data rows using ``export()`` (available on SDK >=3.56). Upgrading to `export()` is recommended as it is a more scalable solution.\n", + "### Key changes\n", + "The `export()` method adds the following changes and benefits compared to Export V1 methods:\n", + "1. Flexibility: `export()` provides parameters and filters to select and export data you need.\n", + "2. Simplicity: `export()` allows you to do all type of export operations using a single method.\n", + "3. Scalability: `export()` allows you to stream an **unlimited** number of data rows.\n", "\n", - "For complete details on how to use export V2 methods please see the [Export V2 methods](https://docs.labelbox.com/reference/label-export#export-v2-methods) documentation." + "For complete details on how to use `export()`, see the [Export overview](https://docs.labelbox.com/reference/export-overview)." ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "gzHbqq8r2lAX" }, { - "metadata": {}, + "metadata": { + "id": "Pxzeuk0t2lAX" + }, "source": [ "### Export V1 deprecated methods:\n", "Project methods :\n", @@ -73,68 +92,97 @@ "1. ```model_run.export_labels()```\n", "\n" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "Pxzeuk0t2lAX" }, { - "metadata": {}, + "metadata": { + "id": "MIMSF6cG2lAY" + }, "source": [ "# Imports" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "MIMSF6cG2lAY" }, { - "metadata": {}, - "source": "%pip install -q \"labelbox[data]\"", + "metadata": { + "id": "PDEKcCWJ2lAY" + }, + "source": [ + "%pip install -q \"labelbox[data]\"" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "PDEKcCWJ2lAY" }, { - "metadata": {}, - "source": "import labelbox as lb\nimport pprint\n\npp = pprint.PrettyPrinter(width=30, compact=True)", + "metadata": { + "id": "qdY6Lmtf2lAY" + }, + "source": [ + "import labelbox as lb\n", + "import pprint\n", + "\n", + "pp = pprint.PrettyPrinter(width=30, compact=True)" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "qdY6Lmtf2lAY" }, { - "metadata": {}, + "metadata": { + "id": "haHB7K0c2lAY" + }, "source": [ "## API Key and Client\n", "See the developer guide for [creating an API key](https://docs.labelbox.com/reference/create-api-key)." ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "haHB7K0c2lAY" }, { - "metadata": {}, - "source": "API_KEY = \"\"\nclient = lb.Client(api_key=API_KEY)", + "metadata": { + "id": "U3u1vj962lAY" + }, + "source": [ + "API_KEY = \"\"\n", + "client = lb.Client(api_key=API_KEY)" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "U3u1vj962lAY" }, { - "metadata": {}, + "metadata": { + "id": "YWMQlB3h2lAY" + }, "source": [ - "# Exports V1 to V2 guidelines\n", - "The follow sections will demonstrate how to use the export V2 methods to fetch data from your projects, datasets, batches and model runs." + "## Export labels from a project\n" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "YWMQlB3h2lAY" }, { - "metadata": {}, + "metadata": { + "id": "FPs4Vb7l2lAY" + }, "source": [ - "## Export labels from a project\n" + "PROJECT_ID = \"\"\n", + "project = client.get_project(PROJECT_ID)" ], - "cell_type": "markdown" - }, - { - "metadata": {}, - "source": "PROJECT_ID = \"\"\nproject = client.get_project(PROJECT_ID)", "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "FPs4Vb7l2lAY" }, { - "metadata": {}, + "metadata": { + "id": "i437rto_2lAY" + }, "source": [ "##### Export V1 (deprecated) \n", "1. ```project.export_labels()```\n", @@ -143,79 +191,191 @@ " - ```timeout_seconds: int = 1800```\n", " - Output : (str | List[Dict[Any, Any]] | None)" ], - "cell_type": "markdown" - }, - { - "metadata": {}, - "source": [ - "For a comprehensive example of Export V1 ``project.export_labels()`` output, please refer to our documentation: [Export V1 sample export](https://docs.labelbox.com/reference/export-image-annotations#sample-project-export)" - ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "i437rto_2lAY" }, { - "metadata": {}, + "metadata": { + "id": "UBUCnmdw2lAY" + }, "source": [ "2. ```project.label_generator()```\n", " - Parameters:\n", " - ```timeout_seconds: int = 600```\n", " - Output: LabelGenerator" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "UBUCnmdw2lAY" }, { - "metadata": {}, - "source": "# Single entry from the output of project.label_generator() (deprecated)\n# Label objects will not be deprecated.\nsingle_output_from_generator = \"\"\"\n\nLabel(\n uid='clrf5csho2ihx07ilffgp2fzj',\n data=ImageData(\n im_bytes=None,\n file_path=None,\n url='https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg',\n arr=None\n ),\n annotations=[\n ObjectAnnotation(\n confidence=None,\n name='bounding_box',\n feature_schema_id='clrf5ck4a0b9b071paa9ncu15',\n extra={\n 'instanceURI': 'https://api.labelbox.com/masks/feature/clrf5csvi6ofm07lsf9pygwvi?token='\n 'color': '#ff0000',\n 'feature_id': 'clrf5csvi6ofm07lsf9pygwvi',\n 'value': 'bounding_box',\n 'page': None,\n 'unit': None\n },\n value=Rectangle(\n extra={},\n start=Point(extra={}, x=2096.0, y=1264.0),\n end=Point(extra={}, x=2240.0, y=1689.0)\n ),\n classifications=[]\n ),\n # Add more annotations as needed\n # ...\n ],\n extra={\n 'Created By': 'aovalle@labelbox.com',\n 'Project Name': 'Image Annotation Import Demo',\n 'Created At': '2024-01-15T16:35:59.000Z',\n 'Updated At': '2024-01-15T16:51:56.000Z',\n 'Seconds to Label': 66.0,\n 'Agreement': -1.0,\n 'Benchmark Agreement': -1.0,\n 'Benchmark ID': None,\n 'Dataset Name': 'image-demo-dataset',\n 'Reviews': [],\n 'View Label': 'https://editor.labelbox.com?project=clrf5ckex09m9070x1te223u5&label=clrf5csho2ihx07ilffgp2fzj',\n 'Has Open Issues': 0.0,\n 'Skipped': False,\n 'media_type': 'image',\n 'Data Split': None,\n 'Global Key': '2560px-Kitano_Street_Kobe01s5s41102.jpeg'\n }\n)\n\n\"\"\"", + "metadata": { + "id": "NuxhjvUn2lAY" + }, + "source": [ + "# Single entry from the output of project.label_generator() (deprecated)\n", + "# Label objects will not be deprecated.\n", + "single_output_from_generator = \"\"\"\n", + "\n", + "Label(\n", + " uid='clrf5csho2ihx07ilffgp2fzj',\n", + " data=ImageData(\n", + " im_bytes=None,\n", + " file_path=None,\n", + " url='https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg',\n", + " arr=None\n", + " ),\n", + " annotations=[\n", + " ObjectAnnotation(\n", + " confidence=None,\n", + " name='bounding_box',\n", + " feature_schema_id='clrf5ck4a0b9b071paa9ncu15',\n", + " extra={\n", + " 'instanceURI': 'https://api.labelbox.com/masks/feature/clrf5csvi6ofm07lsf9pygwvi?token='\n", + " 'color': '#ff0000',\n", + " 'feature_id': 'clrf5csvi6ofm07lsf9pygwvi',\n", + " 'value': 'bounding_box',\n", + " 'page': None,\n", + " 'unit': None\n", + " },\n", + " value=Rectangle(\n", + " extra={},\n", + " start=Point(extra={}, x=2096.0, y=1264.0),\n", + " end=Point(extra={}, x=2240.0, y=1689.0)\n", + " ),\n", + " classifications=[]\n", + " ),\n", + " # Add more annotations as needed\n", + " # ...\n", + " ],\n", + " extra={\n", + " 'Created By': 'aovalle@labelbox.com',\n", + " 'Project Name': 'Image Annotation Import Demo',\n", + " 'Created At': '2024-01-15T16:35:59.000Z',\n", + " 'Updated At': '2024-01-15T16:51:56.000Z',\n", + " 'Seconds to Label': 66.0,\n", + " 'Agreement': -1.0,\n", + " 'Benchmark Agreement': -1.0,\n", + " 'Benchmark ID': None,\n", + " 'Dataset Name': 'image-demo-dataset',\n", + " 'Reviews': [],\n", + " 'View Label': 'https://editor.labelbox.com?project=clrf5ckex09m9070x1te223u5&label=clrf5csho2ihx07ilffgp2fzj',\n", + " 'Has Open Issues': 0.0,\n", + " 'Skipped': False,\n", + " 'media_type': 'image',\n", + " 'Data Split': None,\n", + " 'Global Key': '2560px-Kitano_Street_Kobe01s5s41102.jpeg'\n", + " }\n", + ")\n", + "\n", + "\"\"\"" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "NuxhjvUn2lAY" }, { - "metadata": {}, + "metadata": { + "id": "10XiIiAr2lAY" + }, "source": [ - "##### Export V2\n", + "##### Export\n", "\n", "For complete details on the supported filters abd parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", "\n", - "1. ```project.export()``` : Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "`project.export()`\n", + "- Parameters: \n", + " - ```\"label_details\": True```\n", + " - ```\"attachments\": True```\n", + " - ```\"data_row_details\": True```\n", + " - ```\"project_details\": True```\n", + " - ```\"label_details\": True```\n", + " - ```\"performance_details\": True```\n", + "- Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + ], + "cell_type": "markdown", + "id": "10XiIiAr2lAY" + }, + { + "metadata": { + "id": "hG4qwJgM2lAY" + }, + "source": [ + "## Set the export parameters to only export labels\n", + "export_params = {\n", + " \"attachments\": True,\n", + " \"metadata_fields\": True,\n", + " \"data_row_details\": True,\n", + " \"project_details\": True,\n", + " \"label_details\": True,\n", + " \"performance_details\": True,\n", + "}\n", + "# You also have the option to include additional filtering to narrow down the list of labels\n", + "filters = {}\n", "\n", - " - Parameters: \n", - " - ```\"label_details\": True```\n", - " - ```\"attachments\": True```\n", - " - ```\"data_row_details\": True```\n", - " - ```\"project_details\": True```\n", - " - ```\"label_details\": True```\n", - " - ```\"performance_details\": True```\n", - " - Output:\n", - " - ```ExportTask```\n", - " - `ExportTask.has_result()` return type: bool\n", - " - `ExportTask.has_errors()` return type: bool\n", - " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" - ], - "cell_type": "markdown" - }, - { - "metadata": {}, - "source": "## Set the export parameters to only export labels\nexport_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"label_details\": True,\n \"performance_details\": True,\n}\n# You also have the option to include additional filtering to narrow down the list of labels\nfilters = {}\n\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", + "export_task = project.export(params=export_params, filters=filters)\n", + "export_task.wait_till_done()" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "hG4qwJgM2lAY" }, { - "metadata": {}, - "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", + "metadata": { + "id": "lmFgiIwo2lAZ" + }, + "source": [ + "# Provide results with JSON converter\n", + "# Returns streamed JSON output strings from export task results/errors, one by one\n", + "\n", + "\n", + "# Callback used for JSON Converter\n", + "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", + " print(output.json)\n", + "\n", + "\n", + "if export_task.has_errors():\n", + " export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n", + " stream_handler=lambda error: print(error))\n", + "\n", + "if export_task.has_result():\n", + " export_json = export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.RESULT).start(\n", + " stream_handler=json_stream_handler)\n", + "\n", + "print(\n", + " \"file size: \",\n", + " export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n", + ")\n", + "print(\n", + " \"line count: \",\n", + " export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n", + ")" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "lmFgiIwo2lAZ" }, { - "metadata": {}, + "metadata": { + "id": "0qgmwVEs2lAZ" + }, "source": [ "## Export queued (\"To Label\") data rows from a project" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "0qgmwVEs2lAZ" }, { - "metadata": {}, + "metadata": { + "id": "nxqVz4sD2lAZ" + }, "source": [ "##### Export V1 (deprecated): \n", "1. ``project.export_queued_data_rows()`` :\n", @@ -224,263 +384,625 @@ " - ``include_metadata: bool = False``\n", " - Output: List[Dict[str, str]]" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "nxqVz4sD2lAZ" }, { - "metadata": {}, - "source": "# Single entry from the output of project.export_queued_data_rows() (deprecated)\nsingle_output_example = \"\"\"\n[\n {'id': 'clpouak6nap2g0783ajd1d6pf',\n 'createdAt': '2023-12-03T02:04:34.062Z',\n 'updatedAt': '2023-12-03T02:05:33.797Z',\n 'externalId': None,\n 'globalKey': 'b57c9ab2-304f-4c17-ba5f-c536f39a6a46',\n 'metadataFields': [],\n 'customMetadata': [],\n 'rowData': 'https://storage.googleapis.com/labelbox-developer-testing-assets/image/data_files/santa.jpeg',\n 'mediaAttributes': {'assetType': 'image',\n 'contentLength': 305973,\n 'height': 1333,\n 'mimeType': 'image/jpeg',\n 'subType': 'jpeg',\n 'superType': 'image',\n 'width': 2000}}\n]\n\n\"\"\"", + "metadata": { + "id": "GGJZFq2o2lAZ" + }, + "source": [ + "# Single entry from the output of project.export_queued_data_rows() (deprecated)\n", + "single_output_example = \"\"\"\n", + "[\n", + " {'id': 'clpouak6nap2g0783ajd1d6pf',\n", + " 'createdAt': '2023-12-03T02:04:34.062Z',\n", + " 'updatedAt': '2023-12-03T02:05:33.797Z',\n", + " 'externalId': None,\n", + " 'globalKey': 'b57c9ab2-304f-4c17-ba5f-c536f39a6a46',\n", + " 'metadataFields': [],\n", + " 'customMetadata': [],\n", + " 'rowData': 'https://storage.googleapis.com/labelbox-developer-testing-assets/image/data_files/santa.jpeg',\n", + " 'mediaAttributes': {'assetType': 'image',\n", + " 'contentLength': 305973,\n", + " 'height': 1333,\n", + " 'mimeType': 'image/jpeg',\n", + " 'subType': 'jpeg',\n", + " 'superType': 'image',\n", + " 'width': 2000}}\n", + "]\n", + "\n", + "\"\"\"" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "GGJZFq2o2lAZ" }, { - "metadata": {}, + "metadata": { + "id": "CvrTvKTg2lAZ" + }, "source": [ - "##### Export V2\n", - "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", + "##### Export\n", "\n", - "1. ```project.export()``` : Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "`project.export()`:\n", "\n", - " - Parameters (Minimum required parameters): \n", - " - ```\"data_row_details\": True```\n", - " - ```\"project_details\": True```\n", - " - Required filters:\n", - " - ``` \"workflow_status\": \"ToLabel\"```\n", - " - Output:\n", - " - ```ExportTask```\n", - " - `ExportTask.has_result()` return type: bool\n", - " - `ExportTask.has_errors()` return type: bool\n", - " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + "- Parameters (Minimum required parameters): \n", + " - ```\"data_row_details\": True```\n", + " - ```\"project_details\": True```\n", + "- Required filters:\n", + " - ``` \"workflow_status\": \"ToLabel\"```\n", + " - Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]\n", + " \n", + " For complete details on supported filters and parameters, including how they are used and what information is included, see [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters)." ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "CvrTvKTg2lAZ" }, { - "metadata": {}, - "source": "export_params = {\n \"attachments\": True, # Set to true if you want to export attachments\n \"metadata_fields\": True, # Set to true if you want to export metadata\n \"data_row_details\": True,\n \"project_details\": True,\n}\nfilters = {\n \"workflow_status\":\n \"ToLabel\" ## Using this filter will only export queued data rows\n}\n\n# An ExportTask is returned, this provides additional information about the status of your task, such as\n# any errors encountered and includes additional methods to stream your data\n\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", + "metadata": { + "id": "b97uh8MT2lAZ" + }, + "source": [ + "export_params = {\n", + " \"attachments\": True, # Set to true if you want to export attachments\n", + " \"metadata_fields\": True, # Set to true if you want to export metadata\n", + " \"data_row_details\": True,\n", + " \"project_details\": True,\n", + "}\n", + "filters = {\n", + " \"workflow_status\":\n", + " \"ToLabel\" ## Using this filter will only export queued data rows\n", + "}\n", + "\n", + "# An ExportTask is returned, this provides additional information about the status of your task, such as\n", + "# any errors encountered and includes additional methods to stream your data\n", + "\n", + "export_task = project.export(params=export_params, filters=filters)\n", + "export_task.wait_till_done()" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "b97uh8MT2lAZ" }, { - "metadata": {}, - "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", + "metadata": { + "id": "bi-mq9gi2lAZ" + }, + "source": [ + "# Provide results with JSON converter\n", + "# Returns streamed JSON output strings from export task results/errors, one by one\n", + "\n", + "\n", + "# Callback used for JSON Converter\n", + "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", + " print(output.json)\n", + "\n", + "\n", + "if export_task.has_errors():\n", + " export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n", + " stream_handler=lambda error: print(error))\n", + "\n", + "if export_task.has_result():\n", + " export_json = export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.RESULT).start(\n", + " stream_handler=json_stream_handler)\n", + "\n", + "print(\n", + " \"file size: \",\n", + " export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n", + ")\n", + "print(\n", + " \"line count: \",\n", + " export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n", + ")" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "bi-mq9gi2lAZ" }, { - "metadata": {}, + "metadata": { + "id": "vfCTKG3x2lAZ" + }, "source": [ "## Export data rows from a Dataset" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "vfCTKG3x2lAZ" }, { - "metadata": {}, - "source": "DATASET_ID = \"\"\ndataset = client.get_dataset(DATASET_ID)", + "metadata": { + "id": "-balByBh2lAZ" + }, + "source": [ + "DATASET_ID = \"\"\n", + "dataset = client.get_dataset(DATASET_ID)" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "-balByBh2lAZ" }, { - "metadata": {}, + "metadata": { + "id": "wuS4V7kF2lAZ" + }, "source": [ "#### Export V1 (deprecated):\n", "\n", - "1. ```dataset.export_data_rows()```\n", + "`dataset.export_data_rows()`\n", " - Parameters: \n", " - ``timeout_seconds=120``\n", " - ``include_metadata: bool = True``\n", " - Output:\n", " - Data row object generator\n" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "wuS4V7kF2lAZ" }, { - "metadata": {}, - "source": "# Single entry from the output of dataset.export_data_rows() (deprecated)\n# Data row objects will not be deprecated.\n\nsingle_output_from_data_row_generator = \"\"\"\n\n\"\"\"", + "metadata": { + "id": "BV40etU62lAZ" + }, + "source": [ + "# Single entry from the output of dataset.export_data_rows() (deprecated)\n", + "# Data row objects will not be deprecated.\n", + "\n", + "single_output_from_data_row_generator = \"\"\"\n", + "\n", + "\"\"\"" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "BV40etU62lAZ" }, { - "metadata": {}, + "metadata": { + "id": "9z06jScZ2lAZ" + }, "source": [ - "#### Export V2\n", - "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", + "#### Export\n", "\n", - "1. ```project.export()``` : Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "`project.export()`:\n", "\n", - " - Parameters (minimum required parameters): \n", - " - ``\"data_row_details\": True``\n", - " - Output:\n", - " - ```ExportTask```\n", - " - `ExportTask.has_result()` return type: bool\n", - " - `ExportTask.has_errors()` return type: bool\n", - " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + "- Parameters (minimum required parameters): \n", + " - ``\"data_row_details\": True``\n", + "- Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]\n", + "\n", + " For complete details on supported filters and parameters, including how they are used and what information is included, see [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters).\n", + " " ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "9z06jScZ2lAZ" }, { - "metadata": {}, - "source": "export_params = {\n \"attachments\": True, # Set to true if you want to export attachments\n \"metadata_fields\": True, # Set to true if you want to export metadata\n \"data_row_details\": True,\n}\nfilters = {}\n\n# A task is returned, this provides additional information about the status of your task, such as\n# any errors encountered\nexport_task = dataset.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", + "metadata": { + "id": "lDtRjIUR2lAZ" + }, + "source": [ + "export_params = {\n", + " \"attachments\": True, # Set to true if you want to export attachments\n", + " \"metadata_fields\": True, # Set to true if you want to export metadata\n", + " \"data_row_details\": True,\n", + "}\n", + "filters = {}\n", + "\n", + "# A task is returned, this provides additional information about the status of your task, such as\n", + "# any errors encountered\n", + "export_task = dataset.export(params=export_params, filters=filters)\n", + "export_task.wait_till_done()" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "lDtRjIUR2lAZ" }, { - "metadata": {}, - "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", + "metadata": { + "id": "_sL7lrq_2lAZ" + }, + "source": [ + "# Provide results with JSON converter\n", + "# Returns streamed JSON output strings from export task results/errors, one by one\n", + "\n", + "\n", + "# Callback used for JSON Converter\n", + "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", + " print(output.json)\n", + "\n", + "\n", + "if export_task.has_errors():\n", + " export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n", + " stream_handler=lambda error: print(error))\n", + "\n", + "if export_task.has_result():\n", + " export_json = export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.RESULT).start(\n", + " stream_handler=json_stream_handler)\n", + "\n", + "print(\n", + " \"file size: \",\n", + " export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n", + ")\n", + "print(\n", + " \"line count: \",\n", + " export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n", + ")" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "_sL7lrq_2lAZ" }, { - "metadata": {}, + "metadata": { + "id": "ITcc1b5t2lAZ" + }, "source": [ "## Export data rows from a batch" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "ITcc1b5t2lAZ" }, { - "metadata": {}, + "metadata": { + "id": "qaJn_Sy72lAZ" + }, "source": [ "#### Export V1 (deprecated):\n", - "1. ```batch.export_data_rows()```\n", + "`batch.export_data_rows()`\n", " - Parameters: \n", " - ``timeout_seconds=120``\n", " - ``include_metadata: bool = True``\n", " - Output:\n", " - Data row object generator" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "qaJn_Sy72lAZ" }, { - "metadata": {}, - "source": "# Single output from batch.export_data_rows() method (deprecated)\n# Data row objects will not be deprecated\n\nsingle_output_from_data_row_generator = \"\"\"\n\n\"\"\"", + "metadata": { + "id": "WYPWrNnO2lAZ" + }, + "source": [ + "# Single output from batch.export_data_rows() method (deprecated)\n", + "# Data row objects will not be deprecated\n", + "\n", + "single_output_from_data_row_generator = \"\"\"\n", + "\n", + "\"\"\"" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "WYPWrNnO2lAZ" }, { - "metadata": {}, + "metadata": { + "id": "0pwNtTFE2lAa" + }, "source": [ "#### Export V2\n", - "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", - "\n", - "1. ```project.export()``` : Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", "\n", - " - Required parameters: \n", - " - ```\"data_row_details\": True```,\n", - " - ```\"batch_ids\": [] ```\n", - " - Output:\n", - " - ```ExportTask```\n", - " - `ExportTask.has_result()` return type: bool\n", - " - `ExportTask.has_errors()` return type: bool\n", - " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + "`project.export()`:\n", + "- Required parameters: \n", + " - ```\"data_row_details\": True```,\n", + " - ```\"batch_ids\": [] ```\n", + "- Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]\n", + " \n", + " For complete details on supported filters and parameters, including how they are used and what information is included, see [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters)." ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "0pwNtTFE2lAa" }, { - "metadata": {}, - "source": "# Find the batch ID by navigating to \"Batches\" --> \"Manage batches\" --> \"Copy Batch ID\"\nBATCH_ID = \"\"", + "metadata": { + "id": "ItJ9n2lR2lAa" + }, + "source": [ + "# Find the batch ID by navigating to \"Batches\" --> \"Manage batches\" --> \"Copy Batch ID\"\n", + "BATCH_ID = \"\"" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "ItJ9n2lR2lAa" }, { - "metadata": {}, - "source": "export_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"performance_details\": True,\n \"batch_ids\": [\n BATCH_ID\n ], # Include batch ids if you only want to export specific batches, otherwise,\n # you can export all the data without using this parameter\n}\nfilters = {}\n\n# A task is returned, this provides additional information about the status of your task, such as\n# any errors encountered\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", + "metadata": { + "id": "d0RSIrMm2lAd" + }, + "source": [ + "export_params = {\n", + " \"attachments\": True,\n", + " \"metadata_fields\": True,\n", + " \"data_row_details\": True,\n", + " \"project_details\": True,\n", + " \"performance_details\": True,\n", + " \"batch_ids\": [\n", + " BATCH_ID\n", + " ], # Include batch ids if you only want to export specific batches, otherwise,\n", + " # you can export all the data without using this parameter\n", + "}\n", + "filters = {}\n", + "\n", + "# A task is returned, this provides additional information about the status of your task, such as\n", + "# any errors encountered\n", + "export_task = project.export(params=export_params, filters=filters)\n", + "export_task.wait_till_done()" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "d0RSIrMm2lAd" }, { - "metadata": {}, - "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", + "metadata": { + "id": "tXIZUMtI2lAd" + }, + "source": [ + "# Provide results with JSON converter\n", + "# Returns streamed JSON output strings from export task results/errors, one by one\n", + "\n", + "\n", + "# Callback used for JSON Converter\n", + "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", + " print(output.json)\n", + "\n", + "\n", + "if export_task.has_errors():\n", + " export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n", + " stream_handler=lambda error: print(error))\n", + "\n", + "if export_task.has_result():\n", + " export_json = export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.RESULT).start(\n", + " stream_handler=json_stream_handler)\n", + "\n", + "print(\n", + " \"file size: \",\n", + " export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n", + ")\n", + "print(\n", + " \"line count: \",\n", + " export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n", + ")" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "tXIZUMtI2lAd" }, { - "metadata": {}, + "metadata": { + "id": "mcWXemnL2lAd" + }, "source": [ "## Export data rows from a Model" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "mcWXemnL2lAd" }, { - "metadata": {}, + "metadata": { + "id": "5F317zYX2lAd" + }, "source": [ "#### Export V1 (deprecated):\n", - "1. ```model_run.export_labels(downlaod=True)```\n", - " - Parameters: \n", - " - ```download: bool = False```\n", - " - ```timeout_seconds: int = 1800```\n", - " - Output : (str | List[Dict[Any, Any]] | None)" + "`model_run.export_labels(downlaod=True)`\n", + "- Parameters: \n", + " - ```download: bool = False```\n", + " - ```timeout_seconds: int = 1800```\n", + "- Output : (str | List[Dict[Any, Any]] | None)" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "5F317zYX2lAd" }, { - "metadata": {}, - "source": "# Single output from model_run.export_labels()\nsingle_output_example = \"\"\"\n[\n {'ID': '1c48a7a0-3016-48e0-b0e3-47430f974869',\n 'Data Split': 'training',\n 'DataRow ID': 'clpqdyf650xd40712pycshy6a',\n 'External ID': './resume/BANKING/99124477.pdf',\n 'Labeled Data': 'https://storage.labelbox.com/cl5bn8qvq1av907xtb3bp8q60%2F8c6afc38-42a4-b2e1-a2e3-1e3b0c2998fc-99124477.pdf?Expires=1706637969726&KeyName=labelbox-assets-key-3&Signature=2nVt3sJ21CbjGS9I64yFquUELRw',\n 'Media Attributes': {'assetType': 'pdf',\n 'contentLength': 42535,\n 'mimeType': 'application/pdf',\n 'pageCount': 3,\n 'subType': 'pdf',\n 'superType': 'application'},\n 'Label': {'objects': [{'featureId': 'b9f3b584-0f45-050a-88d4-39c2a169c8e1',\n 'schemaId': 'clq1ckwbd08jp07z91q9mch5j',\n 'title': 'Test',\n 'value': 'test',\n 'color': '#1CE6FF',\n 'data': {'location': [{'text-bbox': {'page': 1,\n 'top': 158.44,\n 'left': 58.765,\n 'height': 13.691,\n 'width': 78.261}}],\n 'unit': 'POINTS'}}],\n 'classifications': [],\n 'relationships': []}}\n ]\n \"\"\"", + "metadata": { + "id": "d9gd1UZG2lAd" + }, + "source": [ + "# Single output from model_run.export_labels()\n", + "single_output_example = \"\"\"\n", + "[\n", + " {'ID': '1c48a7a0-3016-48e0-b0e3-47430f974869',\n", + " 'Data Split': 'training',\n", + " 'DataRow ID': 'clpqdyf650xd40712pycshy6a',\n", + " 'External ID': './resume/BANKING/99124477.pdf',\n", + " 'Labeled Data': 'https://storage.labelbox.com/cl5bn8qvq1av907xtb3bp8q60%2F8c6afc38-42a4-b2e1-a2e3-1e3b0c2998fc-99124477.pdf?Expires=1706637969726&KeyName=labelbox-assets-key-3&Signature=2nVt3sJ21CbjGS9I64yFquUELRw',\n", + " 'Media Attributes': {'assetType': 'pdf',\n", + " 'contentLength': 42535,\n", + " 'mimeType': 'application/pdf',\n", + " 'pageCount': 3,\n", + " 'subType': 'pdf',\n", + " 'superType': 'application'},\n", + " 'Label': {'objects': [{'featureId': 'b9f3b584-0f45-050a-88d4-39c2a169c8e1',\n", + " 'schemaId': 'clq1ckwbd08jp07z91q9mch5j',\n", + " 'title': 'Test',\n", + " 'value': 'test',\n", + " 'color': '#1CE6FF',\n", + " 'data': {'location': [{'text-bbox': {'page': 1,\n", + " 'top': 158.44,\n", + " 'left': 58.765,\n", + " 'height': 13.691,\n", + " 'width': 78.261}}],\n", + " 'unit': 'POINTS'}}],\n", + " 'classifications': [],\n", + " 'relationships': []}}\n", + " ]\n", + " \"\"\"" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "d9gd1UZG2lAd" }, { - "metadata": {}, + "metadata": { + "id": "4vr5GMb_2lAd" + }, "source": [ - "#### Export V2\n", - "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", + "#### Export\n", "\n", - "1. ```model_run.export()```: Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "`model_run.export()`:\n", "\n", - " - Required parameters: \n", - " - ```\"data_row_details\": True```\n", - " - ```\"project_details\": True```\n", - " - ```\"label_details\": True```\n", - " - Required filters:\n", - " - N/A -> Filters not supported\n", - " - Output:\n", - " - ```ExportTask```\n", - " - `ExportTask.has_result()` return type: bool\n", - " - `ExportTask.has_errors()` return type: bool\n", - " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + "- Required parameters: \n", + " - ```\"data_row_details\": True```\n", + " - ```\"project_details\": True```\n", + " - ```\"label_details\": True```\n", + "- Required filters:\n", + " - N/A -> Filters not supported\n", + "- Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]\n", + "\n", + "For complete details on supported filters and parameters, including how they are used and what information is included, see [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters)." ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "4vr5GMb_2lAd" }, { - "metadata": {}, - "source": "MODEL_RUN_ID = \"\"\nmodel_run = client.get_model_run(MODEL_RUN_ID)", + "metadata": { + "id": "-oqWcSiC2lAd" + }, + "source": [ + "MODEL_RUN_ID = \"\"\n", + "model_run = client.get_model_run(MODEL_RUN_ID)" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "-oqWcSiC2lAd" }, { - "metadata": {}, - "source": "export_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"performance_details\": True,\n}\n\nexport_task = model_run.export(params=export_params)\nexport_task.wait_till_done()", + "metadata": { + "id": "bniUCa_G2lAd" + }, + "source": [ + "export_params = {\n", + " \"attachments\": True,\n", + " \"metadata_fields\": True,\n", + " \"data_row_details\": True,\n", + " \"project_details\": True,\n", + " \"performance_details\": True,\n", + "}\n", + "\n", + "export_task = model_run.export(params=export_params)\n", + "export_task.wait_till_done()" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "bniUCa_G2lAd" }, { - "metadata": {}, - "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", + "metadata": { + "id": "8lLY3rs62lAd" + }, + "source": [ + "# Provide results with JSON converter\n", + "# Returns streamed JSON output strings from export task results/errors, one by one\n", + "\n", + "\n", + "# Callback used for JSON Converter\n", + "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", + " print(output.json)\n", + "\n", + "\n", + "if export_task.has_errors():\n", + " export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n", + " stream_handler=lambda error: print(error))\n", + "\n", + "if export_task.has_result():\n", + " export_json = export_task.get_buffered_stream(\n", + " stream_type=lb.StreamType.RESULT).start(\n", + " stream_handler=json_stream_handler)\n", + "\n", + "print(\n", + " \"file size: \",\n", + " export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n", + ")\n", + "print(\n", + " \"line count: \",\n", + " export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n", + ")" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "8lLY3rs62lAd" }, { - "metadata": {}, + "metadata": { + "id": "OiBgYekn2lAd" + }, "source": [ "## Export data rows from a video project\n", "Video projects include additional fields. Please refer to the example below to extract specific fields from video exports.\n" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "OiBgYekn2lAd" }, { - "metadata": {}, + "metadata": { + "id": "0Mdshde-2lAe" + }, "source": [ "##### Export V1 (deprecated) \n", "1. ```project.export_labels()```\n", @@ -489,103 +1011,194 @@ " - ```timeout_seconds: int = 1800```\n", " - Output : (str | List[Dict[Any, Any]] | None)" ], - "cell_type": "markdown" - }, - { - "metadata": {}, - "source": [ - "For a comprehensive example of Export V1 ``project.export_labels()`` output, please refer to our documentation: [Export V1 sample export](https://docs.labelbox.com/reference/export-image-annotations#sample-project-export)" - ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "0Mdshde-2lAe" }, { - "metadata": {}, + "metadata": { + "id": "idCN2wbo2lAe" + }, "source": [ "2. ```project.video_label_generator()```\n", " - Parameters:\n", " - ```timeout_seconds: int = 600```\n", " - Output: LabelGenerator" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "idCN2wbo2lAe" }, { - "metadata": {}, + "metadata": { + "id": "dT-RxuCr2lAe" + }, "source": [ - "##### Export V2\n", - "\n", - "For complete details on the supported filters and parameters, including how they are used and what information is included, please see the [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters) documentation.\n", + "##### Export\n", "\n", - "1. ```project.export()```: Starting from SDK version 3.56, a streamable method is available, this method allows you to stream unlimited number of data rows. However, if you are using an earlier version, you can still utilize the ```export_v2()``` function with identical parameters. It's important to note that the output task type differs, and streaming data methods are not included in `export_v2()`.\n", + "1. `project.export()`:\n", "\n", - " - Required parameters: \n", - " - ```\"attachments\": True```\n", - " - ```\"data_row_details\": True```\n", - " - ```\"project_details\": True```\n", - " - ```\"label_details\": True```\n", - " - ```\"performance_details\": True```\n", - " - Output:\n", - " - ```ExportTask```\n", - " - `ExportTask.has_result()` return type: bool\n", - " - `ExportTask.has_errors()` return type: bool\n", - " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" + "- Required parameters: \n", + " - ```\"attachments\": True```\n", + " - ```\"data_row_details\": True```\n", + " - ```\"project_details\": True```\n", + " - ```\"label_details\": True```\n", + " - ```\"performance_details\": True```\n", + "- Output:\n", + " - ```ExportTask```\n", + " - `ExportTask.has_result()` return type: bool\n", + " - `ExportTask.has_errors()` return type: bool\n", + " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]\n", + " \n", + " For complete details on supported filters and parameters, including how they are used and what information is included, see [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters)." ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "dT-RxuCr2lAe" }, { - "metadata": {}, - "source": "VIDEO_PROJECT_ID = \"\"\nproject = client.get_project(VIDEO_PROJECT_ID)", + "metadata": { + "id": "AiRRCkBo2lAe" + }, + "source": [ + "VIDEO_PROJECT_ID = \"\"\n", + "project = client.get_project(VIDEO_PROJECT_ID)" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "AiRRCkBo2lAe" }, { - "metadata": {}, - "source": "export_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"performance_details\": True,\n \"label_details\": True,\n \"interpolated_frames\":\n True, # For additional information on interpolated frames please visit our documentation https://docs.labelbox.com/docs/video-annotations#video-editor-components\n}\nfilters = {}\n\n# A task is returned, this provides additional information about the status of your task, such as\n# any errors encountered\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", + "metadata": { + "id": "neH75roK2lAe" + }, + "source": [ + "export_params = {\n", + " \"attachments\": True,\n", + " \"metadata_fields\": True,\n", + " \"data_row_details\": True,\n", + " \"project_details\": True,\n", + " \"performance_details\": True,\n", + " \"label_details\": True,\n", + " \"interpolated_frames\":\n", + " True, # For additional information on interpolated frames please visit our documentation https://docs.labelbox.com/docs/video-annotations#video-editor-components\n", + "}\n", + "filters = {}\n", + "\n", + "# A task is returned, this provides additional information about the status of your task, such as\n", + "# any errors encountered\n", + "export_task = project.export(params=export_params, filters=filters)\n", + "export_task.wait_till_done()" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "neH75roK2lAe" }, { - "metadata": {}, + "metadata": { + "id": "0yDLblWZ2lAe" + }, "source": [ "Fetch frame specific objects and frame or global classifications" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "0yDLblWZ2lAe" }, { - "metadata": {}, - "source": "import pprint as pp # Assuming pp is imported from pprint module\n\nframes_objects_class_list = []\nglobal_class_list = []\n\nstream = export_task.get_buffered_stream()\nfor output in stream:\n output_json = output.json\n for dr in output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]:\n frames_data = dr[\"annotations\"][\"frames\"]\n for k, v in frames_data.items():\n frames_objects_class_list.append({k: v})\n global_class_list.extend(dr[\"annotations\"][\"classifications\"])\n\n print(\"------- Frame specific classifications and objects -------\")\n pp.pprint(frames_objects_class_list)\n\n print(\"------ Global classifications -------\")\n pp.pprint(global_class_list)", + "metadata": { + "id": "vznQZCu32lAe" + }, + "source": [ + "import pprint as pp # Assuming pp is imported from pprint module\n", + "\n", + "frames_objects_class_list = []\n", + "global_class_list = []\n", + "\n", + "stream = export_task.get_buffered_stream()\n", + "for output in stream:\n", + " output_json = output.json\n", + " for dr in output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]:\n", + " frames_data = dr[\"annotations\"][\"frames\"]\n", + " for k, v in frames_data.items():\n", + " frames_objects_class_list.append({k: v})\n", + " global_class_list.extend(dr[\"annotations\"][\"classifications\"])\n", + "\n", + " print(\"------- Frame specific classifications and objects -------\")\n", + " pp.pprint(frames_objects_class_list)\n", + "\n", + " print(\"------ Global classifications -------\")\n", + " pp.pprint(global_class_list)" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "vznQZCu32lAe" }, { - "metadata": {}, + "metadata": { + "id": "i-Gb9rFL2lAe" + }, "source": [ "Fetch key frame feature map" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "i-Gb9rFL2lAe" }, { - "metadata": {}, - "source": "keyframe_map = []\n\nstream = export_task.get_buffered_stream()\nfor output in stream:\n output_json = output.json\n labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n for label in labels:\n annotations = label[\"annotations\"][\"key_frame_feature_map\"]\n for key, value in annotations.items():\n keyframe_map.append({key: value})\n\nprint(\"----- Keyframe Feature Map -----\")\npp.pprint(keyframe_map)", + "metadata": { + "id": "CfQgmlmS2lAe" + }, + "source": [ + "keyframe_map = []\n", + "\n", + "stream = export_task.get_buffered_stream()\n", + "for output in stream:\n", + " output_json = output.json\n", + " labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n", + " for label in labels:\n", + " annotations = label[\"annotations\"][\"key_frame_feature_map\"]\n", + " for key, value in annotations.items():\n", + " keyframe_map.append({key: value})\n", + "\n", + "print(\"----- Keyframe Feature Map -----\")\n", + "pp.pprint(keyframe_map)" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "CfQgmlmS2lAe" }, { - "metadata": {}, + "metadata": { + "id": "zxTrn1UX2lAe" + }, "source": [ "Fetch segments" ], - "cell_type": "markdown" + "cell_type": "markdown", + "id": "zxTrn1UX2lAe" }, { - "metadata": {}, - "source": "segments_map = []\nstream = export_task.get_buffered_stream()\nfor output in stream:\n output_json = output.json\n labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n for label in labels:\n annotations = label[\"annotations\"][\"segments\"]\n for key, value in annotations.items():\n segments_map.append({key: value})\n\nprint(\"----- Segments Feature Map -----\")\npp.pprint(segments_map)", + "metadata": { + "id": "klORakI52lAe" + }, + "source": [ + "segments_map = []\n", + "stream = export_task.get_buffered_stream()\n", + "for output in stream:\n", + " output_json = output.json\n", + " labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n", + " for label in labels:\n", + " annotations = label[\"annotations\"][\"segments\"]\n", + " for key, value in annotations.items():\n", + " segments_map.append({key: value})\n", + "\n", + "print(\"----- Segments Feature Map -----\")\n", + "pp.pprint(segments_map)" + ], "cell_type": "code", "outputs": [], - "execution_count": null + "execution_count": null, + "id": "klORakI52lAe" } ] -} \ No newline at end of file +} From fdd4ddb435f2766169a0e7168d0046ba416be877 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 18 Jul 2024 22:46:40 +0000 Subject: [PATCH 5/5] :art: Cleaned --- .../export_v1_to_v2_migration_support.ipynb | 928 +++--------------- 1 file changed, 148 insertions(+), 780 deletions(-) diff --git a/examples/exports/export_v1_to_v2_migration_support.ipynb b/examples/exports/export_v1_to_v2_migration_support.ipynb index 48d17f4d4..9fed974f6 100644 --- a/examples/exports/export_v1_to_v2_migration_support.ipynb +++ b/examples/exports/export_v1_to_v2_migration_support.ipynb @@ -1,26 +1,19 @@ { "nbformat": 4, "nbformat_minor": 5, - "metadata": { - "colab": { - "provenance": [] - } - }, + "metadata": {}, "cells": [ { - "metadata": { - "id": "ZvandVSm2lAW" - }, + "metadata": {}, "source": [ - " \n" + "", + " ", + "\n" ], - "cell_type": "markdown", - "id": "ZvandVSm2lAW" + "cell_type": "markdown" }, { - "metadata": { - "id": "XvojFdVV2lAX" - }, + "metadata": {}, "source": [ "\n", "\n", "" ], - "cell_type": "markdown", - "id": "XvojFdVV2lAX" + "cell_type": "markdown" }, { - "metadata": { - "id": "YUi47Rww2lAX" - }, + "metadata": {}, "source": [ "# Export V1 migration" ], - "cell_type": "markdown", - "id": "YUi47Rww2lAX" + "cell_type": "markdown" }, { - "metadata": { - "id": "V1YDn3Gd2lAX" - }, + "metadata": {}, "source": [ "Export V1 is no longer available in any version of the SDK in favor of the latest `export()` method, which allows you to export data with granular control. This notebook provide guidelines and comparisons on migrating from Export V1 to `export()` to ensure successful data export." ], - "cell_type": "markdown", - "id": "V1YDn3Gd2lAX" + "cell_type": "markdown" }, { - "metadata": { - "id": "gzHbqq8r2lAX" - }, + "metadata": {}, "source": [ "### Key changes\n", "The `export()` method adds the following changes and benefits compared to Export V1 methods:\n", @@ -68,13 +52,10 @@ "\n", "For complete details on how to use `export()`, see the [Export overview](https://docs.labelbox.com/reference/export-overview)." ], - "cell_type": "markdown", - "id": "gzHbqq8r2lAX" + "cell_type": "markdown" }, { - "metadata": { - "id": "Pxzeuk0t2lAX" - }, + "metadata": {}, "source": [ "### Export V1 deprecated methods:\n", "Project methods :\n", @@ -92,97 +73,60 @@ "1. ```model_run.export_labels()```\n", "\n" ], - "cell_type": "markdown", - "id": "Pxzeuk0t2lAX" + "cell_type": "markdown" }, { - "metadata": { - "id": "MIMSF6cG2lAY" - }, + "metadata": {}, "source": [ "# Imports" ], - "cell_type": "markdown", - "id": "MIMSF6cG2lAY" + "cell_type": "markdown" }, { - "metadata": { - "id": "PDEKcCWJ2lAY" - }, - "source": [ - "%pip install -q \"labelbox[data]\"" - ], + "metadata": {}, + "source": "%pip install -q \"labelbox[data]\"", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "PDEKcCWJ2lAY" + "execution_count": null }, { - "metadata": { - "id": "qdY6Lmtf2lAY" - }, - "source": [ - "import labelbox as lb\n", - "import pprint\n", - "\n", - "pp = pprint.PrettyPrinter(width=30, compact=True)" - ], + "metadata": {}, + "source": "import labelbox as lb\nimport pprint\n\npp = pprint.PrettyPrinter(width=30, compact=True)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "qdY6Lmtf2lAY" + "execution_count": null }, { - "metadata": { - "id": "haHB7K0c2lAY" - }, + "metadata": {}, "source": [ "## API Key and Client\n", "See the developer guide for [creating an API key](https://docs.labelbox.com/reference/create-api-key)." ], - "cell_type": "markdown", - "id": "haHB7K0c2lAY" + "cell_type": "markdown" }, { - "metadata": { - "id": "U3u1vj962lAY" - }, - "source": [ - "API_KEY = \"\"\n", - "client = lb.Client(api_key=API_KEY)" - ], + "metadata": {}, + "source": "API_KEY = \"\"\nclient = lb.Client(api_key=API_KEY)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "U3u1vj962lAY" + "execution_count": null }, { - "metadata": { - "id": "YWMQlB3h2lAY" - }, + "metadata": {}, "source": [ "## Export labels from a project\n" ], - "cell_type": "markdown", - "id": "YWMQlB3h2lAY" + "cell_type": "markdown" }, { - "metadata": { - "id": "FPs4Vb7l2lAY" - }, - "source": [ - "PROJECT_ID = \"\"\n", - "project = client.get_project(PROJECT_ID)" - ], + "metadata": {}, + "source": "PROJECT_ID = \"\"\nproject = client.get_project(PROJECT_ID)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "FPs4Vb7l2lAY" + "execution_count": null }, { - "metadata": { - "id": "i437rto_2lAY" - }, + "metadata": {}, "source": [ "##### Export V1 (deprecated) \n", "1. ```project.export_labels()```\n", @@ -191,93 +135,27 @@ " - ```timeout_seconds: int = 1800```\n", " - Output : (str | List[Dict[Any, Any]] | None)" ], - "cell_type": "markdown", - "id": "i437rto_2lAY" + "cell_type": "markdown" }, { - "metadata": { - "id": "UBUCnmdw2lAY" - }, + "metadata": {}, "source": [ "2. ```project.label_generator()```\n", " - Parameters:\n", " - ```timeout_seconds: int = 600```\n", " - Output: LabelGenerator" ], - "cell_type": "markdown", - "id": "UBUCnmdw2lAY" + "cell_type": "markdown" }, { - "metadata": { - "id": "NuxhjvUn2lAY" - }, - "source": [ - "# Single entry from the output of project.label_generator() (deprecated)\n", - "# Label objects will not be deprecated.\n", - "single_output_from_generator = \"\"\"\n", - "\n", - "Label(\n", - " uid='clrf5csho2ihx07ilffgp2fzj',\n", - " data=ImageData(\n", - " im_bytes=None,\n", - " file_path=None,\n", - " url='https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg',\n", - " arr=None\n", - " ),\n", - " annotations=[\n", - " ObjectAnnotation(\n", - " confidence=None,\n", - " name='bounding_box',\n", - " feature_schema_id='clrf5ck4a0b9b071paa9ncu15',\n", - " extra={\n", - " 'instanceURI': 'https://api.labelbox.com/masks/feature/clrf5csvi6ofm07lsf9pygwvi?token='\n", - " 'color': '#ff0000',\n", - " 'feature_id': 'clrf5csvi6ofm07lsf9pygwvi',\n", - " 'value': 'bounding_box',\n", - " 'page': None,\n", - " 'unit': None\n", - " },\n", - " value=Rectangle(\n", - " extra={},\n", - " start=Point(extra={}, x=2096.0, y=1264.0),\n", - " end=Point(extra={}, x=2240.0, y=1689.0)\n", - " ),\n", - " classifications=[]\n", - " ),\n", - " # Add more annotations as needed\n", - " # ...\n", - " ],\n", - " extra={\n", - " 'Created By': 'aovalle@labelbox.com',\n", - " 'Project Name': 'Image Annotation Import Demo',\n", - " 'Created At': '2024-01-15T16:35:59.000Z',\n", - " 'Updated At': '2024-01-15T16:51:56.000Z',\n", - " 'Seconds to Label': 66.0,\n", - " 'Agreement': -1.0,\n", - " 'Benchmark Agreement': -1.0,\n", - " 'Benchmark ID': None,\n", - " 'Dataset Name': 'image-demo-dataset',\n", - " 'Reviews': [],\n", - " 'View Label': 'https://editor.labelbox.com?project=clrf5ckex09m9070x1te223u5&label=clrf5csho2ihx07ilffgp2fzj',\n", - " 'Has Open Issues': 0.0,\n", - " 'Skipped': False,\n", - " 'media_type': 'image',\n", - " 'Data Split': None,\n", - " 'Global Key': '2560px-Kitano_Street_Kobe01s5s41102.jpeg'\n", - " }\n", - ")\n", - "\n", - "\"\"\"" - ], + "metadata": {}, + "source": "# Single entry from the output of project.label_generator() (deprecated)\n# Label objects will not be deprecated.\nsingle_output_from_generator = \"\"\"\n\nLabel(\n uid='clrf5csho2ihx07ilffgp2fzj',\n data=ImageData(\n im_bytes=None,\n file_path=None,\n url='https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg',\n arr=None\n ),\n annotations=[\n ObjectAnnotation(\n confidence=None,\n name='bounding_box',\n feature_schema_id='clrf5ck4a0b9b071paa9ncu15',\n extra={\n 'instanceURI': 'https://api.labelbox.com/masks/feature/clrf5csvi6ofm07lsf9pygwvi?token='\n 'color': '#ff0000',\n 'feature_id': 'clrf5csvi6ofm07lsf9pygwvi',\n 'value': 'bounding_box',\n 'page': None,\n 'unit': None\n },\n value=Rectangle(\n extra={},\n start=Point(extra={}, x=2096.0, y=1264.0),\n end=Point(extra={}, x=2240.0, y=1689.0)\n ),\n classifications=[]\n ),\n # Add more annotations as needed\n # ...\n ],\n extra={\n 'Created By': 'aovalle@labelbox.com',\n 'Project Name': 'Image Annotation Import Demo',\n 'Created At': '2024-01-15T16:35:59.000Z',\n 'Updated At': '2024-01-15T16:51:56.000Z',\n 'Seconds to Label': 66.0,\n 'Agreement': -1.0,\n 'Benchmark Agreement': -1.0,\n 'Benchmark ID': None,\n 'Dataset Name': 'image-demo-dataset',\n 'Reviews': [],\n 'View Label': 'https://editor.labelbox.com?project=clrf5ckex09m9070x1te223u5&label=clrf5csho2ihx07ilffgp2fzj',\n 'Has Open Issues': 0.0,\n 'Skipped': False,\n 'media_type': 'image',\n 'Data Split': None,\n 'Global Key': '2560px-Kitano_Street_Kobe01s5s41102.jpeg'\n }\n)\n\n\"\"\"", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "NuxhjvUn2lAY" + "execution_count": null }, { - "metadata": { - "id": "10XiIiAr2lAY" - }, + "metadata": {}, "source": [ "##### Export\n", "\n", @@ -297,85 +175,31 @@ " - `ExportTask.has_errors()` return type: bool\n", " - `ExportTask.get_buffered_stream()` return type: Stream[BufferedJsonConverterOutput]" ], - "cell_type": "markdown", - "id": "10XiIiAr2lAY" + "cell_type": "markdown" }, { - "metadata": { - "id": "hG4qwJgM2lAY" - }, - "source": [ - "## Set the export parameters to only export labels\n", - "export_params = {\n", - " \"attachments\": True,\n", - " \"metadata_fields\": True,\n", - " \"data_row_details\": True,\n", - " \"project_details\": True,\n", - " \"label_details\": True,\n", - " \"performance_details\": True,\n", - "}\n", - "# You also have the option to include additional filtering to narrow down the list of labels\n", - "filters = {}\n", - "\n", - "export_task = project.export(params=export_params, filters=filters)\n", - "export_task.wait_till_done()" - ], + "metadata": {}, + "source": "## Set the export parameters to only export labels\nexport_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"label_details\": True,\n \"performance_details\": True,\n}\n# You also have the option to include additional filtering to narrow down the list of labels\nfilters = {}\n\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "hG4qwJgM2lAY" + "execution_count": null }, { - "metadata": { - "id": "lmFgiIwo2lAZ" - }, - "source": [ - "# Provide results with JSON converter\n", - "# Returns streamed JSON output strings from export task results/errors, one by one\n", - "\n", - "\n", - "# Callback used for JSON Converter\n", - "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", - " print(output.json)\n", - "\n", - "\n", - "if export_task.has_errors():\n", - " export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n", - " stream_handler=lambda error: print(error))\n", - "\n", - "if export_task.has_result():\n", - " export_json = export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.RESULT).start(\n", - " stream_handler=json_stream_handler)\n", - "\n", - "print(\n", - " \"file size: \",\n", - " export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n", - ")\n", - "print(\n", - " \"line count: \",\n", - " export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n", - ")" - ], + "metadata": {}, + "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "lmFgiIwo2lAZ" + "execution_count": null }, { - "metadata": { - "id": "0qgmwVEs2lAZ" - }, + "metadata": {}, "source": [ "## Export queued (\"To Label\") data rows from a project" ], - "cell_type": "markdown", - "id": "0qgmwVEs2lAZ" + "cell_type": "markdown" }, { - "metadata": { - "id": "nxqVz4sD2lAZ" - }, + "metadata": {}, "source": [ "##### Export V1 (deprecated): \n", "1. ``project.export_queued_data_rows()`` :\n", @@ -384,45 +208,17 @@ " - ``include_metadata: bool = False``\n", " - Output: List[Dict[str, str]]" ], - "cell_type": "markdown", - "id": "nxqVz4sD2lAZ" + "cell_type": "markdown" }, { - "metadata": { - "id": "GGJZFq2o2lAZ" - }, - "source": [ - "# Single entry from the output of project.export_queued_data_rows() (deprecated)\n", - "single_output_example = \"\"\"\n", - "[\n", - " {'id': 'clpouak6nap2g0783ajd1d6pf',\n", - " 'createdAt': '2023-12-03T02:04:34.062Z',\n", - " 'updatedAt': '2023-12-03T02:05:33.797Z',\n", - " 'externalId': None,\n", - " 'globalKey': 'b57c9ab2-304f-4c17-ba5f-c536f39a6a46',\n", - " 'metadataFields': [],\n", - " 'customMetadata': [],\n", - " 'rowData': 'https://storage.googleapis.com/labelbox-developer-testing-assets/image/data_files/santa.jpeg',\n", - " 'mediaAttributes': {'assetType': 'image',\n", - " 'contentLength': 305973,\n", - " 'height': 1333,\n", - " 'mimeType': 'image/jpeg',\n", - " 'subType': 'jpeg',\n", - " 'superType': 'image',\n", - " 'width': 2000}}\n", - "]\n", - "\n", - "\"\"\"" - ], + "metadata": {}, + "source": "# Single entry from the output of project.export_queued_data_rows() (deprecated)\nsingle_output_example = \"\"\"\n[\n {'id': 'clpouak6nap2g0783ajd1d6pf',\n 'createdAt': '2023-12-03T02:04:34.062Z',\n 'updatedAt': '2023-12-03T02:05:33.797Z',\n 'externalId': None,\n 'globalKey': 'b57c9ab2-304f-4c17-ba5f-c536f39a6a46',\n 'metadataFields': [],\n 'customMetadata': [],\n 'rowData': 'https://storage.googleapis.com/labelbox-developer-testing-assets/image/data_files/santa.jpeg',\n 'mediaAttributes': {'assetType': 'image',\n 'contentLength': 305973,\n 'height': 1333,\n 'mimeType': 'image/jpeg',\n 'subType': 'jpeg',\n 'superType': 'image',\n 'width': 2000}}\n]\n\n\"\"\"", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "GGJZFq2o2lAZ" + "execution_count": null }, { - "metadata": { - "id": "CvrTvKTg2lAZ" - }, + "metadata": {}, "source": [ "##### Export\n", "\n", @@ -441,100 +237,38 @@ " \n", " For complete details on supported filters and parameters, including how they are used and what information is included, see [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters)." ], - "cell_type": "markdown", - "id": "CvrTvKTg2lAZ" + "cell_type": "markdown" }, { - "metadata": { - "id": "b97uh8MT2lAZ" - }, - "source": [ - "export_params = {\n", - " \"attachments\": True, # Set to true if you want to export attachments\n", - " \"metadata_fields\": True, # Set to true if you want to export metadata\n", - " \"data_row_details\": True,\n", - " \"project_details\": True,\n", - "}\n", - "filters = {\n", - " \"workflow_status\":\n", - " \"ToLabel\" ## Using this filter will only export queued data rows\n", - "}\n", - "\n", - "# An ExportTask is returned, this provides additional information about the status of your task, such as\n", - "# any errors encountered and includes additional methods to stream your data\n", - "\n", - "export_task = project.export(params=export_params, filters=filters)\n", - "export_task.wait_till_done()" - ], + "metadata": {}, + "source": "export_params = {\n \"attachments\": True, # Set to true if you want to export attachments\n \"metadata_fields\": True, # Set to true if you want to export metadata\n \"data_row_details\": True,\n \"project_details\": True,\n}\nfilters = {\n \"workflow_status\":\n \"ToLabel\" ## Using this filter will only export queued data rows\n}\n\n# An ExportTask is returned, this provides additional information about the status of your task, such as\n# any errors encountered and includes additional methods to stream your data\n\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "b97uh8MT2lAZ" + "execution_count": null }, { - "metadata": { - "id": "bi-mq9gi2lAZ" - }, - "source": [ - "# Provide results with JSON converter\n", - "# Returns streamed JSON output strings from export task results/errors, one by one\n", - "\n", - "\n", - "# Callback used for JSON Converter\n", - "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", - " print(output.json)\n", - "\n", - "\n", - "if export_task.has_errors():\n", - " export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n", - " stream_handler=lambda error: print(error))\n", - "\n", - "if export_task.has_result():\n", - " export_json = export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.RESULT).start(\n", - " stream_handler=json_stream_handler)\n", - "\n", - "print(\n", - " \"file size: \",\n", - " export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n", - ")\n", - "print(\n", - " \"line count: \",\n", - " export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n", - ")" - ], + "metadata": {}, + "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "bi-mq9gi2lAZ" + "execution_count": null }, { - "metadata": { - "id": "vfCTKG3x2lAZ" - }, + "metadata": {}, "source": [ "## Export data rows from a Dataset" ], - "cell_type": "markdown", - "id": "vfCTKG3x2lAZ" + "cell_type": "markdown" }, { - "metadata": { - "id": "-balByBh2lAZ" - }, - "source": [ - "DATASET_ID = \"\"\n", - "dataset = client.get_dataset(DATASET_ID)" - ], + "metadata": {}, + "source": "DATASET_ID = \"\"\ndataset = client.get_dataset(DATASET_ID)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "-balByBh2lAZ" + "execution_count": null }, { - "metadata": { - "id": "wuS4V7kF2lAZ" - }, + "metadata": {}, "source": [ "#### Export V1 (deprecated):\n", "\n", @@ -545,48 +279,17 @@ " - Output:\n", " - Data row object generator\n" ], - "cell_type": "markdown", - "id": "wuS4V7kF2lAZ" + "cell_type": "markdown" }, { - "metadata": { - "id": "BV40etU62lAZ" - }, - "source": [ - "# Single entry from the output of dataset.export_data_rows() (deprecated)\n", - "# Data row objects will not be deprecated.\n", - "\n", - "single_output_from_data_row_generator = \"\"\"\n", - "\n", - "\"\"\"" - ], + "metadata": {}, + "source": "# Single entry from the output of dataset.export_data_rows() (deprecated)\n# Data row objects will not be deprecated.\n\nsingle_output_from_data_row_generator = \"\"\"\n\n\"\"\"", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "BV40etU62lAZ" + "execution_count": null }, { - "metadata": { - "id": "9z06jScZ2lAZ" - }, + "metadata": {}, "source": [ "#### Export\n", "\n", @@ -603,82 +306,31 @@ " For complete details on supported filters and parameters, including how they are used and what information is included, see [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters).\n", " " ], - "cell_type": "markdown", - "id": "9z06jScZ2lAZ" + "cell_type": "markdown" }, { - "metadata": { - "id": "lDtRjIUR2lAZ" - }, - "source": [ - "export_params = {\n", - " \"attachments\": True, # Set to true if you want to export attachments\n", - " \"metadata_fields\": True, # Set to true if you want to export metadata\n", - " \"data_row_details\": True,\n", - "}\n", - "filters = {}\n", - "\n", - "# A task is returned, this provides additional information about the status of your task, such as\n", - "# any errors encountered\n", - "export_task = dataset.export(params=export_params, filters=filters)\n", - "export_task.wait_till_done()" - ], + "metadata": {}, + "source": "export_params = {\n \"attachments\": True, # Set to true if you want to export attachments\n \"metadata_fields\": True, # Set to true if you want to export metadata\n \"data_row_details\": True,\n}\nfilters = {}\n\n# A task is returned, this provides additional information about the status of your task, such as\n# any errors encountered\nexport_task = dataset.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "lDtRjIUR2lAZ" + "execution_count": null }, { - "metadata": { - "id": "_sL7lrq_2lAZ" - }, - "source": [ - "# Provide results with JSON converter\n", - "# Returns streamed JSON output strings from export task results/errors, one by one\n", - "\n", - "\n", - "# Callback used for JSON Converter\n", - "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", - " print(output.json)\n", - "\n", - "\n", - "if export_task.has_errors():\n", - " export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n", - " stream_handler=lambda error: print(error))\n", - "\n", - "if export_task.has_result():\n", - " export_json = export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.RESULT).start(\n", - " stream_handler=json_stream_handler)\n", - "\n", - "print(\n", - " \"file size: \",\n", - " export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n", - ")\n", - "print(\n", - " \"line count: \",\n", - " export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n", - ")" - ], + "metadata": {}, + "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "_sL7lrq_2lAZ" + "execution_count": null }, { - "metadata": { - "id": "ITcc1b5t2lAZ" - }, + "metadata": {}, "source": [ "## Export data rows from a batch" ], - "cell_type": "markdown", - "id": "ITcc1b5t2lAZ" + "cell_type": "markdown" }, { - "metadata": { - "id": "qaJn_Sy72lAZ" - }, + "metadata": {}, "source": [ "#### Export V1 (deprecated):\n", "`batch.export_data_rows()`\n", @@ -688,48 +340,17 @@ " - Output:\n", " - Data row object generator" ], - "cell_type": "markdown", - "id": "qaJn_Sy72lAZ" + "cell_type": "markdown" }, { - "metadata": { - "id": "WYPWrNnO2lAZ" - }, - "source": [ - "# Single output from batch.export_data_rows() method (deprecated)\n", - "# Data row objects will not be deprecated\n", - "\n", - "single_output_from_data_row_generator = \"\"\"\n", - "\n", - "\"\"\"" - ], + "metadata": {}, + "source": "# Single output from batch.export_data_rows() method (deprecated)\n# Data row objects will not be deprecated\n\nsingle_output_from_data_row_generator = \"\"\"\n\n\"\"\"", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "WYPWrNnO2lAZ" + "execution_count": null }, { - "metadata": { - "id": "0pwNtTFE2lAa" - }, + "metadata": {}, "source": [ "#### Export V2\n", "\n", @@ -745,101 +366,38 @@ " \n", " For complete details on supported filters and parameters, including how they are used and what information is included, see [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters)." ], - "cell_type": "markdown", - "id": "0pwNtTFE2lAa" + "cell_type": "markdown" }, { - "metadata": { - "id": "ItJ9n2lR2lAa" - }, - "source": [ - "# Find the batch ID by navigating to \"Batches\" --> \"Manage batches\" --> \"Copy Batch ID\"\n", - "BATCH_ID = \"\"" - ], + "metadata": {}, + "source": "# Find the batch ID by navigating to \"Batches\" --> \"Manage batches\" --> \"Copy Batch ID\"\nBATCH_ID = \"\"", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "ItJ9n2lR2lAa" + "execution_count": null }, { - "metadata": { - "id": "d0RSIrMm2lAd" - }, - "source": [ - "export_params = {\n", - " \"attachments\": True,\n", - " \"metadata_fields\": True,\n", - " \"data_row_details\": True,\n", - " \"project_details\": True,\n", - " \"performance_details\": True,\n", - " \"batch_ids\": [\n", - " BATCH_ID\n", - " ], # Include batch ids if you only want to export specific batches, otherwise,\n", - " # you can export all the data without using this parameter\n", - "}\n", - "filters = {}\n", - "\n", - "# A task is returned, this provides additional information about the status of your task, such as\n", - "# any errors encountered\n", - "export_task = project.export(params=export_params, filters=filters)\n", - "export_task.wait_till_done()" - ], + "metadata": {}, + "source": "export_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"performance_details\": True,\n \"batch_ids\": [\n BATCH_ID\n ], # Include batch ids if you only want to export specific batches, otherwise,\n # you can export all the data without using this parameter\n}\nfilters = {}\n\n# A task is returned, this provides additional information about the status of your task, such as\n# any errors encountered\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "d0RSIrMm2lAd" + "execution_count": null }, { - "metadata": { - "id": "tXIZUMtI2lAd" - }, - "source": [ - "# Provide results with JSON converter\n", - "# Returns streamed JSON output strings from export task results/errors, one by one\n", - "\n", - "\n", - "# Callback used for JSON Converter\n", - "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", - " print(output.json)\n", - "\n", - "\n", - "if export_task.has_errors():\n", - " export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n", - " stream_handler=lambda error: print(error))\n", - "\n", - "if export_task.has_result():\n", - " export_json = export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.RESULT).start(\n", - " stream_handler=json_stream_handler)\n", - "\n", - "print(\n", - " \"file size: \",\n", - " export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n", - ")\n", - "print(\n", - " \"line count: \",\n", - " export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n", - ")" - ], + "metadata": {}, + "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "tXIZUMtI2lAd" + "execution_count": null }, { - "metadata": { - "id": "mcWXemnL2lAd" - }, + "metadata": {}, "source": [ "## Export data rows from a Model" ], - "cell_type": "markdown", - "id": "mcWXemnL2lAd" + "cell_type": "markdown" }, { - "metadata": { - "id": "5F317zYX2lAd" - }, + "metadata": {}, "source": [ "#### Export V1 (deprecated):\n", "`model_run.export_labels(downlaod=True)`\n", @@ -848,53 +406,17 @@ " - ```timeout_seconds: int = 1800```\n", "- Output : (str | List[Dict[Any, Any]] | None)" ], - "cell_type": "markdown", - "id": "5F317zYX2lAd" + "cell_type": "markdown" }, { - "metadata": { - "id": "d9gd1UZG2lAd" - }, - "source": [ - "# Single output from model_run.export_labels()\n", - "single_output_example = \"\"\"\n", - "[\n", - " {'ID': '1c48a7a0-3016-48e0-b0e3-47430f974869',\n", - " 'Data Split': 'training',\n", - " 'DataRow ID': 'clpqdyf650xd40712pycshy6a',\n", - " 'External ID': './resume/BANKING/99124477.pdf',\n", - " 'Labeled Data': 'https://storage.labelbox.com/cl5bn8qvq1av907xtb3bp8q60%2F8c6afc38-42a4-b2e1-a2e3-1e3b0c2998fc-99124477.pdf?Expires=1706637969726&KeyName=labelbox-assets-key-3&Signature=2nVt3sJ21CbjGS9I64yFquUELRw',\n", - " 'Media Attributes': {'assetType': 'pdf',\n", - " 'contentLength': 42535,\n", - " 'mimeType': 'application/pdf',\n", - " 'pageCount': 3,\n", - " 'subType': 'pdf',\n", - " 'superType': 'application'},\n", - " 'Label': {'objects': [{'featureId': 'b9f3b584-0f45-050a-88d4-39c2a169c8e1',\n", - " 'schemaId': 'clq1ckwbd08jp07z91q9mch5j',\n", - " 'title': 'Test',\n", - " 'value': 'test',\n", - " 'color': '#1CE6FF',\n", - " 'data': {'location': [{'text-bbox': {'page': 1,\n", - " 'top': 158.44,\n", - " 'left': 58.765,\n", - " 'height': 13.691,\n", - " 'width': 78.261}}],\n", - " 'unit': 'POINTS'}}],\n", - " 'classifications': [],\n", - " 'relationships': []}}\n", - " ]\n", - " \"\"\"" - ], + "metadata": {}, + "source": "# Single output from model_run.export_labels()\nsingle_output_example = \"\"\"\n[\n {'ID': '1c48a7a0-3016-48e0-b0e3-47430f974869',\n 'Data Split': 'training',\n 'DataRow ID': 'clpqdyf650xd40712pycshy6a',\n 'External ID': './resume/BANKING/99124477.pdf',\n 'Labeled Data': 'https://storage.labelbox.com/cl5bn8qvq1av907xtb3bp8q60%2F8c6afc38-42a4-b2e1-a2e3-1e3b0c2998fc-99124477.pdf?Expires=1706637969726&KeyName=labelbox-assets-key-3&Signature=2nVt3sJ21CbjGS9I64yFquUELRw',\n 'Media Attributes': {'assetType': 'pdf',\n 'contentLength': 42535,\n 'mimeType': 'application/pdf',\n 'pageCount': 3,\n 'subType': 'pdf',\n 'superType': 'application'},\n 'Label': {'objects': [{'featureId': 'b9f3b584-0f45-050a-88d4-39c2a169c8e1',\n 'schemaId': 'clq1ckwbd08jp07z91q9mch5j',\n 'title': 'Test',\n 'value': 'test',\n 'color': '#1CE6FF',\n 'data': {'location': [{'text-bbox': {'page': 1,\n 'top': 158.44,\n 'left': 58.765,\n 'height': 13.691,\n 'width': 78.261}}],\n 'unit': 'POINTS'}}],\n 'classifications': [],\n 'relationships': []}}\n ]\n \"\"\"", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "d9gd1UZG2lAd" + "execution_count": null }, { - "metadata": { - "id": "4vr5GMb_2lAd" - }, + "metadata": {}, "source": [ "#### Export\n", "\n", @@ -914,95 +436,39 @@ "\n", "For complete details on supported filters and parameters, including how they are used and what information is included, see [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters)." ], - "cell_type": "markdown", - "id": "4vr5GMb_2lAd" + "cell_type": "markdown" }, { - "metadata": { - "id": "-oqWcSiC2lAd" - }, - "source": [ - "MODEL_RUN_ID = \"\"\n", - "model_run = client.get_model_run(MODEL_RUN_ID)" - ], + "metadata": {}, + "source": "MODEL_RUN_ID = \"\"\nmodel_run = client.get_model_run(MODEL_RUN_ID)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "-oqWcSiC2lAd" + "execution_count": null }, { - "metadata": { - "id": "bniUCa_G2lAd" - }, - "source": [ - "export_params = {\n", - " \"attachments\": True,\n", - " \"metadata_fields\": True,\n", - " \"data_row_details\": True,\n", - " \"project_details\": True,\n", - " \"performance_details\": True,\n", - "}\n", - "\n", - "export_task = model_run.export(params=export_params)\n", - "export_task.wait_till_done()" - ], + "metadata": {}, + "source": "export_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"performance_details\": True,\n}\n\nexport_task = model_run.export(params=export_params)\nexport_task.wait_till_done()", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "bniUCa_G2lAd" + "execution_count": null }, { - "metadata": { - "id": "8lLY3rs62lAd" - }, - "source": [ - "# Provide results with JSON converter\n", - "# Returns streamed JSON output strings from export task results/errors, one by one\n", - "\n", - "\n", - "# Callback used for JSON Converter\n", - "def json_stream_handler(output: lb.BufferedJsonConverterOutput):\n", - " print(output.json)\n", - "\n", - "\n", - "if export_task.has_errors():\n", - " export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n", - " stream_handler=lambda error: print(error))\n", - "\n", - "if export_task.has_result():\n", - " export_json = export_task.get_buffered_stream(\n", - " stream_type=lb.StreamType.RESULT).start(\n", - " stream_handler=json_stream_handler)\n", - "\n", - "print(\n", - " \"file size: \",\n", - " export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n", - ")\n", - "print(\n", - " \"line count: \",\n", - " export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n", - ")" - ], + "metadata": {}, + "source": "# Provide results with JSON converter\n# Returns streamed JSON output strings from export task results/errors, one by one\n\n\n# Callback used for JSON Converter\ndef json_stream_handler(output: lb.BufferedJsonConverterOutput):\n print(output.json)\n\n\nif export_task.has_errors():\n export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(\n stream_handler=lambda error: print(error))\n\nif export_task.has_result():\n export_json = export_task.get_buffered_stream(\n stream_type=lb.StreamType.RESULT).start(\n stream_handler=json_stream_handler)\n\nprint(\n \"file size: \",\n export_task.get_total_file_size(stream_type=lb.StreamType.RESULT),\n)\nprint(\n \"line count: \",\n export_task.get_total_lines(stream_type=lb.StreamType.RESULT),\n)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "8lLY3rs62lAd" + "execution_count": null }, { - "metadata": { - "id": "OiBgYekn2lAd" - }, + "metadata": {}, "source": [ "## Export data rows from a video project\n", "Video projects include additional fields. Please refer to the example below to extract specific fields from video exports.\n" ], - "cell_type": "markdown", - "id": "OiBgYekn2lAd" + "cell_type": "markdown" }, { - "metadata": { - "id": "0Mdshde-2lAe" - }, + "metadata": {}, "source": [ "##### Export V1 (deprecated) \n", "1. ```project.export_labels()```\n", @@ -1011,26 +477,20 @@ " - ```timeout_seconds: int = 1800```\n", " - Output : (str | List[Dict[Any, Any]] | None)" ], - "cell_type": "markdown", - "id": "0Mdshde-2lAe" + "cell_type": "markdown" }, { - "metadata": { - "id": "idCN2wbo2lAe" - }, + "metadata": {}, "source": [ "2. ```project.video_label_generator()```\n", " - Parameters:\n", " - ```timeout_seconds: int = 600```\n", " - Output: LabelGenerator" ], - "cell_type": "markdown", - "id": "idCN2wbo2lAe" + "cell_type": "markdown" }, { - "metadata": { - "id": "dT-RxuCr2lAe" - }, + "metadata": {}, "source": [ "##### Export\n", "\n", @@ -1050,155 +510,63 @@ " \n", " For complete details on supported filters and parameters, including how they are used and what information is included, see [Export overview](https://docs.labelbox.com/reference/label-export#optional-parameters-and-filters)." ], - "cell_type": "markdown", - "id": "dT-RxuCr2lAe" + "cell_type": "markdown" }, { - "metadata": { - "id": "AiRRCkBo2lAe" - }, - "source": [ - "VIDEO_PROJECT_ID = \"\"\n", - "project = client.get_project(VIDEO_PROJECT_ID)" - ], + "metadata": {}, + "source": "VIDEO_PROJECT_ID = \"\"\nproject = client.get_project(VIDEO_PROJECT_ID)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "AiRRCkBo2lAe" + "execution_count": null }, { - "metadata": { - "id": "neH75roK2lAe" - }, - "source": [ - "export_params = {\n", - " \"attachments\": True,\n", - " \"metadata_fields\": True,\n", - " \"data_row_details\": True,\n", - " \"project_details\": True,\n", - " \"performance_details\": True,\n", - " \"label_details\": True,\n", - " \"interpolated_frames\":\n", - " True, # For additional information on interpolated frames please visit our documentation https://docs.labelbox.com/docs/video-annotations#video-editor-components\n", - "}\n", - "filters = {}\n", - "\n", - "# A task is returned, this provides additional information about the status of your task, such as\n", - "# any errors encountered\n", - "export_task = project.export(params=export_params, filters=filters)\n", - "export_task.wait_till_done()" - ], + "metadata": {}, + "source": "export_params = {\n \"attachments\": True,\n \"metadata_fields\": True,\n \"data_row_details\": True,\n \"project_details\": True,\n \"performance_details\": True,\n \"label_details\": True,\n \"interpolated_frames\":\n True, # For additional information on interpolated frames please visit our documentation https://docs.labelbox.com/docs/video-annotations#video-editor-components\n}\nfilters = {}\n\n# A task is returned, this provides additional information about the status of your task, such as\n# any errors encountered\nexport_task = project.export(params=export_params, filters=filters)\nexport_task.wait_till_done()", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "neH75roK2lAe" + "execution_count": null }, { - "metadata": { - "id": "0yDLblWZ2lAe" - }, + "metadata": {}, "source": [ "Fetch frame specific objects and frame or global classifications" ], - "cell_type": "markdown", - "id": "0yDLblWZ2lAe" + "cell_type": "markdown" }, { - "metadata": { - "id": "vznQZCu32lAe" - }, - "source": [ - "import pprint as pp # Assuming pp is imported from pprint module\n", - "\n", - "frames_objects_class_list = []\n", - "global_class_list = []\n", - "\n", - "stream = export_task.get_buffered_stream()\n", - "for output in stream:\n", - " output_json = output.json\n", - " for dr in output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]:\n", - " frames_data = dr[\"annotations\"][\"frames\"]\n", - " for k, v in frames_data.items():\n", - " frames_objects_class_list.append({k: v})\n", - " global_class_list.extend(dr[\"annotations\"][\"classifications\"])\n", - "\n", - " print(\"------- Frame specific classifications and objects -------\")\n", - " pp.pprint(frames_objects_class_list)\n", - "\n", - " print(\"------ Global classifications -------\")\n", - " pp.pprint(global_class_list)" - ], + "metadata": {}, + "source": "import pprint as pp # Assuming pp is imported from pprint module\n\nframes_objects_class_list = []\nglobal_class_list = []\n\nstream = export_task.get_buffered_stream()\nfor output in stream:\n output_json = output.json\n for dr in output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]:\n frames_data = dr[\"annotations\"][\"frames\"]\n for k, v in frames_data.items():\n frames_objects_class_list.append({k: v})\n global_class_list.extend(dr[\"annotations\"][\"classifications\"])\n\n print(\"------- Frame specific classifications and objects -------\")\n pp.pprint(frames_objects_class_list)\n\n print(\"------ Global classifications -------\")\n pp.pprint(global_class_list)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "vznQZCu32lAe" + "execution_count": null }, { - "metadata": { - "id": "i-Gb9rFL2lAe" - }, + "metadata": {}, "source": [ "Fetch key frame feature map" ], - "cell_type": "markdown", - "id": "i-Gb9rFL2lAe" + "cell_type": "markdown" }, { - "metadata": { - "id": "CfQgmlmS2lAe" - }, - "source": [ - "keyframe_map = []\n", - "\n", - "stream = export_task.get_buffered_stream()\n", - "for output in stream:\n", - " output_json = output.json\n", - " labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n", - " for label in labels:\n", - " annotations = label[\"annotations\"][\"key_frame_feature_map\"]\n", - " for key, value in annotations.items():\n", - " keyframe_map.append({key: value})\n", - "\n", - "print(\"----- Keyframe Feature Map -----\")\n", - "pp.pprint(keyframe_map)" - ], + "metadata": {}, + "source": "keyframe_map = []\n\nstream = export_task.get_buffered_stream()\nfor output in stream:\n output_json = output.json\n labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n for label in labels:\n annotations = label[\"annotations\"][\"key_frame_feature_map\"]\n for key, value in annotations.items():\n keyframe_map.append({key: value})\n\nprint(\"----- Keyframe Feature Map -----\")\npp.pprint(keyframe_map)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "CfQgmlmS2lAe" + "execution_count": null }, { - "metadata": { - "id": "zxTrn1UX2lAe" - }, + "metadata": {}, "source": [ "Fetch segments" ], - "cell_type": "markdown", - "id": "zxTrn1UX2lAe" + "cell_type": "markdown" }, { - "metadata": { - "id": "klORakI52lAe" - }, - "source": [ - "segments_map = []\n", - "stream = export_task.get_buffered_stream()\n", - "for output in stream:\n", - " output_json = output.json\n", - " labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n", - " for label in labels:\n", - " annotations = label[\"annotations\"][\"segments\"]\n", - " for key, value in annotations.items():\n", - " segments_map.append({key: value})\n", - "\n", - "print(\"----- Segments Feature Map -----\")\n", - "pp.pprint(segments_map)" - ], + "metadata": {}, + "source": "segments_map = []\nstream = export_task.get_buffered_stream()\nfor output in stream:\n output_json = output.json\n labels = output_json[\"projects\"][VIDEO_PROJECT_ID][\"labels\"]\n for label in labels:\n annotations = label[\"annotations\"][\"segments\"]\n for key, value in annotations.items():\n segments_map.append({key: value})\n\nprint(\"----- Segments Feature Map -----\")\npp.pprint(segments_map)", "cell_type": "code", "outputs": [], - "execution_count": null, - "id": "klORakI52lAe" + "execution_count": null } ] -} +} \ No newline at end of file