Skip to content

[SN-0] Added methods to modify model_configs #1647

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 3, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 69 additions & 3 deletions examples/project_configuration/model_chat_evaluation_project.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
},
{
"metadata": {},
"source": "%pip install -q \"labelbox[data]\"",
"source": "%pip install -q --upgrade \"labelbox[data]\"",
"cell_type": "code",
"outputs": [],
"execution_count": null
Expand Down Expand Up @@ -134,8 +134,6 @@
"### Set Up Model Chat Evaluation Project\n",
"You do not have to create data rows with a model evaluation project; instead, they are generated for you when you create the project. The method you use to create your project is `client.create_model_evaluation_project`, which takes the same parameters as the traditional `client.create_project` but with a few specific additional parameters. \n",
"\n",
"__Setting up different models that you are evaluating can only be done inside the Labelbox platform__\n",
"\n",
"#### Parameters\n",
"When using `client.create_model_evaluation_project` the following parameters are needed:\n",
"\n",
Expand All @@ -162,6 +160,74 @@
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"## Setting Up Model Configs\n",
"You can create, delete, attach and remove model configs from your Model Chat Evaluation project through the Labelbox-Python SDK. These are the model configs that you will be evaluating for your responses. "
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
"### Creating Model Configs\n",
"The main method associated with creating a model config is `client.create_model_config`. This method takes the following parameters:\n",
"\n",
"- `name`: Name of the model config.\n",
"\n",
"- `model_id`: The ID of the model to configure. You must obtain this through the UI by navigating to the Model tab, selecting the model you are trying to use, and copying the id inside the URL. For supported models, visit the [Model chat evaluation page](https://docs.labelbox.com/docs/model-chat-evaluation#supported-annotation-types).\n",
"\n",
"- `inference_params`: JSON of model configuration parameters. This will vary depending on the model you are trying to set up. It is recommended to first set up a model config inside the UI to learn all the associated parameters.\n",
"\n",
"For the example below, we will be setting up a Google Gemini 1.5 Pro model config."
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": "MODEL_ID = \"270a24ba-b983-40d6-9a1f-98a1bbc2fb65\"\n\ninference_params = {\"max_new_tokens\": 1024, \"use_attachments\": True}\n\nmodel_config = client.create_model_config(\n name=\"Example Model Config\",\n model_id=MODEL_ID,\n inference_params=inference_params,\n)",
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"### Attaching Model Config to Project\n",
"You can attach and remove model configs to your project using `project.add_model_config` or `project.remove_model_config`. Both methods take just a `model_config` ID."
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": "project.add_model_config(model_config.uid)",
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"### Delete Model Config\n",
"You can also delete model configs using the `client.delete_model_config`. You just need to pass in the `model_config` ID in order to delete your model config. You can obtain this ID from your created model config above or get the model configs directly from your project using `project.project_model_configs` and then iterating through the list of model configs attached to your project. Uncomment the code below to delete your model configs. "
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": "# model_configs = project.project_model_configs()\n\n# for model_config in model_configs:\n# client.delete_model_config(model_config.uid)",
"cell_type": "code",
"outputs": [],
"execution_count": null
},
{
"metadata": {},
"source": [
"**To finish setting up your MCE project, you will need to navigate to your project overview inside the Labelbox platform and select _Complete setup_ on the left side panel**"
],
"cell_type": "markdown"
},
{
"metadata": {},
"source": [
Expand Down