-
Notifications
You must be signed in to change notification settings - Fork 68
[PLT-1207] Added prompt classification for python object support #1700
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
3259998
added prompt classification for python object support
Gabefire fa3a612
improved test
Gabefire 98e8554
added prompt text
Gabefire f0ffc0d
typo
Gabefire 6cc96c3
removed bulk import test this is legacy feature and not needed
Gabefire cbc7f39
typo
Gabefire e3e1773
added newline back
Gabefire 9260262
Update pyproject.toml
Gabefire afdfd2a
Merge branch 'develop' into gu/prompt_response_support
Gabefire 17e4be5
made small fix
Gabefire 050ab8e
fix test
Gabefire 10adf7a
added better tests and removed old tests
Gabefire e07d19f
changed pyproject back
Gabefire 70ed7b0
fixed/ improved a few tests
Gabefire 9c8a286
Update pyproject.toml
Gabefire 17496c8
Update bulk_import_request.py
Gabefire 73c22db
Merge branch 'develop' into gu/prompt_response_support
Gabefire 619fd3e
feedback
Gabefire File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 2 additions & 0 deletions
2
libs/labelbox/src/labelbox/data/annotation_types/llm_prompt_response/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
from .prompt import PromptText | ||
from .prompt import PromptClassificationAnnotation |
39 changes: 39 additions & 0 deletions
39
libs/labelbox/src/labelbox/data/annotation_types/llm_prompt_response/prompt.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
from typing import Union | ||
|
||
from labelbox.data.annotation_types.base_annotation import BaseAnnotation | ||
|
||
from labelbox.data.mixins import ConfidenceMixin, CustomMetricsMixin | ||
|
||
from labelbox import pydantic_compat | ||
|
||
|
||
class PromptText(ConfidenceMixin, CustomMetricsMixin, pydantic_compat.BaseModel): | ||
""" Prompt text for LLM data generation | ||
|
||
>>> PromptText(answer = "some text answer", | ||
>>> confidence = 0.5, | ||
>>> custom_metrics = [ | ||
>>> { | ||
>>> "name": "iou", | ||
>>> "value": 0.1 | ||
>>> }]) | ||
""" | ||
answer: str | ||
|
||
|
||
class PromptClassificationAnnotation(BaseAnnotation, ConfidenceMixin, | ||
CustomMetricsMixin): | ||
"""Prompt annotation (non localized) | ||
|
||
>>> PromptClassificationAnnotation( | ||
>>> value=PromptText(answer="my caption message"), | ||
>>> feature_schema_id="my-feature-schema-id" | ||
>>> ) | ||
|
||
Args: | ||
name (Optional[str]) | ||
feature_schema_id (Optional[Cuid]) | ||
value (Union[Text]) | ||
""" | ||
|
||
value: PromptText |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
57 changes: 57 additions & 0 deletions
57
libs/labelbox/tests/data/serialization/ndjson/test_data_gen.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,57 @@ | ||
from copy import copy | ||
import pytest | ||
import labelbox.types as lb_types | ||
from labelbox.data.serialization import NDJsonConverter | ||
from labelbox.data.serialization.ndjson.objects import NDDicomSegments, NDDicomSegment, NDDicomLine | ||
""" | ||
Data gen prompt test data | ||
""" | ||
|
||
prompt_text_annotation = lb_types.PromptClassificationAnnotation( | ||
feature_schema_id="ckrb1sfkn099c0y910wbo0p1a", | ||
name="test", | ||
value=lb_types.PromptText(answer="the answer to the text questions right here"), | ||
) | ||
|
||
prompt_text_ndjson = { | ||
"answer": "the answer to the text questions right here", | ||
"name": "test", | ||
"schemaId": "ckrb1sfkn099c0y910wbo0p1a", | ||
"dataRow": { | ||
"id": "ckrb1sf1i1g7i0ybcdc6oc8ct" | ||
}, | ||
} | ||
|
||
data_gen_label = lb_types.Label( | ||
data={"uid": "ckrb1sf1i1g7i0ybcdc6oc8ct"}, | ||
annotations=[prompt_text_annotation] | ||
) | ||
|
||
""" | ||
Prompt annotation test | ||
""" | ||
|
||
def test_serialize_label(): | ||
serialized_label = next(NDJsonConverter().serialize([data_gen_label])) | ||
# Remove uuid field since this is a random value that can not be specified also meant for relationships | ||
del serialized_label["uuid"] | ||
assert serialized_label == prompt_text_ndjson | ||
|
||
|
||
def test_deserialize_label(): | ||
deserialized_label = next(NDJsonConverter().deserialize([prompt_text_ndjson])) | ||
if hasattr(deserialized_label.annotations[0], 'extra'): | ||
# Extra fields are added to deserialized label by default need removed to match | ||
deserialized_label.annotations[0].extra = {} | ||
assert deserialized_label.annotations == data_gen_label.annotations | ||
|
||
|
||
def test_serialize_deserialize_label(): | ||
serialized = list(NDJsonConverter.serialize([data_gen_label])) | ||
deserialized = next(NDJsonConverter.deserialize(serialized)) | ||
if hasattr(deserialized.annotations[0], 'extra'): | ||
# Extra fields are added to deserialized label by default need removed to match | ||
deserialized.annotations[0].extra = {} | ||
print(data_gen_label.annotations) | ||
print(deserialized.annotations) | ||
assert deserialized.annotations == data_gen_label.annotations |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.