Vb/model offline eval plt 1107 #1682
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vbrodsky
force-pushed
the
VB/model-offline-eval_PLT-1107
branch
from
feb64c7 to
4f90d2a
Compare
vbrodsky
force-pushed
the
VB/model-offline-eval_PLT-1107
branch
from
4f90d2a to
115b78d
Compare
libs/labelbox/src/labelbox/client.py
Outdated
| return self._create(Entity.Project, params, extra_params) | ||
|
|
||
|
|
||
| # **** Create Project end **** |
There was a problem hiding this comment.
Remove this line + extra whitespace
libs/labelbox/src/labelbox/client.py
Outdated
| return self._create_project(**kwargs) | ||
|
|
||
| def create_offline_model_evaluation_project(self, **kwargs) -> Project: | ||
| kwargs["media_type"] = MediaType.Conversational |
There was a problem hiding this comment.
We should type these arguments and add a docstring
There was a problem hiding this comment.
Re: typed arguments, see my proposal above #1682 (comment)
I will add a docstring (forgot to add it). Since these arguments are not user-facing, I propose to clarify them as code comments
There was a problem hiding this comment.
is create_offline_model_evaluation_project not user facing? it's part of the client right?
There was a problem hiding this comment.
it is user facing, but the user is not expected to pass in / set those args
Docstring updates
vbrodsky
pushed a commit
that referenced
this pull request
Jun 20, 2024
adrian-chang
pushed a commit
that referenced
this pull request
Jun 21, 2024
vbrodsky
pushed a commit
that referenced
this pull request
Jun 24, 2024
vbrodsky
added a commit
that referenced
this pull request
Jun 24, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Story: SDK support for offline model evaluation use cases
This story adds support for creation of offline chat model evaluation projects and verifies the following offline and live chat evalutation properties using sdk integration tests.
For an offline project:
For a live chat model evaluation project we now verify that
I refactored create project calls to make sure users will not, inadvertently, cause an issue by supplying incorrect parameters or missing important parameters.
_create_project()methodAdditionally I have updated handling of EditorTaskType. In a previous release, a new editor task typed cause client get_project() method to break. Now we will set editor task type to
Missing=NoneType of change
Please delete options that are not relevant.
All Submissions
New Feature Submissions
Does your submission pass tests?
There failing integration tests not related to my code
Have you added thorough tests for your new feature?
Have you commented your code, particularly in hard-to-understand areas?
Have you added a Docstring?
Changes to Core Features