Skip to content

docs: DOC-258: Add new reject setting for reviewers #6521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Oct 16, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 77 additions & 14 deletions docs/source/guide/project_settings_lse.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ Enable **Show before reviewing** to display a pop-up message to reviewers when t

<dd>

Configure additional settings for reviewers.
Configure what is required for a task to be considered reviewed.

<table>
<thead>
Expand Down Expand Up @@ -232,6 +232,80 @@ In a task where multiple annotators submitted labels, the reviewer needs to acce

</td>
</tr>
</table>

</dd>

<dt id="reject-options">Reject Options</dt>

<dd>

Configure what rejection options are available to reviewers.

<table>
<thead>
<tr>
<th>Field</th>
<th>Description</th>
</tr>
</thead>
<tr>
<td>

**Requeue rejected tasks back to annotators**
</td>
<td>

When a reviewer clicks **Reject**, the annotation is reassigned back to the annotator.

</td>
</tr>
<td>

**Remove rejected tasks from labeling queue**
</td>
<td>

When a reviewer clicks **Reject**, the annotation is not reassigned back to the annotator.

</td>
</tr>
<tr>
<td>

**Allow reviewer to choose: Requeue or Remove**
</td>
<td>

Reviewers see the following options:

* **Accept**
* **Reject** -- When selected, the annotation is rejected and skipped.
* **Requeue** -- When selected, the annotation is rejected and then reassigned back to the annotator.

For example, a reviewer might decide to requeue an annotation that is nearly correct but just needs a slight change. However, an annotation with numerous errors may be easier to simply reject entirely and remove from the queue.

Note that when you click **Reject**, the annotation is also marked as cancelled/skipped. This is reflected in various metrics (for example, Data Manager columns and dashboards), and differentiates between the two rejection actions in the API with `was_cancelled: true`.

</td>
</tr>
</table>

</dd>

<dt id="review-settings">Additional settings</dt>

<dd>

Configure additional reviewer settings

<table>
<thead>
<tr>
<th>Field</th>
<th>Description</th>
</tr>
</thead>
<tr>
<td>

Expand Down Expand Up @@ -276,17 +350,6 @@ However, some information is still hidden from reviewers and they can only view
<tr>
<td>

**Requeue rejected tasks back to annotators**
</td>
<td>

Rejected tasks are reassigned back to the annotator.

</td>
</tr>
<tr>
<td>

**Reviewers must leave a comment on reject**
</td>
<td>
Expand Down Expand Up @@ -382,13 +445,13 @@ Select the [metric](stats#Available-agreement-metrics) that should determine tas
</td>
<td>

Note that to see these options, the project must be set up to [automatically distribute tasks](#distribute-tasks).

You can set a low agreement strategy to ensure that a task is not marked complete until it meets 1) the required [overlap](#overlap) and 2) a minimum agreement level.

* **Do nothing** - Tasks with a low agreement can be marked complete; no additional actions are taken.
* **Assign additional annotator** - Automatically assign an additional annotator to tasks with low agreement.

Note that your project must be set up to [automatically distribute tasks](#distribute-tasks).

</td>
</tr>
<tr>
Expand Down