Judge create endpoint #42
Conversation
WalkthroughA new example script demonstrates creating and running a custom judge using the Root Signals API. The OpenAPI specification is updated to clarify judge creation, increase string length limits, and add a required Changes
Sequence Diagram(s)sequenceDiagram
participant User Script
participant JudgesClass
participant RootSignalsAPI
User Script->>JudgesClass: create(name, intent, evaluator_references)
JudgesClass->>RootSignalsAPI: POST /v1/judges/ (with name, intent, evaluators)
RootSignalsAPI-->>JudgesClass: Judge object
JudgesClass-->>User Script: Judge instance
User Script->>JudgesClass: run(judge, request, response, context)
JudgesClass->>RootSignalsAPI: Evaluate judge (with input data)
RootSignalsAPI-->>JudgesClass: Evaluation result
JudgesClass-->>User Script: Result output
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (2)
✅ Files skipped from review due to trivial changes (2)
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
examples/create_judge.py (2)
1-1: Add module docstring
Include a brief module-level docstring summarizing the purpose of this example script.🧰 Tools
🪛 Pylint (3.3.7)
[convention] 1-1: Missing module docstring
(C0114)
[error] 1-1: Unable to import 'root'
(E0401)
22-26: Refactor long string literals
Lines 22–23 and 25 exceed the recommended line length. Consider using triple-quoted strings or concatenation with parentheses to improve readability.🧰 Tools
🪛 Pylint (3.3.7)
[convention] 25-25: Line too long (137/100)
(C0301)
src/root/judges.py (1)
308-339: Fix line length formatting issue.The asynchronous create method implementation is functionally correct and follows the established async patterns. However, there's a formatting issue that needs to be addressed.
Apply this diff to fix the line length issue:
- return await AJudge._awrap( - await api_instance.judges_create(judge_request=request, _request_timeout=_request_timeout), - client_context=self.client_context, - ) + result = await api_instance.judges_create( + judge_request=request, _request_timeout=_request_timeout + ) + return await AJudge._awrap(result, client_context=self.client_context)🧰 Tools
🪛 Pylint (3.3.7)
[convention] 337-337: Line too long (103/100)
(C0301)
[warning] 336-336: Access to a protected member _awrap of a client class
(W0212)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (22)
src/root/generated/openapi_aclient/api/judges_api.pyis excluded by!**/generated/**src/root/generated/openapi_aclient/models/evaluator.pyis excluded by!**/generated/**src/root/generated/openapi_aclient/models/evaluator_execution_request.pyis excluded by!**/generated/**src/root/generated/openapi_aclient/models/evaluator_list_output.pyis excluded by!**/generated/**src/root/generated/openapi_aclient/models/judge_execution_request.pyis excluded by!**/generated/**src/root/generated/openapi_aclient/models/judge_generator_response.pyis excluded by!**/generated/**src/root/generated/openapi_aclient/models/judge_rectifier_request_request.pyis excluded by!**/generated/**src/root/generated/openapi_aclient/models/judge_request.pyis excluded by!**/generated/**src/root/generated/openapi_aclient/models/objective.pyis excluded by!**/generated/**src/root/generated/openapi_aclient/models/objective_list.pyis excluded by!**/generated/**src/root/generated/openapi_aclient/models/patched_judge_request.pyis excluded by!**/generated/**src/root/generated/openapi_client/api/judges_api.pyis excluded by!**/generated/**src/root/generated/openapi_client/models/evaluator.pyis excluded by!**/generated/**src/root/generated/openapi_client/models/evaluator_execution_request.pyis excluded by!**/generated/**src/root/generated/openapi_client/models/evaluator_list_output.pyis excluded by!**/generated/**src/root/generated/openapi_client/models/judge_execution_request.pyis excluded by!**/generated/**src/root/generated/openapi_client/models/judge_generator_response.pyis excluded by!**/generated/**src/root/generated/openapi_client/models/judge_rectifier_request_request.pyis excluded by!**/generated/**src/root/generated/openapi_client/models/judge_request.pyis excluded by!**/generated/**src/root/generated/openapi_client/models/objective.pyis excluded by!**/generated/**src/root/generated/openapi_client/models/objective_list.pyis excluded by!**/generated/**src/root/generated/openapi_client/models/patched_judge_request.pyis excluded by!**/generated/**
📒 Files selected for processing (3)
examples/create_judge.py(1 hunks)openapi.yaml(8 hunks)src/root/judges.py(2 hunks)
🧰 Additional context used
🪛 Pylint (3.3.7)
examples/create_judge.py
[convention] 2-2: Line too long (102/100)
(C0301)
[convention] 25-25: Line too long (137/100)
(C0301)
[convention] 1-1: Missing module docstring
(C0114)
[error] 1-1: Unable to import 'root'
(E0401)
[error] 2-2: Unable to import 'root.generated.openapi_client.models.evaluator_reference_request'
(E0401)
[error] 3-3: Unable to import 'root.skills'
(E0401)
src/root/judges.py
[convention] 337-337: Line too long (103/100)
(C0301)
[warning] 303-303: Access to a protected member _wrap of a client class
(W0212)
[warning] 336-336: Access to a protected member _awrap of a client class
(W0212)
🔇 Additional comments (12)
openapi.yaml (7)
947-947: Clarify POST/v1/judges/description
Updating the operation description to includeintentandevaluator_referencesaligns with the new requirements. Please ensure downstream documentation and client method docstrings are updated accordingly.
2167-2167: ExtendEvaluatorExecutionRequest.expected_outputlength
IncreasingmaxLengthto 3.5M supports larger evaluation contexts; this change is consistent across execution schemas.
2858-2858: ExtendJudgeExecutionRequest.expected_outputlength
Aligns with other execution requests.
3015-3015: ExtendJudgeRectifierRequestRequest.expected_outputlength
Consistent adjustment for rectifier requests.
2916-2919: Allow nullableerror_codeinJudgeGeneratorResponse
Makingerror_codenullable and removing it fromrequiredavoids schema validation errors when no error occurs.
3054-3059: Add requiredintentfield toJudgeRequestschema
Introducing the write-onlyintentproperty and marking it as required enforces capturing judge intent at creation time. Confirm that all client bindings and example usages pass this field correctly.Also applies to: 3067-3067
3683-3689: IncludeintentinPatchedJudgeRequest
Addingintentto the partial update schema is appropriate. Verify that patch handlers accept and apply intent updates without breaking existing workflows.examples/create_judge.py (3)
1-3: Verify import paths
Static analysis flags import errors forroot,root.generated, androot.skills. Ensure the package is installed in the environment and that these paths are correct according to the project’s packaging layout.🧰 Tools
🪛 Pylint (3.3.7)
[convention] 2-2: Line too long (102/100)
(C0301)
[convention] 1-1: Missing module docstring
(C0114)
[error] 1-1: Unable to import 'root'
(E0401)
[error] 2-2: Unable to import 'root.generated.openapi_client.models.evaluator_reference_request'
(E0401)
[error] 3-3: Unable to import 'root.skills'
(E0401)
13-17: Usecreatemethod for judge creation
Theclient.judges.create(...)invocation correctly passesname,intent, andevaluator_references.
19-27: Confirmrunmethod existence
The example usesclient.judges.run(...)to execute the judge. Verify that the client library exposes arunmethod (or adjust toexecute/execute_create) that maps to the/judges/{id}/execute/endpoint.🧰 Tools
🪛 Pylint (3.3.7)
[convention] 25-25: Line too long (137/100)
(C0301)
src/root/judges.py (2)
15-17: LGTM! Import additions align with new functionality.The new imports for
JudgeRequestandAJudgeRequestare correctly added to support the new judge creation methods.Also applies to: 21-21
275-306: LGTM! Synchronous create method implementation is solid.The method follows the established patterns in the codebase with proper:
- Type hints and parameter documentation
- Request object construction
- API client usage
- Response wrapping with client context
The protected member access warning from static analysis is a false positive -
_wrapis a class method being called appropriately within the same module.🧰 Tools
🪛 Pylint (3.3.7)
[warning] 303-303: Access to a protected member _wrap of a client class
(W0212)
Summary by CodeRabbit
New Features
Documentation