-
Notifications
You must be signed in to change notification settings - Fork 327
Open
Description
Description:
Currently, leaderboard schemas must be manually modified for a new benchmark to appear in leaderboards. This manual step increases the effort and friction required when adding benchmarks, making the process more error-prone and less scalable.
Proposed Improvement:
A RunSpec should be sufficient to provide all the information needed for a leaderboard schema. If the RunSpec fully specifies the benchmark details, the system should be able to automatically generate or update the corresponding leaderboard entry without requiring manual schema edits.
Benefits:
- Reduces manual intervention when adding benchmarks.
- Ensures consistency between RunSpec definitions and leaderboard schemas.
- Lowers maintenance overhead for leaderboard management.
- Makes it easier for contributors to add new benchmarks.
Metadata
Metadata
Assignees
Labels
No labels