Skip to content

Need Support for Schema-Less Benchmarks #3823

@MiguelAFH

Description

@MiguelAFH

Description:
Currently, leaderboard schemas must be manually modified for a new benchmark to appear in leaderboards. This manual step increases the effort and friction required when adding benchmarks, making the process more error-prone and less scalable.

Proposed Improvement:
A RunSpec should be sufficient to provide all the information needed for a leaderboard schema. If the RunSpec fully specifies the benchmark details, the system should be able to automatically generate or update the corresponding leaderboard entry without requiring manual schema edits.

Benefits:

  • Reduces manual intervention when adding benchmarks.
  • Ensures consistency between RunSpec definitions and leaderboard schemas.
  • Lowers maintenance overhead for leaderboard management.
  • Makes it easier for contributors to add new benchmarks.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions