Benchmark History

[FabianKruger]

Hi,

Is it possible to view the submission history or past states of the benchmark leaderboards? I'm planning to reference my model's performance and was wondering if it's sufficient to state the submission date, or if I should use an archiving service (like the Wayback Machine) to capture a snapshot of the leaderboard at that time and include that link in my reference.

Thanks in advance for the clarification!

[cwognum]

Hi @FabianKruger , I'm assuming you're talking about the ASAP competition? Unfortunately, we have no way to reference the previous intermediate leaderboards anymore.

[FabianKruger]

Hi @cwognum , Thank you for your quick response! I wasn’t referring to the competition, but rather to the benchmark leaderboards at https://polarishub.io/benchmarks.
Is there any way to access previous versions or snapshots of those leaderboards, or are they overwritten as new submissions come in?

[cwognum]

Ah, in that case:

We don't maintain old versions of the leaderboard. However, we do show all results that were ever uploaded, even if a user uploaded multiple times. It's an interesting idea to be able to filter by the timestamp of these results. We've long wanted to switch from the leaderboard to more of an evaluation dashboard, see e.g. this blogpost. This feature could fit in nicely.

What's the use case? Is it to have a "timeless" reference you can include in a paper?

[FabianKruger]

Hi @cwognum, thanks for the clarification and the blog post! It looks super nice. A timestamp filter sounds like a really useful and interesting feature.

My use case is that I mention the leaderboard briefly in a paper and wanted a way to reference my model's performance at that time. I’ve used the Wayback Machine to snapshot the page, but was wondering if there’s a more elegant or recommended approach for this kind of citation.