Clarification needed on RACE evaluation metric results

[alcholiclg]

Description

First of all, thank you for your great work on this benchmark. I encountered some questions regarding the evaluation metrics in the repository.

Steps to Reproduce

1. Review the definition of the RACE metric, which calculates:
   res(target) / ( res(target) + res(reference) )
2. Compare the expected value range with the values shown on:

• GitHub homepage
• Hugging Face leaderboard
• The paper

3. Check the example results in:

• claude-3-7-sonnet-latest/race_result.txt

Expected Behavior

• The metric result should normally fall between 0 and 1 based on its definition.
• The file results should be consistent with those reported on the GitHub homepage, Hugging Face leaderboard, and the paper.

Actual Behavior

• GitHub homepage, Hugging Face leaderboard, and the paper all show values ranging from 0 to 100.
• The race_result.txt file results cannot be matched with any of these sources.

Questions

• Are the reported values on the homepage/leaderboard/paper simply multiplied by 100 compared to the raw 0–1 metric values?
• Which set of results should be considered the correct reference: the ones in the repository file, the leaderboard, or the paper? Even after reviewing issue #7, I still cannot reach a clear conclusion.

[Ayanami0730]

Thank you for the detailed issue and for pointing out this inconsistency. Your confusion is completely understandable, and we appreciate the opportunity to clarify.

To answer your questions directly:

To Q1: Yes, you are correct. The scores reported on the leaderboard and in the paper are the raw 0–1 metric values multiplied by 100 for better readability.

To Q2: The Hugging Face Leaderboard should be considered the single source of truth for all official results and rankings. The results in our paper are from several months ago and primarily serve to describe the evaluation framework itself. Since then, the backbone judge model has been updated to the more stable, official gemini-2.5-pro. The leaderboard is the only place where we consistently track the latest results.

We apologize for the lack of detailed explanation in our previous updates. To prevent this confusion in the future, we will be making the following changes:

• We are preparing an update soon with results from many new agents submitted by various teams and companies.
• In this next update, we will remove the static result files from the GitHub repository to ensure that everyone focuses only on the leaderboard for the canonical scores.
• We will add a clear note stating that the leaderboard contains the most up-to-date results, while the paper's results are a snapshot from its publication date.

Thanks again for your help in improving the clarity of our benchmark

[alcholiclg]

Thank you for your timely and detailed explanation. My question regarding the inconsistency between the metrics and evaluation results has been resolved.