How to mitigate length bias in the benchmark?

[YiLIU001]

Hi @Ayanami0730. Thanks for this great benchmark, which is very helpful for developing search agents.

From what I understand, there are several types of biases in LLM-as-a-judge framework. One of the more prominent ones is the length bias. And in our test, we found a positive correlation between the report length and the RACE score with different variants of implementation. So I am wondering: is this observation really an indicator of the real report quality or it also includes some spurious correlation? Does DeepResearch Bench have a way to decouple the influence of length bias?

Thanks in advance for your reply!

[Ayanami0730]

Hi @YiLIU001. Thank you for raising this question.

You are right to point out that LLM-as-a-judge-based methods have several inherent biases, and we've discussed these extensively in other issues and communications. The existence of biases—including length bias, same-model-family bias, and positional bias—is a well-known consensus in the research community.

However, LLM-as-a-judge remains one of the most reliable and scalable methods for evaluating open-ended content. Our goal is to have more experiments to test whether frontier models are becoming less susceptible to these biases, thereby increasing the trustworthiness of our results. We are also actively exploring metrics that are complementary to LLM-as-a-judge to provide a more objective picture of a DRA article's true quality.

To your specific point about length, we recently conducted some targeted experiments. We programmatically expanded articles from different agents, increasing their length section by section without adding any new information. When we evaluated these longer, less dense articles with the RACE framework, we found that their scores decreased slightly. We plan to release those exp results soon, but they suggest that simply making an article longer is not an easy way to "hack" the current frontier models.

Of course, we also observe a general positive correlation between the RACE score and article length in our overall dataset, which confirms your finding. We believe this is not just a spurious correlation from the judge model, but may also reflect a human cognitive bias. In our human experiments, we found that people also tend to prefer longer articles, likely because they appear more comprehensive and information-rich on the surface. Deeper qualities, such as logical depth or innovativeness, may require different evaluation methods, which we are actively exploring.

Your question about decoupling article length from quality is a great one, and we are trying to address it:

• Current Approach (FACT Framework): We use metrics like citation accuracy and the number of valid cited facts as auxiliary dimensions to measure an agent's capabilities in a way that is less tied to the prose length.

• Future Directions: We are exploring things like adding factuality detection for non-cited claims and calculating an "information density" score, potentially as a ratio of the number of cited facts to the article's total length.

Thank you again for your insightful question. We are actively gathering feedback like this from the community to improve our benchmark and hope to release our related experimental results soon.

[YiLIU001]

Thanks a lot for your reply. The experiments of augmenting report length is really interesting and inspirational. Looking forward to seeing them updated in the publication!

Another question that are somewhat related is the redundancy in the generated responses. For example, a same sentences can be paraphrased and appear several times in the response. Do you think that this should be also taken into account when evaluating the deep research agents, for example by introducing a metric like information density to quantify this phenomenon? Thanks in advance for your reply!