Different judgment in reward server

[kaiyliu]

Hi, thanks for you great work!

My question is about

efficient-reasoning/reward_server/math_server.py

Line 65 in d017399

if tokenizer.eos_token_id not in tokenizer.encode(response, add_special_tokens=False):

and

efficient-reasoning/reward_server/math_server.py

Line 84 in d017399

if tokenizer.eos_token_id not in tokenizer.encode(response, add_special_tokens=False):

In first line, "response" is from "all_responses".
But in the second line, "response" is from "query".

If the response exceeds the maximum generation length, it will be truncated.
In this way, all "response" in "all_responses" does not contain the "<｜end▁of▁sentence｜>".
But second "response" will have "<｜end▁of▁sentence｜>".

The results of the second judgment will be inconsistent.

[kaiyliu]

A naive solution is

query_response = query['response']
query_response_idx = None
for all_response_idx, all_response_info in enumerate(all_responses):
    if all_response_info[:len(query_response) - 50] == query_response[:len(query_response) - 50]:
        query_response_idx = all_response_idx

and the second "response" is all_responses[query_response_idx]