partial data evaluation

[Steven-Xiong]

Hi, I notice that your evauation code requires generation all the 396 videos and do evaluation. Could you help support only evaluate on given list of ids, like evaluate on a small subset? Thank you.

[sam-motamed]

Hi @Steven-Xiong, thanks for the question! We do plan to add a per-scenario view of the results in the future, which will allow users to better understand how models perform across different types of physical reasoning challenges.

However, since the Physics-IQ leaderboard is designed to provide a holistic assessment of model performance on the full benchmark, we don't currently see value in supporting evaluation on only a subset of the data. A partial evaluation would undermine comparability and could misrepresent a model’s overall reasoning ability.

That said, for debugging or internal analysis, you're absolutely welcome to run inference on selected scenarios locally — and we may consider more flexible tools to support that use case later on.

[Steven-Xiong]

But I do think if you can provide code for evaluating specific category data is useful( for instance, Solid Mechanics), so that we can learn the model's performance on certain physics category. Would be appreciated if you can provide such evaluation. I tried to write myself, but find that they always try to match all the videos.