v0.3.4

What's Changed

• Support VSI-Bench Evaluation by @vealocia in #511
• [Fix] Better Qwen omni and linting by @kcz358 in #647
• Fix the bug in issue #648 by @ashun989 in #649
• [New Model] Aero-1-Audio by @kcz358 in #658
• [improve]: catch import error; remove unused modules by @VincentYCYao in #650
• [Fix] fixing the video path of MVBench & adding default hf_home to percepti… by @jihanyang in #655
• Update vllm.py by @VincentYCYao in #652
• [FIX]: Fix question_for_eval key in MathVerse evaluator for Vision-Only data by @ForJadeForest in #657
• [Task] Add new benchmark: CAPability by @lntzm in #656
• Mathvision bug fixes , Reproduce Qwen2.5VL results by @RadhaGulhane13 in #660
• Fix issue with killing process in sglang by @ravi03071991 in #666
• Fixes Metadata Reading from Released PLM Checkpoints by @mmaaz60 in #665
• [fix] modify the GPT evaluation model by @jujeongho0 in #668
• [Fix] Correct rating logic for VITATECS benchmark by @erfanbsoula in #671
• Update README.md by @pufanyi in #675
• delete unused test_parse.py file by @pbcong in #676
• [fix] add reminder for interleave_visual for Qwen2.5-VL, update version control. by @Luodian in #678
• [fix] Fix task listing in CLI evaluation by updating to use 'all_tasks' instead of 'list_all_tasks' for improved clarity. by @Luodian in #687
• [Task] V*-Bench (Visual Star Benchmark) by @Luodian in #683
• support distributed executor backend - torchrun by @kaiyuyue in #680
• [Task] Add new task: XLRS-Bench-lite by @nanocm in #684
• Added direction for locally cached dataset in task_guide.md by @JulyanZhu in #691
• Pass in the 'cache_dir' to use local cache by @JulyanZhu in #690
• [FIX] Fix parameter name in qwen25vl.sh by @MasterBeeee in #693
• [TASK & FIX] add task VideoEval-Pro and fix tar file concat by @iamtonymwt in #694

New Contributors

• @vealocia made their first contribution in #511
• @ashun989 made their first contribution in #649
• @VincentYCYao made their first contribution in #650
• @jihanyang made their first contribution in #655
• @ForJadeForest made their first contribution in #657
• @lntzm made their first contribution in #656
• @RadhaGulhane13 made their first contribution in #660
• @ravi03071991 made their first contribution in #666
• @erfanbsoula made their first contribution in #671
• @kaiyuyue made their first contribution in #680
• @nanocm made their first contribution in #684
• @JulyanZhu made their first contribution in #691
• @MasterBeeee made their first contribution in #693
• @iamtonymwt made their first contribution in #694

Full Changelog: v0.3.3...v0.3.4