Skip to content

Issues: EvolvingLMMs-Lab/lmms-eval

Beta
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

217 Open 89 Closed
217 Open 89 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Bug: The images blending within the same batch leads to the performance degradation of Qwen2.5_VL when the batch size increases
#648 opened Apr 22, 2025 by ashun989
How can I become the contributor？
#646 opened Apr 22, 2025 by Hoantrbl
Big results gap for qwenvl2.5-7b
#645 opened Apr 21, 2025 by Blissy-32
2
Inconsistent Output from lmms-eval vs. Custom Inference Script with Identical Inputs
#640 opened Apr 18, 2025 by 1023604540
API calls are Sync
#636 opened Apr 17, 2025 by CLARKBENHAM
Is batch inference enabled now?
#635 opened Apr 15, 2025 by Cola-any
1
How to eval llava by pruned
#634 opened Apr 15, 2025 by Darryl-lilin
Evaluating llava Qwen
#627 opened Apr 11, 2025 by mzamini92
VILA error: 'get_model_name_from_path' is not defined
#626 opened Apr 11, 2025 by YongchengYAO
2
openai_compatible model not complatible with chat/completions endpoint due to max_output_tokens
#625 opened Apr 9, 2025 by dtrawins
about llava-interleave model inconsistent performance
#623 opened Apr 8, 2025 by lllxxzzzzz
2
Can't debug with only one process on one GPU.
#619 opened Apr 7, 2025 by Creator404
about LVbench
#617 opened Apr 5, 2025 by hello-bluedog
1
Bug using local dataset help wanted Extra attention is needed
#612 opened Mar 31, 2025 by ZhangChuanyu1999
1
How to properly call lmms-eval's evaluate function within LLaVA's training loop?
#611 opened Mar 31, 2025 by CiChina
4
llava-interleave-bench
#608 opened Mar 30, 2025 by lllxxzzzzz
internvl 2.5 8b 和qwen2.5vl的结果
#603 opened Mar 27, 2025 by hello-bluedog
6
Way to do model response and post-processing seprately. help wanted Extra attention is needed
#602 opened Mar 27, 2025 by HaozheQi
1
Metadata not implemented for LLaVA Onevision + Loglikelihoods Outputs
#599 opened Mar 27, 2025 by varungupta31
Qwen2_5_vl Eval Error when dealing with image reso less than 28
#598 opened Mar 26, 2025 by FerryHuang
1
[BUG] MLVU evaluation bug, KeyError: 'needle'
#595 opened Mar 25, 2025 by 1023604540
1
Recommended Practices for Video Benchmark Submission + Broken Link
#594 opened Mar 25, 2025 by varungupta31
issues with batch_size (Index error)
#592 opened Mar 22, 2025 by llltttwww
1
Local docvqa dataset loading speed slow
#586 opened Mar 19, 2025 by lynshwoo2022
visualize results
#583 opened Mar 17, 2025 by diankun-wu
ProTip! Add no:assignee to see everything that’s not assigned.