The download of the dataset failed when evaluating the VLM #868
Description
Self-Check List
Before submitting an issue, please ensure you have completed the following steps:
- I have carefully read the relevant user documentation
- I have reviewed the Frequently Asked Questions
- I have searched and reviewed existing issues to confirm this is not a duplicate problem
Problem Description
Please briefly describe the problem you encountered.
The evaluation code reported an error due to the failure of the dataset download
EvalScope Version (Required)
v1.0.2
Tools Used
- Native / Native framework
- Opencompass backend
- VLMEvalKit backend
- RAGEval backend
- Perf / Model inference stress testing tool
- Arena / Arena mode
Executed Code or Instructions
Please provide the main code or instructions you executed. reference web
from evalscope import TaskConfig, run_task
task_cfg_dict = TaskConfig(
work_dir='outputs',
eval_backend='VLMEvalKit',
eval_config={
'data': ['BLINK'],
'mode': 'all',
'model': [
{'api_base': 'http://xx.xx.xx.xx:xxx/v1/chat/completions',
'key': 'HikLLM!@#',
'name': 'Qwen2-VL-7B-Instruct',
'temperature': 0.6,
'type': 'qwen2.5-vl-7b-instruct',
'img_size': -1,
'video_llm': False,
'max_tokens': 2048,}
],
'reuse': False,
'nproc': 16,
'judge': 'exact_matching'},
)
run_task(task_cfg=task_cfg_dict)Error Log
Please paste the complete error log or console output.
[2025-10-13 19:16:58,728] WARNING - RUN - run.py: run_task - 168: --reuse is not set, will not reuse previous (before one day) temporary files
2025-10-13 19:16:58,728 - RUN - WARNING - --reuse is not set, will not reuse previous (before one day) temporary files
fatal: Not a git repository (or any parent up to mount point /data)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
/usr/python/lib/python3.11/site-packages/vlmeval/dataset/image_base.py:93: UserWarning: The dataset tsv is not downloaded
warnings.warn('The dataset tsv is not downloaded')
BLINK.tsv: 0.00B [00:00, ?B/s]
2025-10-13 19:16:59,138 - root - WARNING - <class 'urllib.error.URLError'>: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)>
[2025-10-13 19:16:59,138] ERROR - RUN - run.py: run_task - 420: Model Qwen2-VL-7B-Instruct x Dataset BLINK combination failed: Failed to download https://opencompass.openxlab.space/utils/VLMEval/BLINK.tsv, skipping this combination.
Traceback (most recent call last):
File "/usr/python/lib/python3.11/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/usr/python/lib/python3.11/http/client.py", line 1303, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/python/lib/python3.11/http/client.py", line 1349, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.11/http/client.py", line 1298, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/python/lib/python3.11/http/client.py", line 1058, in _send_output
self.send(msg)
File "/usr/python/lib/python3.11/http/client.py", line 996, in send
self.connect()
File "/usr/python/lib/python3.11/http/client.py", line 1475, in connect
self.sock = self._context.wrap_socket(self.sock,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.11/ssl.py", line 517, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.11/ssl.py", line 1104, in _create
self.do_handshake()
File "/usr/python/lib/python3.11/ssl.py", line 1382, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/python/lib/python3.11/site-packages/vlmeval/smp/file.py", line 209, in download_file
urllib.request.urlretrieve(url, filename=filename, reporthook=t.update_to)
File "/usr/python/lib/python3.11/urllib/request.py", line 241, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.11/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.11/urllib/request.py", line 519, in open
response = self._open(req, data)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.11/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/usr/python/lib/python3.11/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.11/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)>Running Environment
- Operating System: Centos7.9
- Python Version: python 3.11
Additional Information
If there is any other relevant information, please provide it here.
In fact, I was testing Qwen2.5-VL-7B-Instruct, but due to the model name assertion, I had to use the names in the list, which might lead to mistakes when recording.
AssertionError: Invalid models: ['Qwen2.5-VL-7B-Instruct'], refer to the following list to get proper model name: ['TransCore_M', 'PandaGPT_13B', 'flamingov2', 'VisualGLM_6b', 'mPLUG-Owl2', 'mPLUG-Owl3', 'OmniLMM_12B', 'MGM_7B', 'Bunny-llama3-8B', 'VXVERSE', 'paligemma-3b-mix-448', '360VL-70B', 'Llama-3-MixSenseV1_1', 'Parrot', 'OmChat', 'RBDash_72b', 'Pixtral-12B', 'Falcon2-VLM-11B', 'GPT4V', 'GPT4V_HIGH', 'GPT4V_20240409', 'GPT4V_20240409_HIGH', 'GPT4o', 'GPT4o_HIGH', 'GPT4o_20240806', 'GPT4o_20241120', 'GPT4o_MINI', 'GeminiPro1-0', 'GeminiPro1-5', 'GeminiFlash1-5', 'GeminiFlash2-0', 'GeminiPro1-5-002', 'GeminiFlash1-5-002', 'QwenVLPlus', 'QwenVLMax', 'RekaEdge', 'RekaFlash', 'RekaCore', 'Step1V', 'Step1.5V-mini', 'Yi-Vision', 'Claude3V_Opus', 'Claude3V_Sonnet', 'Claude3V_Haiku', 'Claude3-5V_Sonnet', 'Claude3-5V_Sonnet_20241022', 'GLM4V', 'GLM4V_PLUS', 'GLM4V_PLUS_20250111', 'abab6.5s', 'abab7-preview', 'CloudWalk', 'SenseChat-Vision', 'HunYuan-Vision', 'CustomAPIModel', 'HunYuan-Standard-Vision', 'BailingMM-Lite-1203', 'BailingMM-Pro-0120', 'BlueLM_V', 'JTVL', 'Taiyi', 'TeleMM', 'lmdeploy', 'lmdeploy_internvl_78B_MPO', 'lmdeploy_qvq_72B_preview', 'Taichu-VL-2B', 'DoubaoVL', 'llava-internlm2-7b', 'llava-internlm2-20b', 'llava-internlm-7b', 'llava-v1.5-7b-xtuner', 'llava-v1.5-13b-xtuner', 'llava-llama-3-8b', 'qwen_base', 'qwen_chat', 'qwen-vl-chat', 'monkey', 'monkey-chat', 'minimonkey', 'llava_v1.5_7b', 'llava_v1.5_13b', 'llava_v1_7b', 'sharegpt4v_7b', 'sharegpt4v_13b', 'llava_next_vicuna_7b', 'llava_next_vicuna_13b', 'llava_next_mistral_7b', 'llava_next_yi_34b', 'llava_next_llama3', 'llava_next_72b', 'llava_next_110b', 'llava_next_qwen_32b', 'llava_next_interleave_7b', 'llava_next_interleave_7b_dpo', 'llava-onevision-qwen2-0.5b-ov-hf', 'llava-onevision-qwen2-0.5b-si-hf', 'llava-onevision-qwen2-7b-ov-hf', 'llava-onevision-qwen2-7b-si-hf', 'llava_onevision_qwen2_0.5b_si', 'llava_onevision_qwen2_7b_si', 'llava_onevision_qwen2_72b_si', 'llava_onevision_qwen2_0.5b_ov', 'llava_onevision_qwen2_7b_ov', 'llava_onevision_qwen2_72b_ov', 'Aquila-VL-2B', 'llava_video_qwen2_7b', 'llava_video_qwen2_72b', 'varco-vision-hf', 'InternVL-Chat-V1-1', 'InternVL-Chat-V1-2', 'InternVL-Chat-V1-2-Plus', 'InternVL-Chat-V1-5', 'Mini-InternVL-Chat-2B-V1-5', 'Mini-InternVL-Chat-4B-V1-5', 'InternVL2-1B', 'InternVL2-2B', 'InternVL2-4B', 'InternVL2-8B', 'InternVL2-26B', 'InternVL2-40B', 'InternVL2-76B', 'InternVL2-8B-MPO', 'InternVL2-8B-MPO-CoT', 'InternVL2_5-1B', 'InternVL2_5-2B', 'InternVL2_5-4B', 'InternVL2_5-8B', 'InternVL2_5-26B', 'InternVL2_5-38B', 'InternVL2_5-78B', 'InternVL2_5-1B-MPO', 'InternVL2_5-2B-MPO', 'InternVL2_5-4B-MPO', 'InternVL2_5-8B-MPO', 'InternVL2_5-26B-MPO', 'InternVL2_5-38B-MPO', 'InternVL2_5-78B-MPO', 'Yi_VL_6B', 'Yi_VL_34B', 'XComposer', 'sharecaptioner', 'XComposer2', 'XComposer2_1.8b', 'XComposer2_4KHD', 'XComposer2d5', 'MiniGPT-4-v2', 'MiniGPT-4-v1-7B', 'MiniGPT-4-v1-13B', 'idefics_9b_instruct', 'idefics_80b_instruct', 'idefics2_8b', 'Idefics3-8B-Llama3', 'instructblip_7b', 'instructblip_13b', 'deepseek_vl_7b', 'deepseek_vl_1.3b', 'deepseek_vl2_tiny', 'deepseek_vl2_small', 'deepseek_vl2', 'Janus-1.3B', 'MiniCPM-V', 'MiniCPM-V-2', 'MiniCPM-Llama3-V-2_5', 'MiniCPM-V-2_6', 'MiniCPM-o-2_6', 'cogvlm-grounding-generalist', 'cogvlm-chat', 'cogvlm2-llama3-chat-19B', 'glm-4v-9b', 'WeMM', 'cambrian_8b', 'cambrian_13b', 'cambrian_34b', 'chameleon_7b', 'chameleon_30b', 'Video-LLaVA-7B', 'Video-LLaVA-7B-HF', 'VideoChat2-HD', 'Chat-UniVi-7B', 'Chat-UniVi-7B-v1.5', 'LLaMA-VID-7B', 'Video-ChatGPT', 'PLLaVA-7B', 'PLLaVA-13B', 'PLLaVA-34B', 'Ovis1.5-Llama3-8B', 'Ovis1.5-Gemma2-9B', 'Ovis1.6-Gemma2-9B', 'Ovis1.6-Llama3.2-3B', 'Ovis1.6-Gemma2-27B', 'VILA1.5-3b', 'Llama-3-VILA1.5-8b', 'VILA1.5-13b', 'VILA1.5-40b', 'Mantis-8B-siglip-llama3', 'Mantis-8B-clip-llama3', 'Mantis-8B-Idefics2', 'Mantis-8B-Fuyu', 'MMAlaya', 'MMAlaya2', 'Phi-3-Vision', 'Phi-3.5-Vision', 'xgen-mm-phi3-interleave-r-v1.5', 'xgen-mm-phi3-dpo-r-v1.5', 'Qwen-VL-Max-0809', 'Qwen-VL-Plus-0809', 'QVQ-72B-Preview', 'Qwen2-VL-72B-Instruct', 'Qwen2-VL-7B-Instruct', 'Qwen2-VL-7B-Instruct-AWQ', 'Qwen2-VL-7B-Instruct-GPTQ-Int4', 'Qwen2-VL-7B-Instruct-GPTQ-Int8', 'Qwen2-VL-2B-Instruct', 'Qwen2-VL-2B-Instruct-AWQ', 'Qwen2-VL-2B-Instruct-GPTQ-Int4', 'Qwen2-VL-2B-Instruct-GPTQ-Int8', 'XinYuan-VL-2B-Instruct', 'Slime-7B', 'Slime-8B', 'Slime-13B', 'Eagle-X4-8B-Plus', 'Eagle-X4-13B-Plus', 'Eagle-X5-7B', 'Eagle-X5-13B', 'Eagle-X5-13B-Chat', 'Eagle-X5-34B-Chat', 'Eagle-X5-34B-Plus', 'Moondream1', 'Moondream2', 'Llama-3.2-11B-Vision-Instruct', 'LLaVA-CoT', 'Llama-3.2-90B-Vision-Instruct', 'molmoE-1B-0924', 'molmo-7B-D-0924', 'molmo-7B-O-0924', 'molmo-72B-0924', 'Kosmos2', 'POINTS-Yi-1.5-9B-Chat', 'POINTS-Qwen-2.5-7B-Chat', 'POINTSV15-Qwen-2.5-7B-Chat', 'NVLM', 'Vintern-3B-beta', 'Vintern-1B-v2', 'h2ovl-mississippi-2b', 'h2ovl-mississippi-1b', 'Aria', 'SmolVLM', 'SmolVLM-DPO', 'SmolVLM-Synthetic', 'SAIL-VL-2B', 'valley_eagle', 'vita', 'vita_qwen2', 'ross-qwen2-7b', 'emu2_chat', 'emu3_chat', 'emu3_gen']