evalscope 1.0版本 对于mmlu这种多个sub_list的评测集，会有一个默认值all，报错，且官网没有具体的使用方法

[jingjingpiggy]

self = <evalscope.benchmarks.mmlu.mmlu_adapter.MMLUAdapter object at 0x7f6ce13b38f0>
load_func = functools.partial(<bound method DefaultDataAdapter.load_subset of <evalscope.benchmarks.mmlu.mmlu_adapter.MMLUAdapter object at 0x7f6ce13b38f0>>, data_loader=<class 'evalscope.api.dataset.loader.RemoteDataLoader'>)
is_fewshot = False
(Pdb) l
198 load_func (Callable[[str], Dataset]): Function to load individual subsets
199
200 Returns:
201 DatasetDict: Dictionary containing all loaded subsets
202 """
203 -> if self.reformat_subset:
204 # Load only the default subset
205 subset_data = load_func(self.default_subset)
206 # Reformat the subset to create multiple subsets based on sample keys
207 # NOTE: subset_list and limit is applied here if specified
208 limit = self.few_shot_num if is_fewshot else self.limit
(Pdb) n

/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py(205)load_subsets()
-> subset_data = load_func(self.default_subset)
(Pdb) self.default_subset
'all'
(Pdb)
'all'
(Pdb) c
2025-10-15 20:21:24 - evalscope - INFO: Loading dataset modelscope/mmlu from modelscope > subset: all > split: test ...
Traceback (most recent call last):
File "/mnt/workspace/E2Eeval/eval_public.py", line 68, in
run_task(task_cfg=task_cfg)
File "/usr/local/lib/python3.12/site-packages/evalscope/run.py", line 28, in run_task
return run_single_task(task_cfg, run_time)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/run.py", line 41, in run_single_task
result = evaluate_model(task_cfg, outputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/run.py", line 144, in evaluate_model
res_dict = evaluator.eval()
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/evaluator/evaluator.py", line 94, in eval
dataset_dict = self.benchmark.load_dataset()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 62, in load_dataset
self.test_dataset, self.fewshot_dataset = self.load()
^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 81, in load
return self.load_from_remote()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 93, in load_from_remote
test_dataset = self.load_subsets(test_load_func)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 205, in load_subsets
subset_data = load_func(self.default_subset)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 252, in load_subset
dataset = loader.load()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/dataset/loader.py", line 88, in load
dataset = MsDataset.load(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/ms_dataset.py", line 308, in load
with load_dataset_with_ctx(
File "/usr/local/lib/python3.12/contextlib.py", line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 1475, in load_dataset_with_ctx
dataset_res = DatasetsWrapperHF.load_dataset(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 991, in load_dataset
builder_instance = DatasetsWrapperHF.load_dataset_builder(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 1164, in load_dataset_builder
builder_instance: DatasetBuilder = builder_cls(
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/datasets/builder.py", line 343, in init
self.config, self.config_id = self._create_builder_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/datasets/builder.py", line 570, in _create_builder_config
raise ValueError(
ValueError: BuilderConfig 'all' not found. Available: ['high_school_european_history', 'business_ethics', 'clinical_knowledge', 'medical_genetics', 'high_school_us_history', 'high_school_physics', 'high_school_world_history', 'virology', 'high_school_microeconomics', 'econometrics', 'college_computer_science', 'high_school_biology', 'abstract_algebra', 'professional_accounting', 'philosophy', 'professional_medicine', 'nutrition', 'global_facts', 'machine_learning', 'security_studies', 'public_relations', 'professional_psychology', 'prehistory', 'anatomy', 'human_sexuality', 'college_medicine', 'high_school_government_and_politics', 'college_chemistry', 'logical_fallacies', 'high_school_geography', 'elementary_mathematics', 'human_aging', 'college_mathematics', 'high_school_psychology', 'formal_logic', 'high_school_statistics', 'international_law', 'high_school_mathematics', 'high_school_computer_science', 'conceptual_physics', 'miscellaneous', 'high_school_chemistry', 'marketing', 'professional_law', 'management', 'college_physics', 'jurisprudence', 'world_religions', 'sociology', 'us_foreign_policy', 'high_school_macroeconomics', 'computer_security', 'moral_scenarios', 'moral_disputes', 'electrical_engineering', 'astronomy', 'college_biology']

真不知道这个all是从哪里来了，为什么官网文档又不能很好维护，怎么用呢。。。

[Yunnglin]

请提供一下完整的评测命令，使用的数据是有all这个子集的：

[jingjingpiggy]

task_cfg = TaskConfig(
# model='Qwen3-32B',
# api_url='http://127.0.0.1:8996/v1/chat/completions',
model=args.model,
api_url='http://127.0.0.1:' + str(args.port) + '/v1/chat/completions',
eval_type=args.eval_type,
datasets=[
f'{args.dataset}'
# 'gsm8k','iquiz', 'aime24','mmlu','super_gpqa', 'bbh'
],
dataset_args={
'aime24': {
'dataset_id': 'HuggingFaceH4/aime_2024',
'filters': {'remove_until': ''} # Filter out the content of thinking
},
'iquiz': {
'dataset_id': 'AI-ModelScope/IQuiz',
'filters': {'remove_until': ''} # Filter out the content of thinking
},
'mmlu': {
'dataset_id': 'modelscope/mmlu',
'filters': {'remove_until': ''},
},
'super_gpqa': {
'dataset_id': 'm-a-p/SuperGPQA',
'filters': {'remove_until': ''} # Filter out the content of thinking
},
'bbh': {
'dataset_id': 'modelscope/bbh',
'filters': {'remove_until': ''} # Filter out the content of thinking
},
'gsm8k': {
'dataset_id': 'modelscope/gsm8k',
'filters': {'remove_until': ''} # Filter out the content of thinking
},
},
eval_batch_size=args.batch_size,
generation_config={
'max_tokens': 30000, # Max number of generated tokens, suggested to set a large value to avoid output truncation
'temperature': args.temperature, # Sampling temperature (recommended value per Qwen report)
'top_p': args.top_p, # top-p sampling (recommended value per Qwen report)
'top_k': args.top_k, # top-k sampling (recommended value per Qwen report)
'n': 1, # Number of replies generated per request
},
timeout=70000, # Timeout
stream=False, # Use streaming output
limit=args.sample_size,
ignore_errors=True# Set to 100 samples for testing
)

run_task(task_cfg=task_cfg)

上面是评测脚本的主要部分
评测命令是： python3 eval_public2.py --model Qwen3-30B-A3B --dataset mmlu --sample_size 120 --eval_type server

输出是：
root@dsw-640862-7ffbf6d56-27h7l:/mnt/workspace/E2Eeval# python3 eval_public2.py --model Qwen3-30B-A3B --dataset mmlu --sample_size 120 --eval_type server
2025-10-16 19:17:14,936 - numexpr.utils - INFO: Note: NumExpr detected 44 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2025-10-16 19:17:14,936 - numexpr.utils - INFO: NumExpr defaulting to 16 threads.
usage: eval_public2.py [-h] --model MODEL --dataset DATASET --port PORT [--eval_type EVAL_TYPE] [--sample_size SAMPLE_SIZE] [--batch_size BATCH_SIZE]
[--temperature TEMPERATURE] [--top_p TOP_P] [--top_k TOP_K]
eval_public2.py: error: the following arguments are required: --port
root@dsw-640862-7ffbf6d56-27h7l:/mnt/workspace/E2Eeval# python3 eval_public2.py --model Qwen3-30B-A3B --dataset mmlu --sample_size 120 --eval_type server --port 8000
2025-10-16 19:17:21,486 - numexpr.utils - INFO: Note: NumExpr detected 44 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2025-10-16 19:17:21,486 - numexpr.utils - INFO: NumExpr defaulting to 16 threads.
In evalscope Current temperature: 0.6
In evalscope Current top_p: 0.95
In evalscope Current top_k: 20
2025-10-16 19:17:21 - evalscope - WARNING: Deprecated: The timeout parameter is deprecated and will be removed in v1.1.0. Use generation_config.timeout instead.
2025-10-16 19:17:21 - evalscope - WARNING: Deprecated: The stream parameter is deprecated and will be removed in v1.1.0. Use generation_config.stream instead.
2025-10-16 19:17:21 - evalscope - INFO: Args: Task config is provided with TaskConfig type.
2025-10-16 19:17:22 - evalscope - INFO: Creating model Qwen3-30B-A3B with eval_type=server base_url=http://127.0.0.1:8000/v1/chat/completions, config={'timeout': 70000, 'batch_size': 10, 'stream': False, 'max_tokens': 30000, 'top_p': 0.95, 'temperature': 0.6, 'top_k': 20, 'n': 1}, model_args={}
2025-10-16 19:17:22 - evalscope - WARNING: server in model_apis.py has been deprecated since version 1.0.0. and will be removed in version 1.1.0. Use openai_api instead
2025-10-16 19:17:22 - evalscope - INFO: Dump task config to ./outputs/20251016_191721/configs/task_config_c3b3a0.yaml
2025-10-16 19:17:22 - evalscope - INFO: {
"model": "Qwen3-30B-A3B",
"model_id": "Qwen3-30B-A3B",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"mmlu"
],
"dataset_args": {
"aime24": {
"dataset_id": "HuggingFaceH4/aime_2024",
"filters": {
"remove_until": ""
}
},
"iquiz": {
"dataset_id": "AI-ModelScope/IQuiz",
"filters": {
"remove_until": ""
}
},
"mmlu": {
"name": "mmlu",
"dataset_id": "modelscope/mmlu",
"output_types": [
"generation"
],
"subset_list": [
"abstract_algebra",
"anatomy",
"astronomy",
"business_ethics",
"clinical_knowledge",
"college_biology",
"college_chemistry",
"college_computer_science",
"college_mathematics",
"college_medicine",
"college_physics",
"computer_security",
"conceptual_physics",
"econometrics",
"electrical_engineering",
"elementary_mathematics",
"formal_logic",
"global_facts",
"high_school_biology",
"high_school_chemistry",
"high_school_computer_science",
"high_school_european_history",
"high_school_geography",
"high_school_government_and_politics",
"high_school_macroeconomics",
"high_school_mathematics",
"high_school_microeconomics",
"high_school_physics",
"high_school_psychology",
"high_school_statistics",
"high_school_us_history",
"high_school_world_history",
"human_aging",
"human_sexuality",
"international_law",
"jurisprudence",
"logical_fallacies",
"machine_learning",
"management",
"marketing",
"medical_genetics",
"miscellaneous",
"moral_disputes",
"moral_scenarios",
"nutrition",
"philosophy",
"prehistory",
"professional_accounting",
"professional_law",
"professional_medicine",
"professional_psychology",
"public_relations",
"security_studies",
"sociology",
"us_foreign_policy",
"virology",
"world_religions"
],
"default_subset": "all",
"few_shot_num": 5,
"few_shot_random": false,
"train_split": "dev",
"eval_split": "test",
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "MMLU",
"description": "The MMLU (Massive Multitask Language Understanding) benchmark is a comprehensive evaluation suite designed to assess the performance of language models across a wide range of subjects and tasks. It includes multiple-choice questions from various domains, such as history, science, mathematics, and more, providing a robust measure of a model's understanding and reasoning capabilities.",
"tags": [
"Knowledge",
"MCQ"
],
"filters": {
"remove_until": ""
},
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"review_timeout": null,
"extra_params": {}
},
"super_gpqa": {
"dataset_id": "m-a-p/SuperGPQA",
"filters": {
"remove_until": ""
}
},
"bbh": {
"dataset_id": "modelscope/bbh",
"filters": {
"remove_until": ""
}
},
"gsm8k": {
"dataset_id": "modelscope/gsm8k",
"filters": {
"remove_until": ""
}
}
},
"dataset_dir": "/root/.cache/modelscope/hub/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"timeout": 70000,
"batch_size": 10,
"stream": false,
"max_tokens": 30000,
"top_p": 0.95,
"temperature": 0.6,
"top_k": 20,
"n": 1
},
"eval_type": "server",
"eval_backend": "Native",
"eval_config": null,
"limit": 120,
"eval_batch_size": 10,
"use_cache": null,
"rerun_review": false,
"work_dir": "./outputs/20251016_191721",
"ignore_errors": true,
"debug": false,
"seed": 42,
"api_url": "http://127.0.0.1:8000/v1/chat/completions",
"timeout": 70000,
"stream": false,
"judge_strategy": "auto",
"judge_worker_num": 1,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {},
"sandbox_config": {}
}
2025-10-16 19:17:22 - evalscope - INFO: Loading dataset modelscope/mmlu from modelscope > subset: all > split: test ...
Downloading [README.md]: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 503/503 [00:00<00:00, 6.74MB/s]
Downloading [README.md]: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.26k/5.26k [00:00<00:00, 39.0MB/s]
Traceback (most recent call last):
File "/mnt/workspace/E2Eeval/eval_public2.py", line 78, in
run_task(task_cfg=task_cfg)
File "/usr/local/lib/python3.12/site-packages/evalscope/run.py", line 28, in run_task
return run_single_task(task_cfg, run_time)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/run.py", line 41, in run_single_task
result = evaluate_model(task_cfg, outputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/run.py", line 144, in evaluate_model
res_dict = evaluator.eval()
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/evaluator/evaluator.py", line 94, in eval
dataset_dict = self.benchmark.load_dataset()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 62, in load_dataset
self.test_dataset, self.fewshot_dataset = self.load()
^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 81, in load
return self.load_from_remote()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 89, in load_from_remote
test_dataset = self.load_subsets(test_load_func)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 201, in load_subsets
subset_data = load_func(self.default_subset)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 248, in load_subset
dataset = loader.load()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/evalscope/api/dataset/loader.py", line 88, in load
dataset = MsDataset.load(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/ms_dataset.py", line 308, in load
with load_dataset_with_ctx(
File "/usr/local/lib/python3.12/contextlib.py", line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 1475, in load_dataset_with_ctx
dataset_res = DatasetsWrapperHF.load_dataset(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 991, in load_dataset
builder_instance = DatasetsWrapperHF.load_dataset_builder(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 1164, in load_dataset_builder
builder_instance: DatasetBuilder = builder_cls(
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/datasets/builder.py", line 343, in init
self.config, self.config_id = self._create_builder_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/datasets/builder.py", line 570, in _create_builder_config
raise ValueError(
ValueError: BuilderConfig 'all' not found. Available: ['high_school_european_history', 'business_ethics', 'clinical_knowledge', 'medical_genetics', 'high_school_us_history', 'high_school_physics', 'high_school_world_history', 'virology', 'high_school_microeconomics', 'econometrics', 'college_computer_science', 'high_school_biology', 'abstract_algebra', 'professional_accounting', 'philosophy', 'professional_medicine', 'nutrition', 'global_facts', 'machine_learning', 'security_studies', 'public_relations', 'professional_psychology', 'prehistory', 'anatomy', 'human_sexuality', 'college_medicine', 'high_school_government_and_politics', 'college_chemistry', 'logical_fallacies', 'high_school_geography', 'elementary_mathematics', 'human_aging', 'college_mathematics', 'high_school_psychology', 'formal_logic', 'high_school_statistics', 'international_law', 'high_school_mathematics', 'high_school_computer_science', 'conceptual_physics', 'miscellaneous', 'high_school_chemistry', 'marketing', 'professional_law', 'management', 'college_physics', 'jurisprudence', 'world_religions', 'sociology', 'us_foreign_policy', 'high_school_macroeconomics', 'computer_security', 'moral_scenarios', 'moral_disputes', 'electrical_engineering', 'astronomy', 'college_biology']

[Yunnglin]

脚本中不需要设置dataset id，会自动寻找兼容版本的数据集，你手动设置数据集的并不兼容

[jingjingpiggy]

ok，get，再多问一句，是不是所有的 评测集的dataset id 都不需要设置，比如iquiz 这些

[Yunnglin]

ok，get，再多问一句，是不是所有的 评测集的dataset id 都不需要设置，比如iquiz 这些

都不需要手动设置，除非你需要从本地进行加载数据集

[jingjingpiggy]

],
dataset_args={
    'aime24': {
        'dataset_id': 'HuggingFaceH4/aime_2024',
        'filters': {'remove_until': '</think>'}  # Filter out the content of thinking
        },
    'iquiz': {
        # 'local_path': "/usr/local/lib/python3.12/site-packages/evalscope/custom_eval/",
        # 'subset_list':['IQ', 'EQ'],
        # 'dataset_id': 'AI-ModelScope/IQuiz',
        'filters': {'remove_until': '</think>'}  # Filter out the content of thinking
        },
    'mmlu': {
        # 'dataset_id': 'modelscope/mmlu',
        'dataset_id': 'cais/mmlu',   #1.0.2版本不需要dataset_id, 默认的是cais/mmlu
        'filters': {'remove_until': '</think>'},
        # 'subset_list':mmlu_subset_list
        },
    'super_gpqa': {
        'dataset_id': 'm-a-p/SuperGPQA',
        'filters': {'remove_until': '</think>'}  # Filter out the content of thinking
        },
    'bbh': {
        'dataset_id': 'modelscope/bbh',
        'filters': {'remove_until': '</think>'}  # Filter out the content of thinking
        }, 
    'gsm8k': {
        'dataset_id': 'modelscope/gsm8k',
        'filters': {'remove_until': '</think>'}  # Filter out the content of thinking
    },
},

同样的运行命令：
python3 eval_public.py --model Qwen3-30B-A3B --port 8000 --eval_type server --sample_size 10 --dataset iquiz

在iquiz上报错
builder_instance = DatasetsWrapperHF.load_dataset_builder(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 1125, in load_dataset_builder
dataset_module = DatasetsWrapperHF.dataset_module_factory(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 1425, in dataset_module_factory
raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at /mnt/workspace/E2Eeval/AI-ModelScope/IQuiz/IQuiz.py or any data file in the same directory. Couldn't find 'AI-ModelScope/IQuiz' on the Hugging Face Hub either: FileNotFoundError: Unable to find 'hf://datasets/AI-ModelScope/IQuiz@master/IQ.jsonl' with any supported extension ['.csv', '.tsv', '.json', '.jsonl', '.ndjson', '.parquet', '.geoparquet', '.gpq', '.arrow', '.txt', '.tar', '.xml', '.blp', '.bmp', '.dib', '.bufr', '.cur', '.pcx', '.dcx', '.dds', '.ps', '.eps', '.fit', '.fits', '.fli', '.flc', '.ftc', '.ftu', '.gbr', '.gif', '.grib', '.png', '.apng', '.jp2', '.j2k', '.jpc', '.jpf', '.jpx', '.j2c', '.icns', '.ico', '.im', '.iim', '.tif', '.tiff', '.jfif', '.jpe', '.jpg', '.jpeg', '.mpg', '.mpeg', '.msp', '.pcd', '.pxr', '.pbm', '.pgm', '.ppm', '.pnm', '.psd', '.bw', '.rgb', '.rgba', '.sgi', '.ras', '.tga', '.icb', '.vda', '.vst', '.webp', '.wmf', '.emf', '.xbm', '.xpm', '.BLP', '.BMP', '.DIB', '.BUFR', '.CUR', '.PCX', '.DCX', '.DDS', '.PS', '.EPS', '.FIT', '.FITS', '.FLI', '.FLC', '.FTC', '.FTU', '.GBR', '.GIF', '.GRIB', '.PNG', '.APNG', '.JP2', '.J2K', '.JPC', '.JPF', '.JPX', '.J2C', '.ICNS', '.ICO', '.IM', '.IIM', '.TIF', '.TIFF', '.JFIF', '.JPE', '.JPG', '.JPEG', '.MPG', '.MPEG', '.MSP', '.PCD', '.PXR', '.PBM', '.PGM', '.PPM', '.PNM', '.PSD', '.BW', '.RGB', '.RGBA', '.SGI', '.RAS', '.TGA', '.ICB', '.VDA', '.VST', '.WEBP', '.WMF', '.EMF', '.XBM', '.XPM', '.aiff', '.au', '.avr', '.caf', '.flac', '.htk', '.svx', '.mat4', '.mat5', '.mpc2k', '.ogg', '.paf', '.pvf', '.raw', '.rf64', '.sd2', '.sds', '.ircam', '.voc', '.w64', '.wav', '.nist', '.wavex', '.wve', '.xi', '.mp3', '.opus', '.AIFF', '.AU', '.AVR', '.CAF', '.FLAC', '.HTK', '.SVX', '.MAT4', '.MAT5', '.MPC2K', '.OGG', '.PAF', '.PVF', '.RAW', '.RF64', '.SD2', '.SDS', '.IRCAM', '.VOC', '.W64', '.WAV', '.NIST', '.WAVEX', '.WVE', '.XI', '.MP3', '.OPUS', '.mkv', '.mp4', '.avi', '.mov', '.MKV', '.MP4', '.AVI', '.MOV', '.pdf', '.PDF', '.zip']

[jingjingpiggy]

补充一下，试了一下 去掉dataset_id, mmlu去掉dataset_id是ok的，iquiz 不 ok

[Yunnglin]

你本地的这个路径是否存在/mnt/workspace/E2Eeval/AI-ModelScope/IQuiz，若存在需要把它删掉

[jingjingpiggy]

root@dsw-640862-5df5bf847b-rb8vg:/mnt/workspace/E2Eeval# ls AI-ModelScope/IQuiz
ls: cannot access 'AI-ModelScope/IQuiz': No such file or directory
root@dsw-640862-5df5bf847b-rb8vg:/mnt/workspace/E2Eeval# ls AI-ModelScope
ls: cannot access 'AI-ModelScope': No such file or directory

没有这两个路径