Evaluation issues when conducting eval using the pre-downloaded and local dataset, scanrefer/cv-bench/blink-spatial, etc

[TyroneLi]

Hi,

After finishing the 3d spatial reasoning and 3d scene understanding training, I tried to eval on scanrefer while following your data folders structures, but had these errors:

task_object = ConfigurableTask(config=config, model_name=self.model_name) File "/leonardo_scratch/fast/IscrC_EEOL/3D_LLM_Embodiment/3D_Spatial_Mind/codebases/VG-LLM/src/lmms_eval/api/task.py", line 720, in __init__ self.download(self.config.dataset_kwargs) File "/leonardo_scratch/fast/IscrC_EEOL/softwares/local/anaconda3/envs/vgllm_py310/lib/python3.10/site-packages/tenacity/__init__.py", line 330, in wrapped_f return self(f, *args, **kw) File "/leonardo_scratch/fast/IscrC_EEOL/softwares/local/anaconda3/envs/vgllm_py310/lib/python3.10/site-packages/tenacity/__init__.py", line 467, in __call__ do = self.iter(retry_state=retry_state) File "/leonardo_scratch/fast/IscrC_EEOL/softwares/local/anaconda3/envs/vgllm_py310/lib/python3.10/site-packages/tenacity/__init__.py", line 368, in iter result = action(retry_state) File "/leonardo_scratch/fast/IscrC_EEOL/softwares/local/anaconda3/envs/vgllm_py310/lib/python3.10/site-packages/tenacity/__init__.py", line 411, in exc_check raise retry_exc from fut.exception() tenacity.RetryError: RetryError[<Future at 0x1497da4c2cb0 state=finished raised OSError>]

And also, I found it's very hard to conduct evaluations on VSI_bench, CV_Bench, Blink_spatial, etc, if I pre-downloaded these datasets and specify the data folder at there yaml using lmms_eval. Do you have any suggestions for evaluating these offline since my server cannot access the network.