Scenarios failed to be loaded during evaluation

[SagiIsumi]

Hi authors,

I encountered an issue during evaluation that produced the following summary:
Finished running evaluation.
Number of successful scenarios: 11,904
Number of failed scenarios: 242
Final average score of valid results: 0.8795204478315151

Upon further inspection, the detailed logs indicate repeated errors of the form:
FileNotFoundError: [Errno 2] No such file or directory: '~/navsim_workspace/dataset/sensor_blobs/test/2021.08.30.16.16.44_veh-40_01099_01351/CAM_F0/e06ab145b84e55c3.jpg' [repeated 10x across cluster]

Does this suggest that the dataset may have been corrupted or incompletely downloaded? I would appreciate any guidance on how to verify dataset integrity or recover the missing files.

[SagiIsumi]

BTW, Did you encounter this problem:

2025-07-10 15:41:59,095 WARNING {/mnt/nas-data-1/zhanglingjun.zlj1/navsim_workspace/navsim/navsim/planning/script/run_pdm_score.py:382} ----------- Failed to calculate pseudo closed-loop weights or comfort:
Traceback (most recent call last):
File "/mnt/nas-data-1/zhanglingjun.zlj1/navsim_workspace/navsim/navsim/planning/script/run_pdm_score.py", line 375, in main
pdm_score_df = create_scene_aggregators(
File "/mnt/nas-data-1/zhanglingjun.zlj1/navsim_workspace/navsim/navsim/planning/script/run_pdm_score.py", line 313, in create_scene_aggregators
updated_rows = aggregator.aggregate_scores()
File "/mnt/nas-data-1/zhanglingjun.zlj1/navsim_workspace/navsim/navsim/planning/simulation/planner/pdm_planner/scoring/scene_aggregator.py", line 95, in aggregate_scores
second_stage_now_scores = self.score_df.loc[second_stage_now_tokens]
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/indexing.py", line 1191, in getitem
return self._getitem_axis(maybe_callable, axis=axis)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/indexing.py", line 1420, in _getitem_axis
return self._getitem_iterable(key, axis=axis)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/indexing.py", line 1360, in _getitem_iterable
keyarr, indexer = self._get_listlike_indexer(key, axis)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/indexing.py", line 1558, in _get_listlike_indexer
keyarr, indexer = ax._get_indexer_strict(key, axis_name)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 6212, in _get_indexer_strict
self._raise_if_missing(keyarr, indexer, axis_name)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 6261, in _raise_if_missing
raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['5421fa2f2e2b4cb13', '13aeb83dad95704fa', '46ccc7e9752595671',\n 'c864abb5afff1f7f4', '6b5b128424d5357b0', '7be779b9f100e6373',\n '0e55c21e09e86d8ba', '778f661244251a842', '530edf0e35aa025b6',\n 'ea1bac206caf8d18e', 'b4750fa326ea1085b'],\n dtype='object', name='token')] are in the [index]"
Error executing job with overrides: ['train_test_split=navhard_two_stage', 'experiment_name=cv_agent', 'metric_cache_path=/mnt/nas-data-1/zhanglingjun.zlj1/navsim_workspace/exp/metric_cache', 'synthetic_sensor_path=/mnt/nas-data-1/zhanglingjun.zlj1/navsim_workspace/dataset/navhard_two_stage/sensor_blobs', 'synthetic_scenes_path=/mnt/nas-data-1/zhanglingjun.zlj1/navsim_workspace/dataset/navhard_two_stage/synthetic_scene_pickles']
Traceback (most recent call last):
File "/mnt/nas-data-1/zhanglingjun.zlj1/navsim_workspace/navsim/navsim/planning/script/run_pdm_score.py", line 403, in main
pcl_group_score, pcl_stage1_score, pcl_stage2_score = calculate_individual_mapping_scores(
File "/mnt/nas-data-1/zhanglingjun.zlj1/navsim_workspace/navsim/navsim/planning/script/run_pdm_score.py", line 287, in calculate_individual_mapping_scores
pd.DataFrame(all_group_scores).mean(),
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/frame.py", line 11700, in mean
result = super().mean(axis, skipna, numeric_only, **kwargs)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/generic.py", line 12439, in mean
return self._stat_function(
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/generic.py", line 12396, in _stat_function
return self._reduce(
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/frame.py", line 11569, in _reduce
res = df._mgr.reduce(blk_func)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 1500, in reduce
nbs = blk.reduce(func)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 406, in reduce
result = func(self.values)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/frame.py", line 11488, in blk_func
return op(values, axis=axis, skipna=skipna, **kwds)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/nanops.py", line 147, in f
result = alt(values, axis=axis, skipna=skipna, **kwds)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/nanops.py", line 404, in new_func
result = func(values, axis=axis, skipna=skipna, mask=mask, **kwargs)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/nanops.py", line 720, in nanmean
the_sum = _ensure_numeric(the_sum)
File "/mnt/nas-data-1/zhanglingjun.zlj_env/envs/navsim_0/lib/python3.9/site-packages/pandas/core/nanops.py", line 1686, in _ensure_numeric
raise TypeError(f"Could not convert {x} to numeric")
TypeError: Could not convert [array([nan, nan, nan, nan, nan]) array([nan, nan, nan, nan, nan])] to numeric

It is kind of strange that the results encompassing the nan value, I already visit #119 , but I still failed to complete the evaluation of Latent Trasfuser on navhard_two_stage when I cache the metric cahe for navhard_two_stage .

Here is my shell script:

TRAIN_TEST_SPLIT=navhard_two_stage
CHECKPOINT=/work/HHRI-AI/CVTeam/joseph/llava_ds/vlm_ds/navsim/navsim_workspace/exp/training_latent_transfuser_agent/2025.09.04.05.29.25/lightning_logs/version_0/checkpoints/epoch_99-step_133000.ckpt
CACHE_PATH=/work/HHRI-AI/CVTeam/joseph/llava_ds/vlm_ds/navsim/navsim_workspace/exp/metric_cache_hard
SYNTHETIC_SENSOR_PATH=$OPENSCENE_DATA_ROOT/navhard_two_stage/sensor_blobs
SYNTHETIC_SCENES_PATH=$OPENSCENE_DATA_ROOT/navhard_two_stage/synthetic_scene_pickles

CUDA_VISIBLE_DEVICES=1 python $NAVSIM_DEVKIT_ROOT/planning/script/run_pdm_score.py
train_test_split=$TRAIN_TEST_SPLIT
agent=transfuser_agent
agent.config.latent=True
worker=ray_distributed
agent.checkpoint_path=$CHECKPOINT
experiment_name=latent_transfuser_agent
metric_cache_path=$CACHE_PATH
synthetic_sensor_path=$SYNTHETIC_SENSOR_PATH
synthetic_scenes_path=$SYNTHETIC_SCENES_PATH \

Is there any potential mistakes in my scripts? Besides, I wonder how the worker functions in the evaluation.