KeyError: 'Series UID' in TCIA_PROSTATEx_Prostate_MRI_Anatomy_Model.ipynb

[KumoLiu]

[2025-09-17T02:57:30.769Z] Running ./model_zoo/TCIA_PROSTATEx_Prostate_MRI_Anatomy_Model.ipynb

[2025-09-17T02:57:30.769Z] Checking PEP8 compliance...

[2025-09-17T02:57:31.356Z] Running notebook...

[2025-09-17T02:57:41.458Z] <frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()

[2025-09-17T02:57:42.887Z] <frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()

[2025-09-17T02:57:46.245Z] MONAI version: 1.5.0+20.g401ea4a0

[2025-09-17T02:57:46.245Z] Numpy version: 1.24.4

[2025-09-17T02:57:46.245Z] Pytorch version: 2.5.0a0+e000cf0ad9.nv24.10

[2025-09-17T02:57:46.245Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False

[2025-09-17T02:57:46.245Z] MONAI rev id: 401ea4a0af2b9d475989a78e1bae387875fa2384

[2025-09-17T02:57:46.245Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py

[2025-09-17T02:57:46.245Z] 

[2025-09-17T02:57:46.245Z] Optional dependencies:

[2025-09-17T02:57:46.245Z] Pytorch Ignite version: 0.4.11

[2025-09-17T02:57:46.245Z] ITK version: 5.4.4

[2025-09-17T02:57:46.245Z] Nibabel version: 5.3.2

[2025-09-17T02:57:46.245Z] scikit-image version: 0.19.3

[2025-09-17T02:57:46.245Z] scipy version: 1.14.0

[2025-09-17T02:57:46.245Z] Pillow version: 11.3.0

[2025-09-17T02:57:46.245Z] Tensorboard version: 2.16.2

[2025-09-17T02:57:46.245Z] gdown version: 5.2.0

[2025-09-17T02:57:46.245Z] TorchVision version: 0.20.0a0

[2025-09-17T02:57:46.245Z] tqdm version: 4.66.5

[2025-09-17T02:57:46.245Z] lmdb version: 1.7.3

[2025-09-17T02:57:46.245Z] psutil version: 6.0.0

[2025-09-17T02:57:46.245Z] pandas version: 2.2.2

[2025-09-17T02:57:46.245Z] einops version: 0.8.0

[2025-09-17T02:57:46.245Z] transformers version: 4.40.2

[2025-09-17T02:57:46.245Z] mlflow version: 3.3.2

[2025-09-17T02:57:46.245Z] pynrrd version: 1.1.3

[2025-09-17T02:57:46.245Z] clearml version: 2.0.3rc0

[2025-09-17T02:57:46.245Z] 

[2025-09-17T02:57:46.245Z] For details about installing the optional dependencies, please visit:

[2025-09-17T02:57:46.245Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

[2025-09-17T02:57:46.245Z] 

[2025-09-17T02:57:48.887Z] papermill  --progress-bar --log-output -k python3

[2025-09-17T02:57:49.179Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -

[2025-09-17T02:57:49.179Z]   warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")

[2025-09-17T02:57:49.179Z] 
[2025-09-17T02:57:50.175Z] Executing:   0%|          | 0/32 [00:00<?, ?cell/s]
[2025-09-17T02:59:11.808Z] Executing:   3%|▎         | 1/32 [00:00<00:27,  1.11cell/s]
[2025-09-17T02:59:24.154Z] Executing:  22%|██▏       | 7/32 [01:20<04:59, 12.00s/cell]
[2025-09-17T02:59:27.504Z] Executing:  28%|██▊       | 9/32 [01:34<03:59, 10.43s/cell]
[2025-09-17T02:59:28.907Z] Executing:  38%|███▊      | 12/32 [01:38<02:17,  6.87s/cell]
[2025-09-17T02:59:33.159Z] Executing:  44%|████▍     | 14/32 [01:39<01:34,  5.23s/cell]
[2025-09-17T02:59:33.159Z] Executing:  44%|████▍     | 14/32 [01:43<02:12,  7.38s/cell]

[2025-09-17T02:59:33.159Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -

[2025-09-17T02:59:33.159Z]   warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")

[2025-09-17T02:59:33.159Z] Traceback (most recent call last):

[2025-09-17T02:59:33.159Z]   File "/usr/local/bin/papermill", line 7, in <module>

[2025-09-17T02:59:33.159Z]     sys.exit(papermill())

[2025-09-17T02:59:33.159Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__

[2025-09-17T02:59:33.159Z]     return self.main(*args, **kwargs)

[2025-09-17T02:59:33.159Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main

[2025-09-17T02:59:33.159Z]     rv = self.invoke(ctx)

[2025-09-17T02:59:33.159Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke

[2025-09-17T02:59:33.159Z]     return ctx.invoke(self.callback, **ctx.params)

[2025-09-17T02:59:33.159Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke

[2025-09-17T02:59:33.159Z]     return __callback(*args, **kwargs)

[2025-09-17T02:59:33.159Z]   File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func

[2025-09-17T02:59:33.159Z]     return f(get_current_context(), *args, **kwargs)

[2025-09-17T02:59:33.159Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 235, in papermill

[2025-09-17T02:59:33.159Z]     execute_notebook(

[2025-09-17T02:59:33.159Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 131, in execute_notebook

[2025-09-17T02:59:33.159Z]     raise_for_execution_errors(nb, output_path)

[2025-09-17T02:59:33.159Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 251, in raise_for_execution_errors

[2025-09-17T02:59:33.159Z]     raise error

[2025-09-17T02:59:33.159Z] papermill.exceptions.PapermillExecutionError: 

[2025-09-17T02:59:33.159Z] ---------------------------------------------------------------------------

[2025-09-17T02:59:33.159Z] Exception encountered at "In [5]":

[2025-09-17T02:59:33.159Z] ---------------------------------------------------------------------------

[2025-09-17T02:59:33.159Z] KeyError                                  Traceback (most recent call last)

[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py:3805, in Index.get_loc(self, key)

[2025-09-17T02:59:33.159Z]    3804 try:

[2025-09-17T02:59:33.159Z] -> 3805     return self._engine.get_loc(casted_key)

[2025-09-17T02:59:33.159Z]    3806 except KeyError as err:

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] File pandas/_libs/hashtable_class_helper.pxi:7081, in pandas._libs.hashtable.PyObjectHashTable.get_item()

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] File pandas/_libs/hashtable_class_helper.pxi:7089, in pandas._libs.hashtable.PyObjectHashTable.get_item()

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] KeyError: 'Series UID'

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] The above exception was the direct cause of the following exception:

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] KeyError                                  Traceback (most recent call last)

[2025-09-17T02:59:33.159Z] Cell In[5], line 4

[2025-09-17T02:59:33.159Z]       1 dicom_data_dir = "tciaDownload"

[2025-09-17T02:59:33.159Z]       3 # The series_uid defines their directory where the MR data was stored on disk.

[2025-09-17T02:59:33.159Z] ----> 4 mr_series_uid = df.at[df.Modality.eq("MR").idxmax(), "Series UID"]

[2025-09-17T02:59:33.159Z]       5 mr_dir = os.path.join(dicom_data_dir, mr_series_uid)

[2025-09-17T02:59:33.159Z]       7 # Read the DICOM MR series' objects and reconstruct them into a 3D ITK image.

[2025-09-17T02:59:33.159Z]       8 #   The itk.F option is added to store the image in memory using floating-point precision pixels (useful if you will filter the image or use it with MONAI).

[2025-09-17T02:59:33.159Z]       9 #   For more info on imread, see https://itkpythonpackage.readthedocs.io/en/master/Quick_start_guide.html.

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py:2575, in _AtIndexer.__getitem__(self, key)

[2025-09-17T02:59:33.159Z]    2572         raise ValueError("Invalid call for scalar access (getting)!")

[2025-09-17T02:59:33.159Z]    2573     return self.obj.loc[key]

[2025-09-17T02:59:33.159Z] -> 2575 return super().__getitem__(key)

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py:2527, in _ScalarAccessIndexer.__getitem__(self, key)

[2025-09-17T02:59:33.159Z]    2524         raise ValueError("Invalid call for scalar access (getting)!")

[2025-09-17T02:59:33.159Z]    2526 key = self._convert_key(key)

[2025-09-17T02:59:33.159Z] -> 2527 return self.obj._get_value(*key, takeable=self._takeable)

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/frame.py:4214, in DataFrame._get_value(self, index, col, takeable)

[2025-09-17T02:59:33.159Z]    4211     series = self._ixs(col, axis=1)

[2025-09-17T02:59:33.159Z]    4212     return series._values[index]

[2025-09-17T02:59:33.159Z] -> 4214 series = self._get_item_cache(col)

[2025-09-17T02:59:33.159Z]    4215 engine = self.index._engine

[2025-09-17T02:59:33.159Z]    4217 if not isinstance(self.index, MultiIndex):

[2025-09-17T02:59:33.159Z]    4218     # CategoricalIndex: Trying to use the engine fastpath may give incorrect

[2025-09-17T02:59:33.159Z]    4219     #  results if our categories are integers that dont match our codes

[2025-09-17T02:59:33.159Z]    4220     # IntervalIndex: IntervalTree has no get_loc

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/frame.py:4638, in DataFrame._get_item_cache(self, item)

[2025-09-17T02:59:33.159Z]    4633 res = cache.get(item)

[2025-09-17T02:59:33.159Z]    4634 if res is None:

[2025-09-17T02:59:33.159Z]    4635     # All places that call _get_item_cache have unique columns,

[2025-09-17T02:59:33.159Z]    4636     #  pending resolution of GH#33047

[2025-09-17T02:59:33.159Z] -> 4638     loc = self.columns.get_loc(item)

[2025-09-17T02:59:33.159Z]    4639     res = self._ixs(loc, axis=1)

[2025-09-17T02:59:33.159Z]    4641     cache[item] = res

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py:3812, in Index.get_loc(self, key)

[2025-09-17T02:59:33.159Z]    3807     if isinstance(casted_key, slice) or (

[2025-09-17T02:59:33.159Z]    3808         isinstance(casted_key, abc.Iterable)

[2025-09-17T02:59:33.159Z]    3809         and any(isinstance(x, slice) for x in casted_key)

[2025-09-17T02:59:33.159Z]    3810     ):

[2025-09-17T02:59:33.159Z]    3811         raise InvalidIndexError(key)

[2025-09-17T02:59:33.159Z] -> 3812     raise KeyError(key) from err

[2025-09-17T02:59:33.159Z]    3813 except TypeError:

[2025-09-17T02:59:33.159Z]    3814     # If we have a listlike key, _check_indexing_error will raise

[2025-09-17T02:59:33.159Z]    3815     #  InvalidIndexError. Otherwise we fall through and re-raise

[2025-09-17T02:59:33.159Z]    3816     #  the TypeError.

[2025-09-17T02:59:33.159Z]    3817     self._check_indexing_error(key)

[2025-09-17T02:59:33.159Z] 

[2025-09-17T02:59:33.159Z] KeyError: 'Series UID'

[ericspod]

I'm looking into this, but one issue I'm seeing is that pydicom-seg hasn't been updated in 2 years. This might be a dead project we shouldn't rely on as a dependency.

[ericspod]

The table appears to be presenting the Series UID in the column "Series ID" now, I think changing the column selection in various places to that string will work.

[ericspod]

We also need to add huggingface_hub as a new dependency to install, and itkwidgets might require a different version of zarr versus other dependencies and causing a conflict in some configurations.