The included training script doesn't produce viable training samples when target class is nuc

[fgdfgfthgr-fox]

I downloaded the data with csc fetch-data --zip and ran the included train_2D.py example. When I open the tensorboard to inspect the training process, I see this:

And the validation score doesn't decrease overtime. I think something is wrong with the data loading.

Part of the training log:

/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/bin/python /mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/examples/train_2D.py 
Training device: cuda
Reading csv file...
Constructing datasets...
Training datasets: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 231/231 [00:01<00:00, 194.52it/s]
Validation datasets: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42/42 [00:00<00:00, 166.19it/s]
Verifying datasets...
Training datasets: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 231/231 [00:00<00:00, 490.65it/s]
Validation datasets: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42/42 [00:00<00:00, 629.33it/s]
Gathering class counts...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 64/64 [00:00<00:00, 214405.32it/s]

The generated datasplit.csv, which might be useful.
datasplit.csv

[fgdfgfthgr-fox]

Could it because I am using Python venv instead of micromamba to manage the environment? And something went wrong with the dependencies? Below is my package list:

$ pip list
Package                        Version        Editable project location
------------------------------ -------------- ----------------------------------------------------------------------------
absl-py                        2.1.0
aiobotocore                    2.16.0
aiohappyeyeballs               2.4.4
aiohttp                        3.11.11
aioitertools                   0.12.0
aiosignal                      1.3.2
annotated-types                0.7.0
asciitree                      0.3.3
atomicwrites                   1.4.1
attrs                          24.3.0
blinker                        1.9.0
boto3                          1.35.81
botocore                       1.35.81
cachetools                     5.5.0
cellmap-data                   2025.3.1.1937
cellmap-flow                   0.1.3
cellmap-schemas                0.8.0
cellmap-segmentation-challenge 0.0.1          /mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge
cellpose                       3.1.0
certifi                        2024.12.14
charset-normalizer             3.4.0
click                          8.1.8
cloudpickle                    3.1.0
contourpy                      1.3.1
cycler                         0.12.1
daisy                          1.2.2
dask                           2024.12.1
dill                           0.3.9
eval-type-backport             0.1.3
fasteners                      0.19
fastremap                      1.15.0
filelock                       3.13.1
flasgger                       0.9.7.1
Flask                          3.1.0
Flask-Cors                     5.0.0
flexcache                      0.3
flexparser                     0.4
fonttools                      4.55.3
frozenlist                     1.5.0
fsspec                         2024.12.0
funlib.geometry                0.3.0
funlib.math                    0.1
google-apitools                0.5.32
google-auth                    2.37.0
grpcio                         1.68.1
gunicorn                       23.0.0
h5py                           3.12.1
httplib2                       0.22.0
idna                           3.10
imagecodecs                    2024.9.22
imageio                        2.36.1
importlib_metadata             8.5.0
itsdangerous                   2.2.0
Jinja2                         3.1.3
jmespath                       1.0.1
joblib                         1.4.2
jsonschema                     4.23.0
jsonschema-specifications      2024.10.1
kiwisolver                     1.4.7
lazy_loader                    0.4
lightning                      2.5.0.post0
lightning-utilities            0.11.9
llvmlite                       0.43.0
locket                         1.0.0
Markdown                       3.7
markdown-it-py                 3.0.0
MarkupSafe                     2.1.5
marshmallow                    3.26.1
matplotlib                     3.7.0
mdurl                          0.1.2
mistune                        3.1.1
ml_collections                 1.0.0
ml_dtypes                      0.5.0
mpmath                         1.3.0
multidict                      6.1.0
natsort                        8.4.0
networkx                       3.2.1
neuroglancer                   2.40.1
numba                          0.60.0
numcodecs                      0.13.1
numpy                          1.26.3
oauth2client                   4.1.3
opencv-python-headless         4.10.0.84
packaging                      24.2
pandas                         2.2.3
partd                          1.4.2
pillow                         10.2.0
Pint                           0.24.4
pip                            23.2.1
platformdirs                   4.3.6
propcache                      0.2.1
protobuf                       5.29.2
pyasn1                         0.6.1
pyasn1_modules                 0.4.1
pydantic                       2.10.4
pydantic_core                  2.27.2
pydantic-ome-ngff              0.6.0
pydantic-zarr                  0.7.0
Pygments                       2.18.0
pykdtree                       1.4.1
pyparsing                      3.2.0
python-dateutil                2.9.0.post0
python-dotenv                  1.0.1
pytorch-lightning              2.5.0.post0
pytorch-triton-rocm            3.1.0
pytz                           2024.2
PyYAML                         6.0.2
referencing                    0.36.2
requests                       2.32.3
rich                           13.9.4
roifile                        2024.9.15
rpds-py                        0.22.3
rsa                            4.9
s3fs                           2024.12.0
s3transfer                     0.10.4
scikit-image                   0.25.0
scikit-learn                   1.6.0
scipy                          1.14.1
setuptools                     68.2.0
six                            1.17.0
structlog                      24.4.0
sympy                          1.13.1
tensorboard                    2.18.0
tensorboard-data-server        0.7.2
tensorboardX                   2.6.2.2
tensorstore                    0.1.71
threadpoolctl                  3.5.0
tifffile                       2024.12.12
toolz                          1.0.0
torch                          2.5.1+rocm6.2
torchmetrics                   1.6.0
torchvision                    0.20.1+rocm6.2
tornado                        6.4.2
tqdm                           4.67.1
typing_extensions              4.12.2
tzdata                         2024.2
universal_pathlib              0.2.6
urllib3                        2.3.0
Werkzeug                       3.1.3
wheel                          0.41.2
wrapt                          1.17.0
xarray                         2024.11.0
xarray-ome-ngff                3.1.1
xarray-tensorstore             0.1.5
yarl                           1.18.3
zarr                           2.18.4
zipp                           3.21.0

[fgdfgfthgr-fox]

If I set classes = ["er"] then everything is normal. Although I wonder if Full Fov is supposed to look the same as Raw.
Same goes for other classes, I haven't tested all of them but so far only the nuc seem to have this problem.

[rhoadesScholar]

@mzouink / @yuriyzubov Can you check the zipped data to see if this is an issue on our end?

@fgdfgfthgr-fox When did you do this download?

[rhoadesScholar]

Could it because I am using Python venv instead of micromamba to manage the environment? And something went wrong with the dependencies? Below is my package list:

$ pip list
Package                        Version        Editable project location
------------------------------ -------------- ----------------------------------------------------------------------------
absl-py                        2.1.0
aiobotocore                    2.16.0
aiohappyeyeballs               2.4.4
aiohttp                        3.11.11
aioitertools                   0.12.0
aiosignal                      1.3.2
annotated-types                0.7.0
asciitree                      0.3.3
atomicwrites                   1.4.1
attrs                          24.3.0
blinker                        1.9.0
boto3                          1.35.81
botocore                       1.35.81
cachetools                     5.5.0
cellmap-data                   2025.3.1.1937
cellmap-flow                   0.1.3
cellmap-schemas                0.8.0
cellmap-segmentation-challenge 0.0.1          /mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge
cellpose                       3.1.0
certifi                        2024.12.14
charset-normalizer             3.4.0
click                          8.1.8
cloudpickle                    3.1.0
contourpy                      1.3.1
cycler                         0.12.1
daisy                          1.2.2
dask                           2024.12.1
dill                           0.3.9
eval-type-backport             0.1.3
fasteners                      0.19
fastremap                      1.15.0
filelock                       3.13.1
flasgger                       0.9.7.1
Flask                          3.1.0
Flask-Cors                     5.0.0
flexcache                      0.3
flexparser                     0.4
fonttools                      4.55.3
frozenlist                     1.5.0
fsspec                         2024.12.0
funlib.geometry                0.3.0
funlib.math                    0.1
google-apitools                0.5.32
google-auth                    2.37.0
grpcio                         1.68.1
gunicorn                       23.0.0
h5py                           3.12.1
httplib2                       0.22.0
idna                           3.10
imagecodecs                    2024.9.22
imageio                        2.36.1
importlib_metadata             8.5.0
itsdangerous                   2.2.0
Jinja2                         3.1.3
jmespath                       1.0.1
joblib                         1.4.2
jsonschema                     4.23.0
jsonschema-specifications      2024.10.1
kiwisolver                     1.4.7
lazy_loader                    0.4
lightning                      2.5.0.post0
lightning-utilities            0.11.9
llvmlite                       0.43.0
locket                         1.0.0
Markdown                       3.7
markdown-it-py                 3.0.0
MarkupSafe                     2.1.5
marshmallow                    3.26.1
matplotlib                     3.7.0
mdurl                          0.1.2
mistune                        3.1.1
ml_collections                 1.0.0
ml_dtypes                      0.5.0
mpmath                         1.3.0
multidict                      6.1.0
natsort                        8.4.0
networkx                       3.2.1
neuroglancer                   2.40.1
numba                          0.60.0
numcodecs                      0.13.1
numpy                          1.26.3
oauth2client                   4.1.3
opencv-python-headless         4.10.0.84
packaging                      24.2
pandas                         2.2.3
partd                          1.4.2
pillow                         10.2.0
Pint                           0.24.4
pip                            23.2.1
platformdirs                   4.3.6
propcache                      0.2.1
protobuf                       5.29.2
pyasn1                         0.6.1
pyasn1_modules                 0.4.1
pydantic                       2.10.4
pydantic_core                  2.27.2
pydantic-ome-ngff              0.6.0
pydantic-zarr                  0.7.0
Pygments                       2.18.0
pykdtree                       1.4.1
pyparsing                      3.2.0
python-dateutil                2.9.0.post0
python-dotenv                  1.0.1
pytorch-lightning              2.5.0.post0
pytorch-triton-rocm            3.1.0
pytz                           2024.2
PyYAML                         6.0.2
referencing                    0.36.2
requests                       2.32.3
rich                           13.9.4
roifile                        2024.9.15
rpds-py                        0.22.3
rsa                            4.9
s3fs                           2024.12.0
s3transfer                     0.10.4
scikit-image                   0.25.0
scikit-learn                   1.6.0
scipy                          1.14.1
setuptools                     68.2.0
six                            1.17.0
structlog                      24.4.0
sympy                          1.13.1
tensorboard                    2.18.0
tensorboard-data-server        0.7.2
tensorboardX                   2.6.2.2
tensorstore                    0.1.71
threadpoolctl                  3.5.0
tifffile                       2024.12.12
toolz                          1.0.0
torch                          2.5.1+rocm6.2
torchmetrics                   1.6.0
torchvision                    0.20.1+rocm6.2
tornado                        6.4.2
tqdm                           4.67.1
typing_extensions              4.12.2
tzdata                         2024.2
universal_pathlib              0.2.6
urllib3                        2.3.0
Werkzeug                       3.1.3
wheel                          0.41.2
wrapt                          1.17.0
xarray                         2024.11.0
xarray-ome-ngff                3.1.1
xarray-tensorstore             0.1.5
yarl                           1.18.3
zarr                           2.18.4
zipp                           3.21.0

@fgdfgfthgr-fox
venv vs. mamba shouldn't matter - or at least not create a bug like this.

[rhoadesScholar]

If I set classes = ["er"] then everything is normal. Although I wonder if Full Fov is supposed to look the same as Raw.
Same goes for other classes, I haven't tested all of them but so far only the nuc seem to have this problem.

@fgdfgfthgr-fox If the input and target arrays have the same shape and scale, then the full FOV is the same as the FOV of the target image (which is shown in Raw).

[fgdfgfthgr-fox]

@fgdfgfthgr-fox When did you do this download?

Shortly after #77 has been resolved.

[fgdfgfthgr-fox]

Shortly after #77 has been resolved.

Actually should be around 02/14/25

[rhoadesScholar]

I suspect this is because you are trying to train on 8nm data for "nuc" without having downloaded all resolutions of the data.

This is probably a bug in how the metadata is downloaded.

Try using csc visualize -k gt -C nuc to see what data is present.

[fgdfgfthgr-fox]

Try using csc visualize -k gt -C nuc to see what data is present.

$ csc visualize -k gt -C nuc
Traceback (most recent call last):
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/bin/csc", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/src/cellmap_segmentation_challenge/cli/visualize.py", line 58, in visualize_cli
    visualize(
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/src/cellmap_segmentation_challenge/visualize.py", line 91, in visualize
    s.layers["fibsem"] = get_layer(get_raw_path(dataset_path), "image")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/src/cellmap_segmentation_challenge/visualize.py", line 216, in get_layer
    image = open_ds_tensorstore(this_path)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/cellmap_flow/utils/ds.py", line 119, in open_ds_tensorstore
    return dataset_future.result()
           ^^^^^^^^^^^^^^^^^^^^^^^
ValueError: FAILED_PRECONDITION: Error opening "zarr" driver: Error reading local file "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/data/jrc_cos7-1a/jrc_cos7-1a.zarr/recon-1/em/fibsem-uint8/s10/.zarray": Error parsing object member "compressor": Object includes extra members: "checksum" [source locations='tensorstore/internal/json_binding/json_binding.h:829\ntensorstore/internal/json_binding/json_binding.h:865\ntensorstore/internal/json_binding/json_binding.h:829\ntensorstore/internal/json_binding/json_binding.h:388\ntensorstore/driver/zarr/driver.cc:107\ntensorstore/driver/kvs_backed_chunk_driver.cc:1163\ntensorstore/internal/cache/kvs_backed_cache.h:208\ntensorstore/driver/driver.cc:117'] [tensorstore_spec='{\"context\":{\"cache_pool\":{},\"data_copy_concurrency\":{},\"file_io_concurrency\":{},\"file_io_locking\":{},\"file_io_memmap\":false,\"file_io_sync\":true},\"driver\":\"zarr\",\"kvstore\":{\"driver\":\"file\",\"path\":\"/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/data/jrc_cos7-1a/jrc_cos7-1a.zarr/recon-1/em/fibsem-uint8/s10/\"}}']

That command does not work.

[rhoadesScholar]

Try using csc visualize -k gt -C nuc to see what data is present.

$ csc visualize -k gt -C nuc
Traceback (most recent call last):
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/bin/csc", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/src/cellmap_segmentation_challenge/cli/visualize.py", line 58, in visualize_cli
    visualize(
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/src/cellmap_segmentation_challenge/visualize.py", line 91, in visualize
    s.layers["fibsem"] = get_layer(get_raw_path(dataset_path), "image")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/src/cellmap_segmentation_challenge/visualize.py", line 216, in get_layer
    image = open_ds_tensorstore(this_path)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/venv/lib/python3.11/site-packages/cellmap_flow/utils/ds.py", line 119, in open_ds_tensorstore
    return dataset_future.result()
           ^^^^^^^^^^^^^^^^^^^^^^^
ValueError: FAILED_PRECONDITION: Error opening "zarr" driver: Error reading local file "/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/data/jrc_cos7-1a/jrc_cos7-1a.zarr/recon-1/em/fibsem-uint8/s10/.zarray": Error parsing object member "compressor": Object includes extra members: "checksum" [source locations='tensorstore/internal/json_binding/json_binding.h:829\ntensorstore/internal/json_binding/json_binding.h:865\ntensorstore/internal/json_binding/json_binding.h:829\ntensorstore/internal/json_binding/json_binding.h:388\ntensorstore/driver/zarr/driver.cc:107\ntensorstore/driver/kvs_backed_chunk_driver.cc:1163\ntensorstore/internal/cache/kvs_backed_cache.h:208\ntensorstore/driver/driver.cc:117'] [tensorstore_spec='{\"context\":{\"cache_pool\":{},\"data_copy_concurrency\":{},\"file_io_concurrency\":{},\"file_io_locking\":{},\"file_io_memmap\":false,\"file_io_sync\":true},\"driver\":\"zarr\",\"kvstore\":{\"driver\":\"file\",\"path\":\"/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/data/jrc_cos7-1a/jrc_cos7-1a.zarr/recon-1/em/fibsem-uint8/s10/\"}}']

That command does not work.

Unfortunately this is an issue with the data, not the command. @yuriyzubov I thought we fixed the "checksum" issue? I think it was linked to making empty scale arrays, but maybe we only fixed it for ground truth? Might also be fixed by omitting empty scale levels.

[yuriyzubov]

HI, @fgdfgfthgr-fox

I does seem like you have the old version of the data that contained checksum parameter, which caused tensor-store error.
I recommend to download the latest data, since all the issues should be resolved, as well as additional raw fib-sem data was also added when we opened submussion.

Just in case the checksum issue still persists, you can resolve it by removing it from the metadata:

import json
import os
def del_checksum(path_to_zarray : str):
    path_to_zarray = os.path.join(path_to_zarray, '.zarray')
    with open(path_to_zarray, 'r+') as f:
        data = json.load(f)
        try:
            del data['compressor']['checksum']
        except:
            pass

    with open(path_to_zarray, 'w', encoding='utf-8') as f:
        json.dump(data, f, ensure_ascii=False, indent=4)

for a single array you can run this method

del_checksum('/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/data/jrc_cos7-1a/jrc_cos7-1a.zarr/recon-1/em/fibsem-uint8/s10/')

for all fib-sem data:

import zarr
path_to_datasets = '/mnt/2878EBCCAED823C6/PycharmProjects/cellmap/cellmap-segmentation-challenge/data'
datasets = os.listdir(path_to_datasets)
for ds in datasets:
    path_to_fibsem = os.path.join(path_to_datasets, ds, f'{ds}.zarr', 'recon-1/em/fibsem-uint8')
    em = zarr.open(path_to_fibsem, mode = 'r')
    arrs = em.array_keys(recurse=True)
    for arr in arrs:
        del_checksum(os.path.join(path_to_fibsem, arr))

You can try and run these code snippets to check if it fixed the issue for csc visualize -k gt -C nuc with the data you already have on your laptop.

[fgdfgfthgr-fox]

I does seem like you have the old version of the data that contained checksum parameter, which caused tensor-store error. I recommend to download the latest data, since all the issues should be resolved, as well as additional raw fib-sem data was also added when we opened submussion.

After re-download the data, the checksum issue is solved.

[fgdfgfthgr-fox]

The email communication suggests that different organelle labels have different resolutions, and even within the same organelle classes there are different resolutions for different crops, e.g. here are examples of nuclei at 8nm, 32nm.
And even for the same dataset (e.g. jrc_mus-liver) some regions may have 8nm EM data, while other regions are empty at 8nm but have data starting at e.g. 32nm, depends on what organelle labels exist in there.
So, it seems like this issue is due to the dataloader doesn't filter out EM images for nuclei labels at 32nm (in which the 8nm EM image doesn't exist). Is that correct?