Where is the train_crop_manifest.csv?

[rhoadesScholar]

... where is the train_crop_manifest.csv that indicates all the labeled crops' corresponding raw fib-sem images, that is "voxel_size" "translation" "shape", etc.

From @FrankieMAN-YU

[rhoadesScholar]

There is not currently a matching training crop manifest, though src/cellmap_segmentation_challenge/utils/manifest.csv will list which crops correspond to which datasets and which EM. All of the training crops are downloaded inside of the dataset zarr. More information about the data can be found in the data/README.md, and in the docs.

The structure of the downloaded data is:

.
├── <dataset name>
│   └── <dataset name>.zarr
│       └── recon-<number>
│           ├── em
│           │   └── fibsem-uint8
│           │       ├── s0 <-- Highest resolution scale level
│           │       ├── s1
│           │       └── ...
│           │       └── .zattrs <-- Contains metadata about all of the scale levels, including spatial locations
│           │       └── .zgroup
│           └── labels
│               └── groundtruth
│                   └── crop<number>
│                       ├── <label class 1>
│                       │   └── s0 <-- Highest resolution
│                       │   └── ...
│                       │   └── .zattrs <-- Contains metadata about all of the scale levels, including spatial locations
│                       │   └── .zgroup
│                       ├── <label class 2>
│                       ├── ...
│                       └── all <-- All labels combined
├── ...
└── README.md

So, you can find all of the crops within their respective datasets, alongside the EM.
Does that help, @FrankieMAN-YU?

[rhoadesScholar]

Training uses a "datasplit" file, which lists out which crops to use for training and which to use for validation, along with their full file path and the path to the corresponding EM data. By default this splitting is done at random with 15% being used for validation. If a datasplit.csv file does not already exist, the train function calls make_datasplit_csv.

Based on the shape and scale designated in the input_array_info and target_array_info, the dataloader creating in the training function will load batches with data at the specified resolution from the designated ground truth crops and corresponding EM.

@FrankieMAN-YU Would you still like a train_crop_manifest.csv equivalent to the test_crop_manifest.csv?

[ieellee]

Yep, I really need a train_crop_manifest.csv equivalent to the test_crop_manifest.csv, so I can develop my custom algorithms.

datasplit.csv file can really split training and validating datasets, what confused me is that without train_crop_manifest.csv, how can it map crop blocks back to the raw EM data? test_crop_manifest.csv shows a demo that how to find corresponding raw EM images based on crop annotations.

sincerely looking forward your early reply! @rhoadesScholar

[rhoadesScholar]

Each Zarr file contains metadata about its spatial location in its . zattrs file (see above file tree). So, all of their info is self contained. 👍🏼 I will, however, post an equivalent csv in a bit.

[rhoadesScholar]

As promised @FrankieMAN-YU, here is a csv of crops available for training, matching that supplied for the test crops.
train_crop_manifest.csv

Happy training!

[ieellee]

wow! This helps a lot! Thanks!!!