Training DeepLab -- ValueError: Type mismatch (<class 'tuple'> vs. <class 'int'>) with values ((800,) vs. 1024) for config key: INPUT.MIN_SIZE_TRAIN

[michelewang]

Hi, Thank you so much for sharing Detectron and the implementation of DeepLabV3 using Detectron!

I'm trying to perform semantic segmentation on a custom dataset but ran into an issue when trying to train DeepLab on my data. I set up my datasets using the documentation and each of my images is 1024 by 1024. I modified the config files (specifically Base-DeepLabV3-OS16-Semantic.yaml, and deeplab_v3_plus_R_103_os16_mg124_poly_90k_bs16.yaml), only making minor changes (shown below). I also made a copy of train_net.py called train_net_xbd.py but there were no changes to the code.

This is modified "Base-DeepLabV3-OS16-Semantic.yaml" called Base.yaml

_BASE_: "../../../../configs/Base-RCNN-DilatedC5.yaml"
MODEL:
  META_ARCHITECTURE: "SemanticSegmentor"
  BACKBONE:
    FREEZE_AT: 0
  SEM_SEG_HEAD:
    NAME: "DeepLabV3Head"
    IN_FEATURES: ["res5"]
    ASPP_CHANNELS: 256
    ASPP_DILATIONS: [6, 12, 18]
    ASPP_DROPOUT: 0.1
    CONVS_DIM: 256
    COMMON_STRIDE: 16
    NUM_CLASSES: 19
    LOSS_TYPE: "hard_pixel_mining"
DATASETS:
  TRAIN: ("xbddata_train",)
  TEST: ("xbddata_test",)
SOLVER:
  BASE_LR: 0.01
  MAX_ITER: 90000
  LR_SCHEDULER_NAME: "WarmupPolyLR"
  IMS_PER_BATCH: 16
INPUT:
  MIN_SIZE_TRAIN: (1024)
  MIN_SIZE_TRAIN_SAMPLING: "choice"
  MIN_SIZE_TEST: 1024
  MAX_SIZE_TRAIN: 1024
  MAX_SIZE_TEST: 1024
  CROP:
    ENABLED: True
    TYPE: "absolute"
    SIZE: (512, 1024)
    SINGLE_CATEGORY_MAX_AREA: 1.0
DATALOADER:
  NUM_WORKERS: 10

This is modified "deeplab_v3_plus_R_103_os16_mg124_poly_90k_bs16.yaml" called base-deeplabv3.yaml

_BASE_: base.yaml
MODEL:
  WEIGHTS: "detectron2://DeepLab/R-103.pkl"
  PIXEL_MEAN: [123.675, 116.280, 103.530]
  PIXEL_STD: [58.395, 57.120, 57.375]
  BACKBONE:
    NAME: "build_resnet_deeplab_backbone"
  RESNETS:
    DEPTH: 101
    NORM: "SyncBN"
    OUT_FEATURES: ["res2", "res5"]
    RES5_MULTI_GRID: [1, 2, 4]
    STEM_TYPE: "deeplab"
    STEM_OUT_CHANNELS: 128
    STRIDE_IN_1X1: False
  SEM_SEG_HEAD:
    NAME: "DeepLabV3PlusHead"
    IN_FEATURES: ["res2", "res5"]
    PROJECT_FEATURES: ["res2"]
    PROJECT_CHANNELS: [48]
    NORM: "SyncBN"
    COMMON_STRIDE: 4
INPUT:
  FORMAT: "RGB"

I ran a job with the following script, per DeepLab Readme:

cd /n/home07/michelewang/thesis/detectron2/projects/DeepLab
python train_net_xbd.py --config-file configs/xBD-configs/base-deeplabv3.yaml --num-gpus 4

However, I got this type mismatcherror and I'm not sure where it's from or how to resolve it. It seems that my image data is in tuple form with the size being 800 pixels but it is not, its 1024x1024 pixel images. Does anyone know why this might be happening?

Traceback (most recent call last):
  File "train_net_xbd.py", line 132, in <module>
    launch(
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/detectron2/engine/launch.py", line 55, in launch
    mp.spawn(
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/detectron2/engine/launch.py", line 94, in _distributed_worker
    main_func(*args)
  File "/n/home07/michelewang/thesis/detectron2/projects/DeepLab/train_net_xbd.py", line 114, in main
    cfg = setup(args)
  File "/n/home07/michelewang/thesis/detectron2/projects/DeepLab/train_net_xbd.py", line 106, in setup
    cfg.merge_from_file(args.config_file)
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/detectron2/config/config.py", line 54, in merge_from_file
    self.merge_from_other_cfg(loaded_cfg)
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/fvcore/common/config.py", line 123, in merge_from_other_cfg
    return super().merge_from_other_cfg(cfg_other)
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/yacs/config.py", line 217, in merge_from_other_cfg
    _merge_a_into_b(cfg_other, self, self, [])
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/yacs/config.py", line 478, in _merge_a_into_b
    _merge_a_into_b(v, b[k], root, key_list + [k])
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/yacs/config.py", line 474, in _merge_a_into_b
    v = _check_and_coerce_cfg_value_type(v, b[k], k, full_key)
  File "/n/home07/michelewang/.conda/envs/active/lib/python3.8/site-packages/yacs/config.py", line 534, in _check_and_coerce_cfg_value_type
    raise ValueError(
ValueError: Type mismatch (<class 'tuple'> vs. <class 'int'>) with values ((800,) vs. 1024) for config key: INPUT.MIN_SIZE_TRAIN

[ppwwyyxx]

MIN_SIZE_TRAIN is a tuple, so if you need one size, it should be (1024,) not (1024)

[michelewang]

Ah, I see, thank you so much! I'll try that.

[michelewang]

Hi @ppwwyyxx, I just got the same issue for some reason even after changing (1024) to (1024,). I'm not sure where the (800,) from the "class tuple" is coming from since all of my images are size 1024 x 1024 -- do you think the error might have something to do with (800,)?

[ppwwyyxx]

I'm not sure where the (800,) from the "class tuple" is coming from

It's the default value of MIN_SIZE_TRAIN