How to set input format if input is a list when exporting a model to onnx by detectron2?

[4daJKong]

The problem is in My model, it take ViT as backbone and GeneralizedRCNN as object detection from detectron2. It use List[dict_1{"image": , "height": , "width": }, dict_2{ "image": , "height": , "width": } ...] as input, and can get results from model, the output also return List which includes, [ {"instances": Instances()}, {"instances": Instances()}...]

Firstly, I tried torch.onnx.export to export onnx, but when I use same format to onnx, it only can get dict_1{} as input, and shows error

from detectron2.modeling import build_model
model_2 = build_model(cfg)
from detectron2.checkpoint import DetectionCheckpointer
checkpointer = DetectionCheckpointer(model)
checkpointer.load(cfg.MODEL.WEIGHTS)
input_1 = {"image": torch.randn(3,1132,800), "height": 1200, "width": 1000}
input_2 = {"image": torch.randn(3,1132,800), "height": 1200, "width": 1000}
prediction = model_2([input_1, input_2]) #it can get results

torch.onnx.export(
        model_2,
        [input_1, input_2], #actually, it only get input_1, a dict, as input
        "m2.onnx"
    )

Thus, I followed some tutor in detectron2 by export_onnx_model

    import onnx
    from detectron2.export import export_onnx_model
    input = [input_0, input_2]
    with torch.no_grad(): 
        onnxModel = export_onnx_model(cfg, model, input)
        onnx.save(onnxModel,"cdla_pred_beta.onnx")

The error shows, RuntimeError:[enforce fail at context_gpu.cu:240] . out of memory I don't know whether the problem is on my GPU capacity or my input and is it possible to use detectron2 under this input format?

P.S.

torch = 1.10.0
detectron2 = 0.6+cu102

[bouachalazhar]

I test many code but nothing work.

The problem is in My model, it take ViT as backbone and GeneralizedRCNN as object detection from detectron2. It use List[dict_1{"image": , "height": , "width": }, dict_2{ "image": , "height": , "width": } ...] as input, and can get results from model, the output also return List which includes, [ {"instances": Instances()}, {"instances": Instances()}...]

Firstly, I tried torch.onnx.export to export onnx, but when I use same format to onnx, it only can get dict_1{} as input, and shows error

from detectron2.modeling import build_model
model_2 = build_model(cfg)
from detectron2.checkpoint import DetectionCheckpointer
checkpointer = DetectionCheckpointer(model)
checkpointer.load(cfg.MODEL.WEIGHTS)
input_1 = {"image": torch.randn(3,1132,800), "height": 1200, "width": 1000}
input_2 = {"image": torch.randn(3,1132,800), "height": 1200, "width": 1000}
prediction = model_2([input_1, input_2]) #it can get results

torch.onnx.export(
        model_2,
        [input_1, input_2], #actually, it only get input_1, a dict, as input
        "m2.onnx"
    )

Thus, I followed some tutor in detectron2 by export_onnx_model

    import onnx
    from detectron2.export import export_onnx_model
    input = [input_0, input_2]
    with torch.no_grad(): 
        onnxModel = export_onnx_model(cfg, model, input)
        onnx.save(onnxModel,"cdla_pred_beta.onnx")

The error shows, RuntimeError:[enforce fail at context_gpu.cu:240] . out of memory I don't know whether the problem is on my GPU capacity or my input and is it possible to use detectron2 under this input format?

P.S.

torch = 1.10.0
detectron2 = 0.6+cu102

I test many code but nothing work.

[4daJKong]

Hi, I found some cues in detectron2 website, maybe helpful for my exporting
In detectron2/tests/test_export_onnx.py in line 86

    @skipIfUnsupportedMinOpsetVersion(16, STABLE_ONNX_OPSET_VERSION)
    @skipIfUnsupportedMinTorchVersion("1.11.1")
    def testMaskRCNNFPN_with_postproc(self):
        def inference_func(model, image):
            inputs = [{"image": image, "height": image.shape[1], "width": image.shape[2]}]
            return model.inference(inputs, do_postprocess=True)[0]["instances"]

        self._test_model_zoo_from_config_path(
            "COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml",
            inference_func,
        )

my inputs is same as aboved one, so it can work in code by,

with torch.no_grad():
    model_2.inference(inputs, do_postprocess=True)[0]["instances"]

Thus, I rewrite some part of code in my way,

  def inference_func(model, image):
       inputs = [{"image": image, "height": image.shape[1], "width": image.shape[2]}]
       return model.inference(inputs, do_postprocess=True)[0]["instances"]

   from detectron2.export import TracingAdapter
   from detectron2.utils.testing import get_sample_coco_image
   batch = 1
   image = get_sample_coco_image()
   image = image.to(device=torch.device('cuda:1')) # torch.Size([3, 480, 640])
   inputs = tuple(image.clone() for _ in range(batch))
   adapter_model = TracingAdapter(model_2, inputs, inference_func)   
   adapter_model.eval()
   # from detectron2.export import STABLE_ONNX_OPSET_VERSION
   with torch.no_grad():
       torch.onnx.export(
                   adapter_model,
                   adapter_model.flattened_inputs,
                   "model_2.onnx",
                   opset_version=11,
                   verbose=True,
       )

The problem is on opset_version, Exporting the operator amax to ONNX opset version 11 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.

Besides, another cue is on detectron2/tools/deploy/export_model.py line 40

    from detectron2.export import Caffe2Tracer
    tracer = Caffe2Tracer(cfg, model_2, inputs)
    import onnx
    onnx_model = tracer.export_onnx()
    onnx.save(onnx_model, "model.onnx")

I tried aboved code but it still shows RuntimeError:[enforce fail at context_gpu.cu:240] . out of memory

[bouachalazhar]

Caffe2Tracer is deprecated you should be use detectron2/tools/deploy/export_model.py but I test it and it doesn't work because I would like to use export PyTorch MaskRCNN --> ONNX --> OpenVINO MaskRCNN. I think input and output good written are a part of solution.

[4daJKong]

Thanks for your reply, I’ll tried your method but the difficulty is on my backbone network with a (feature pyramid network) FPN, which are imported by detectron2, it combine these features and then forward to rcnn..., I think it is hard to rewrite all of it..

[bouachalazhar]

Thanks for your reply, I’ll tried your method but the difficulty is on my backbone network with a (feature pyramid network) FPN, which are imported by detectron2, it combine these features and then forward to rcnn..., I think it is hard to rewrite all of it..

You can export maskrcnn and FPN separately or check if maskrcnn is GeneralizedRCNN module.

[4daJKong]

Not successed yet, but I think it is close, I followed instruction in detectron2.export import TracingAdapter from export_model.py

    def inference_func(model, image):
        dict_input = {"image": image, "height": image.shape[1], "width": image.shape[2]}
        return model.inference([dict_input], do_postprocess=False)
    from detectron2.export import TracingAdapter
    with torch.no_grad():
        adapter_model = TracingAdapter(model, image, inference_func)   
        traced = torch.jit.trace(adapter_model, adapter_model.flattened_inputs)
        flattened_outputs = traced(*adapter_model.flattened_inputs)  #No error so far 

        from detectron2.utils.file_io import PathManager
        with PathManager.open("_beta.onnx", "wb") as f:    
            torch.onnx.export(
                        adapter_model,
                        adapter_model.flattened_inputs,  #input = (image, )     
                        "model_beta.onnx",
                        # training=torch.onnx.TrainingMode.EVAL,
                        opset_version=11,
                        verbose=True,
            )

However, it shows error in torch.onnx.export,

'torch._C.Value' object is not iterable
(Occurred when translating repeat_interleave).

[bouachalazhar]

Not successed yet, but I think it is close, I followed instruction in detectron2.export import TracingAdapter from export_model.py

    def inference_func(model, image):

        dict_input = {"image": image, "height": image.shape[1], "width": image.shape[2]}

        return model.inference([dict_input], do_postprocess=False)

    from detectron2.export import TracingAdapter

    with torch.no_grad():

        adapter_model = TracingAdapter(model, image, inference_func)   

        traced = torch.jit.trace(adapter_model, adapter_model.flattened_inputs)

        flattened_outputs = traced(*adapter_model.flattened_inputs)  #No error so far 



        from detectron2.utils.file_io import PathManager

        with PathManager.open("_beta.onnx", "wb") as f:    

            torch.onnx.export(

                        adapter_model,

                        adapter_model.flattened_inputs,  #input = (image, )     

                        "model_beta.onnx",

                        # training=torch.onnx.TrainingMode.EVAL,

                        opset_version=11,

                        verbose=True,

            )

However, it shows error in torch.onnx.export,

'torch._C.Value' object is not iterable

(Occurred when translating repeat_interleave).

You can check type of each parameters and eventually their shape.