Main approaches tried for DeepLabv3 Core ML conversion

[himanshunaidu]

Hello All,
So since the day before I have been trying several approaches for converting DeepLabv3 Pytorch model to CoreML.

I wanted to document all the important parts of this in a public page so that this can be discussed on in detail.

[himanshunaidu]

Devices used for CoreML conversion

I ran this code in multiple environments, the main ones being:

1. MacBook Pro macOS Apple M2 Ventura 13.5.2
2. AWS EC2 Ubuntu instance Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz with NVIDIA Corporation TU104GL [Tesla T4]

As it turns out, based on my experiments, running the conversion on Mac worked better. Not ideal, but it isn't surprising.

[himanshunaidu]

Approach 1: Reproduce the from_deeplab.py code from Ihor's deeplabv3coreml repository

Link: https://github.com/NaturalStupidlty/deeplabv3coreml/blob/main/from_deeplab.py

This file tries to use the DeepLabv3 model trained on CityScapes from the DeepLabV3Plus-Pytorch repository

Result

While this successfully ran the conversion, the CoreML model performance ended up being significantly worse.

Original Test Image:

Pytorch Model Output:

CoreML Model Output:

[himanshunaidu]

The output looks even worse when running this in the app on the camera live feed. It mostly looks like random noise.

This issue happened in both environments.

[himanshunaidu]

Approach 2: Reproduce the from_torchvision.py code from Ihor's deeplabv3coreml repository

Link: https://github.com/NaturalStupidlty/deeplabv3coreml/blob/main/from_torchvision.py

This file tries to use the DeepLabv3 models of different weights from the torchvision package

Result

While this successfully ran the conversion, the CoreML model performance ended up being significantly worse.

(I tried this for all the 3 variations of the deepLabv3 weights present in the from_torchvision.py file.)

Original Test Image:

PyTorch Model Output:

CoreML Model Output:

[himanshunaidu]

This issue happened in both environments.

[himanshunaidu]

Approach 3: Use the models provided by Apple

Link: https://developer.apple.com/machine-learning/models/

Utilized the DeepLabV3.mlmodel file.

I converted the MLMultiArray output to a UIImage using code from the following link:
https://dmytrohrebeniuk.medium.com/convert-mlmultiarray-to-image-for-pytorch-models-without-performance-lags-df93de9d2d46

Code used was similar to this: coreml_convet_spec copy.txt

Result

This is the only model that seems to have worked.

Original Test Image:

CoreML Model Output:

However, this doesn't solve the CoreML conversion problem we are having, since this approach involves using an Out-Of-Box model.

[himanshunaidu]

Approach 4: Use the Basic version of official Apple documentation and build up from there

Tutorial Link:
https://apple.github.io/coremltools/docs-guides/source/convert-a-pytorch-segmentation-model.html

The default DeepLabv3 model from the tutorial works, however, we want a model that can be used as a VNCoreMLModel. The tutorial model accepts MLMultiArray, and we want a model that accepts Image. It would also be necessary for it to output an image, for performance as well as code convenience.

It is possible to change the input format by changing the conversion logic. We must remember to normalize the image:
https://apple.github.io/coremltools/docs-guides/source/image-inputs.html

However, we would also like to change the output. When we change the output by taking into account the argmax requirement, we end up getting a failed compile, with the following error:
RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: Encountered an error while compiling a neural network model: validator error: main output 'output' must be a tensor of type Float32 or Float16".

[himanshunaidu]

To note, there were dozens of other configurations tried, however, none of them seemed to have worked.

The Apple documentation doesn't seem to have anything about this case, which is weird since they have clearly worked with conversion of DeepLabv3 model that accepts Image input. So it would have been prudent for them to document the process and make the documentation public.