|
| 1 | +# Conversion of PyTorch Classification Models and Launch with OpenCV C++ {#pytorch_cls_c_tutorial_dnn_conversion} |
| 2 | + |
| 3 | +@prev_tutorial{pytorch_cls_tutorial_dnn_conversion} |
| 4 | + |
| 5 | +| | | |
| 6 | +| -: | :- | |
| 7 | +| Original author | Anastasia Murzova | |
| 8 | +| Compatibility | OpenCV >= 4.5 | |
| 9 | + |
| 10 | +## Goals |
| 11 | +In this tutorial you will learn how to: |
| 12 | +* convert PyTorch classification models into ONNX format |
| 13 | +* run converted PyTorch model with OpenCV C/C++ API |
| 14 | +* provide model inference |
| 15 | + |
| 16 | +We will explore the above-listed points by the example of ResNet-50 architecture. |
| 17 | + |
| 18 | +## Introduction |
| 19 | +Let's briefly view the key concepts involved in the pipeline of PyTorch models transition with OpenCV API. The initial step in conversion of PyTorch models into cv::dnn::Net |
| 20 | +is model transferring into [ONNX](https://onnx.ai/about.html) format. ONNX aims at the interchangeability of the neural networks between various frameworks. There is a built-in function in PyTorch for ONNX conversion: [``torch.onnx.export``](https://pytorch.org/docs/stable/onnx.html#torch.onnx.export). |
| 21 | +Further the obtained ``.onnx`` model is passed into cv::dnn::readNetFromONNX or cv::dnn::readNet. |
| 22 | + |
| 23 | +## Requirements |
| 24 | +To be able to experiment with the below code you will need to install a set of libraries. We will use a virtual environment with python3.7+ for this: |
| 25 | + |
| 26 | +```console |
| 27 | +virtualenv -p /usr/bin/python3.7 <env_dir_path> |
| 28 | +source <env_dir_path>/bin/activate |
| 29 | +``` |
| 30 | + |
| 31 | +For OpenCV-Python building from source, follow the corresponding instructions from the @ref tutorial_py_table_of_contents_setup. |
| 32 | + |
| 33 | +Before you start the installation of the libraries, you can customize the [requirements.txt](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/requirements.txt), excluding or including (for example, ``opencv-python``) some dependencies. |
| 34 | +The below line initiates requirements installation into the previously activated virtual environment: |
| 35 | + |
| 36 | +```console |
| 37 | +pip install -r requirements.txt |
| 38 | +``` |
| 39 | + |
| 40 | +## Practice |
| 41 | +In this part we are going to cover the following points: |
| 42 | +1. create a classification model conversion pipeline |
| 43 | +2. provide the inference, process prediction results |
| 44 | + |
| 45 | +### Model Conversion Pipeline |
| 46 | +The code in this subchapter is located in the ``samples/dnn/dnn_model_runner`` module and can be executed with the line: |
| 47 | + |
| 48 | +```console |
| 49 | +python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_resnet50_onnx |
| 50 | +``` |
| 51 | + |
| 52 | +The following code contains the description of the below-listed steps: |
| 53 | +1. instantiate PyTorch model |
| 54 | +2. convert PyTorch model into ``.onnx`` |
| 55 | + |
| 56 | +```python |
| 57 | +# initialize PyTorch ResNet-50 model |
| 58 | +original_model = models.resnet50(pretrained=True) |
| 59 | + |
| 60 | +# get the path to the converted into ONNX PyTorch model |
| 61 | +full_model_path = get_pytorch_onnx_model(original_model) |
| 62 | +print("PyTorch ResNet-50 model was successfully converted: ", full_model_path) |
| 63 | +``` |
| 64 | + |
| 65 | +``get_pytorch_onnx_model(original_model)`` function is based on ``torch.onnx.export(...)`` call: |
| 66 | + |
| 67 | +```python |
| 68 | +# define the directory for further converted model save |
| 69 | +onnx_model_path = "models" |
| 70 | +# define the name of further converted model |
| 71 | +onnx_model_name = "resnet50.onnx" |
| 72 | + |
| 73 | +# create directory for further converted model |
| 74 | +os.makedirs(onnx_model_path, exist_ok=True) |
| 75 | + |
| 76 | +# get full path to the converted model |
| 77 | +full_model_path = os.path.join(onnx_model_path, onnx_model_name) |
| 78 | + |
| 79 | +# generate model input |
| 80 | +generated_input = Variable( |
| 81 | + torch.randn(1, 3, 224, 224) |
| 82 | +) |
| 83 | + |
| 84 | +# model export into ONNX format |
| 85 | +torch.onnx.export( |
| 86 | + original_model, |
| 87 | + generated_input, |
| 88 | + full_model_path, |
| 89 | + verbose=True, |
| 90 | + input_names=["input"], |
| 91 | + output_names=["output"], |
| 92 | + opset_version=11 |
| 93 | +) |
| 94 | +``` |
| 95 | + |
| 96 | +After the successful execution of the above code we will get the following output: |
| 97 | + |
| 98 | +```console |
| 99 | +PyTorch ResNet-50 model was successfully converted: models/resnet50.onnx |
| 100 | +``` |
| 101 | + |
| 102 | +The proposed in ``dnn/samples`` module ``dnn_model_runner`` allows us to reproduce the above conversion steps for the following PyTorch classification models: |
| 103 | +* alexnet |
| 104 | +* vgg11 |
| 105 | +* vgg13 |
| 106 | +* vgg16 |
| 107 | +* vgg19 |
| 108 | +* resnet18 |
| 109 | +* resnet34 |
| 110 | +* resnet50 |
| 111 | +* resnet101 |
| 112 | +* resnet152 |
| 113 | +* squeezenet1_0 |
| 114 | +* squeezenet1_1 |
| 115 | +* resnext50_32x4d |
| 116 | +* resnext101_32x8d |
| 117 | +* wide_resnet50_2 |
| 118 | +* wide_resnet101_2 |
| 119 | + |
| 120 | +To obtain the converted model, the following line should be executed: |
| 121 | + |
| 122 | +``` |
| 123 | +python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name <pytorch_cls_model_name> --evaluate False |
| 124 | +``` |
| 125 | + |
| 126 | +For the ResNet-50 case the below line should be run: |
| 127 | + |
| 128 | +``` |
| 129 | +python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name resnet50 --evaluate False |
| 130 | +``` |
| 131 | + |
| 132 | +The default root directory for the converted model storage is defined in module ``CommonConfig``: |
| 133 | + |
| 134 | +```python |
| 135 | +@dataclass |
| 136 | +class CommonConfig: |
| 137 | + output_data_root_dir: str = "dnn_model_runner/dnn_conversion" |
| 138 | +``` |
| 139 | + |
| 140 | +Thus, the converted ResNet-50 will be saved in ``dnn_model_runner/dnn_conversion/models``. |
| 141 | + |
| 142 | +### Inference Pipeline |
| 143 | +Now we can use ```models/resnet50.onnx``` for the inference pipeline using OpenCV C/C++ API. The implemented pipeline can be found in [samples/dnn/classification.cpp](https://github.com/opencv/opencv/blob/master/samples/dnn/classification.cpp). |
| 144 | +After the build of samples (``BUILD_EXAMPLES`` flag value should be ``ON``), the appropriate ``example_dnn_classification`` executable file will be provided. |
| 145 | + |
| 146 | +To provide model inference we will use the below [squirrel photo](https://www.pexels.com/photo/brown-squirrel-eating-1564292) (under [CC0](https://www.pexels.com/terms-of-service/) license) corresponding to ImageNet class ID 335: |
| 147 | +```console |
| 148 | +fox squirrel, eastern fox squirrel, Sciurus niger |
| 149 | +``` |
| 150 | + |
| 151 | + |
| 152 | + |
| 153 | +For the label decoding of the obtained prediction, we also need ``imagenet_classes.txt`` file, which contains the full list of the ImageNet classes. |
| 154 | + |
| 155 | +In this tutorial we will run the inference process for the converted PyTorch ResNet-50 model from the build (``samples/build``) directory: |
| 156 | + |
| 157 | +``` |
| 158 | +./dnn/example_dnn_classification --model=../dnn/models/resnet50.onnx --input=../data/squirrel_cls.jpg --width=224 --height=224 --rgb=true --scale="0.003921569" --mean="123.675 116.28 103.53" --std="0.229 0.224 0.225" --crop=true --initial_width=256 --initial_height=256 --classes=../data/dnn/classification_classes_ILSVRC2012.txt |
| 159 | +``` |
| 160 | + |
| 161 | +Let's explore ``classification.cpp`` key points step by step: |
| 162 | + |
| 163 | +1. read the model with cv::dnn::readNet, initialize the network: |
| 164 | + |
| 165 | +```cpp |
| 166 | +Net net = readNet(model, config, framework); |
| 167 | +``` |
| 168 | + |
| 169 | +The ``model`` parameter value is taken from ``--model`` key. In our case, it is ``resnet50.onnx``. |
| 170 | + |
| 171 | +* preprocess input image: |
| 172 | + |
| 173 | +```cpp |
| 174 | +if (rszWidth != 0 && rszHeight != 0) |
| 175 | +{ |
| 176 | + resize(frame, frame, Size(rszWidth, rszHeight)); |
| 177 | +} |
| 178 | + |
| 179 | +// Create a 4D blob from a frame |
| 180 | +blobFromImage(frame, blob, scale, Size(inpWidth, inpHeight), mean, swapRB, crop); |
| 181 | + |
| 182 | +// Check std values. |
| 183 | +if (std.val[0] != 0.0 && std.val[1] != 0.0 && std.val[2] != 0.0) |
| 184 | +{ |
| 185 | + // Divide blob by std. |
| 186 | + divide(blob, std, blob); |
| 187 | +} |
| 188 | +``` |
| 189 | +
|
| 190 | +In this step we use cv::dnn::blobFromImage function to prepare model input. |
| 191 | +We set ``Size(rszWidth, rszHeight)`` with ``--initial_width=256 --initial_height=256`` for the initial image resize as it's described in [PyTorch ResNet inference pipeline](https://pytorch.org/hub/pytorch_vision_resnet/). |
| 192 | +
|
| 193 | +It should be noted that firstly in cv::dnn::blobFromImage mean value is subtracted and only then pixel values are multiplied by scale. |
| 194 | +Thus, we use ``--mean="123.675 116.28 103.53"``, which is equivalent to ``[0.485, 0.456, 0.406]`` multiplied by ``255.0`` to reproduce the original image preprocessing order for PyTorch classification models: |
| 195 | +
|
| 196 | +```python |
| 197 | +img /= 255.0 |
| 198 | +img -= [0.485, 0.456, 0.406] |
| 199 | +img /= [0.229, 0.224, 0.225] |
| 200 | +``` |
| 201 | + |
| 202 | +* make forward pass: |
| 203 | + |
| 204 | +```cpp |
| 205 | +net.setInput(blob); |
| 206 | +Mat prob = net.forward(); |
| 207 | +``` |
| 208 | + |
| 209 | +* process the prediction: |
| 210 | + |
| 211 | +```cpp |
| 212 | +Point classIdPoint; |
| 213 | +double confidence; |
| 214 | +minMaxLoc(prob.reshape(1, 1), 0, &confidence, 0, &classIdPoint); |
| 215 | +int classId = classIdPoint.x; |
| 216 | +``` |
| 217 | +
|
| 218 | +Here we choose the most likely object class. The ``classId`` result for our case is 335 - fox squirrel, eastern fox squirrel, Sciurus niger: |
| 219 | +
|
| 220 | + |
0 commit comments