Skip to content

Commit 8c22198

Browse files
author
Anastasia M
authored
Merge pull request opencv#17604 from LupusSanctus:am/pytorch_tf_cls_tutorial
[GSoC] Added TF and PyTorch classification conversion cases * Added TF and PyTorch classification conversion cases * Modified structure, some processing scripts. Added evaluation pipeline * Minor structure change * Removed extra functions, minor structure change * Modified structure, code corrections * Updated classification code block, added classification tutorials * Added minor modifications of paths * Classification block corrections in accordance with comments
1 parent e85b41f commit 8c22198

29 files changed

+2463
-98
lines changed
Loading
Loading
Loading
Loading
Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# Conversion of PyTorch Classification Models and Launch with OpenCV C++ {#pytorch_cls_c_tutorial_dnn_conversion}
2+
3+
@prev_tutorial{pytorch_cls_tutorial_dnn_conversion}
4+
5+
| | |
6+
| -: | :- |
7+
| Original author | Anastasia Murzova |
8+
| Compatibility | OpenCV >= 4.5 |
9+
10+
## Goals
11+
In this tutorial you will learn how to:
12+
* convert PyTorch classification models into ONNX format
13+
* run converted PyTorch model with OpenCV C/C++ API
14+
* provide model inference
15+
16+
We will explore the above-listed points by the example of ResNet-50 architecture.
17+
18+
## Introduction
19+
Let's briefly view the key concepts involved in the pipeline of PyTorch models transition with OpenCV API. The initial step in conversion of PyTorch models into cv::dnn::Net
20+
is model transferring into [ONNX](https://onnx.ai/about.html) format. ONNX aims at the interchangeability of the neural networks between various frameworks. There is a built-in function in PyTorch for ONNX conversion: [``torch.onnx.export``](https://pytorch.org/docs/stable/onnx.html#torch.onnx.export).
21+
Further the obtained ``.onnx`` model is passed into cv::dnn::readNetFromONNX or cv::dnn::readNet.
22+
23+
## Requirements
24+
To be able to experiment with the below code you will need to install a set of libraries. We will use a virtual environment with python3.7+ for this:
25+
26+
```console
27+
virtualenv -p /usr/bin/python3.7 <env_dir_path>
28+
source <env_dir_path>/bin/activate
29+
```
30+
31+
For OpenCV-Python building from source, follow the corresponding instructions from the @ref tutorial_py_table_of_contents_setup.
32+
33+
Before you start the installation of the libraries, you can customize the [requirements.txt](https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/requirements.txt), excluding or including (for example, ``opencv-python``) some dependencies.
34+
The below line initiates requirements installation into the previously activated virtual environment:
35+
36+
```console
37+
pip install -r requirements.txt
38+
```
39+
40+
## Practice
41+
In this part we are going to cover the following points:
42+
1. create a classification model conversion pipeline
43+
2. provide the inference, process prediction results
44+
45+
### Model Conversion Pipeline
46+
The code in this subchapter is located in the ``samples/dnn/dnn_model_runner`` module and can be executed with the line:
47+
48+
```console
49+
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_resnet50_onnx
50+
```
51+
52+
The following code contains the description of the below-listed steps:
53+
1. instantiate PyTorch model
54+
2. convert PyTorch model into ``.onnx``
55+
56+
```python
57+
# initialize PyTorch ResNet-50 model
58+
original_model = models.resnet50(pretrained=True)
59+
60+
# get the path to the converted into ONNX PyTorch model
61+
full_model_path = get_pytorch_onnx_model(original_model)
62+
print("PyTorch ResNet-50 model was successfully converted: ", full_model_path)
63+
```
64+
65+
``get_pytorch_onnx_model(original_model)`` function is based on ``torch.onnx.export(...)`` call:
66+
67+
```python
68+
# define the directory for further converted model save
69+
onnx_model_path = "models"
70+
# define the name of further converted model
71+
onnx_model_name = "resnet50.onnx"
72+
73+
# create directory for further converted model
74+
os.makedirs(onnx_model_path, exist_ok=True)
75+
76+
# get full path to the converted model
77+
full_model_path = os.path.join(onnx_model_path, onnx_model_name)
78+
79+
# generate model input
80+
generated_input = Variable(
81+
torch.randn(1, 3, 224, 224)
82+
)
83+
84+
# model export into ONNX format
85+
torch.onnx.export(
86+
original_model,
87+
generated_input,
88+
full_model_path,
89+
verbose=True,
90+
input_names=["input"],
91+
output_names=["output"],
92+
opset_version=11
93+
)
94+
```
95+
96+
After the successful execution of the above code we will get the following output:
97+
98+
```console
99+
PyTorch ResNet-50 model was successfully converted: models/resnet50.onnx
100+
```
101+
102+
The proposed in ``dnn/samples`` module ``dnn_model_runner`` allows us to reproduce the above conversion steps for the following PyTorch classification models:
103+
* alexnet
104+
* vgg11
105+
* vgg13
106+
* vgg16
107+
* vgg19
108+
* resnet18
109+
* resnet34
110+
* resnet50
111+
* resnet101
112+
* resnet152
113+
* squeezenet1_0
114+
* squeezenet1_1
115+
* resnext50_32x4d
116+
* resnext101_32x8d
117+
* wide_resnet50_2
118+
* wide_resnet101_2
119+
120+
To obtain the converted model, the following line should be executed:
121+
122+
```
123+
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name <pytorch_cls_model_name> --evaluate False
124+
```
125+
126+
For the ResNet-50 case the below line should be run:
127+
128+
```
129+
python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name resnet50 --evaluate False
130+
```
131+
132+
The default root directory for the converted model storage is defined in module ``CommonConfig``:
133+
134+
```python
135+
@dataclass
136+
class CommonConfig:
137+
output_data_root_dir: str = "dnn_model_runner/dnn_conversion"
138+
```
139+
140+
Thus, the converted ResNet-50 will be saved in ``dnn_model_runner/dnn_conversion/models``.
141+
142+
### Inference Pipeline
143+
Now we can use ```models/resnet50.onnx``` for the inference pipeline using OpenCV C/C++ API. The implemented pipeline can be found in [samples/dnn/classification.cpp](https://github.com/opencv/opencv/blob/master/samples/dnn/classification.cpp).
144+
After the build of samples (``BUILD_EXAMPLES`` flag value should be ``ON``), the appropriate ``example_dnn_classification`` executable file will be provided.
145+
146+
To provide model inference we will use the below [squirrel photo](https://www.pexels.com/photo/brown-squirrel-eating-1564292) (under [CC0](https://www.pexels.com/terms-of-service/) license) corresponding to ImageNet class ID 335:
147+
```console
148+
fox squirrel, eastern fox squirrel, Sciurus niger
149+
```
150+
151+
![Classification model input image](images/squirrel_cls.jpg)
152+
153+
For the label decoding of the obtained prediction, we also need ``imagenet_classes.txt`` file, which contains the full list of the ImageNet classes.
154+
155+
In this tutorial we will run the inference process for the converted PyTorch ResNet-50 model from the build (``samples/build``) directory:
156+
157+
```
158+
./dnn/example_dnn_classification --model=../dnn/models/resnet50.onnx --input=../data/squirrel_cls.jpg --width=224 --height=224 --rgb=true --scale="0.003921569" --mean="123.675 116.28 103.53" --std="0.229 0.224 0.225" --crop=true --initial_width=256 --initial_height=256 --classes=../data/dnn/classification_classes_ILSVRC2012.txt
159+
```
160+
161+
Let's explore ``classification.cpp`` key points step by step:
162+
163+
1. read the model with cv::dnn::readNet, initialize the network:
164+
165+
```cpp
166+
Net net = readNet(model, config, framework);
167+
```
168+
169+
The ``model`` parameter value is taken from ``--model`` key. In our case, it is ``resnet50.onnx``.
170+
171+
* preprocess input image:
172+
173+
```cpp
174+
if (rszWidth != 0 && rszHeight != 0)
175+
{
176+
resize(frame, frame, Size(rszWidth, rszHeight));
177+
}
178+
179+
// Create a 4D blob from a frame
180+
blobFromImage(frame, blob, scale, Size(inpWidth, inpHeight), mean, swapRB, crop);
181+
182+
// Check std values.
183+
if (std.val[0] != 0.0 && std.val[1] != 0.0 && std.val[2] != 0.0)
184+
{
185+
// Divide blob by std.
186+
divide(blob, std, blob);
187+
}
188+
```
189+
190+
In this step we use cv::dnn::blobFromImage function to prepare model input.
191+
We set ``Size(rszWidth, rszHeight)`` with ``--initial_width=256 --initial_height=256`` for the initial image resize as it's described in [PyTorch ResNet inference pipeline](https://pytorch.org/hub/pytorch_vision_resnet/).
192+
193+
It should be noted that firstly in cv::dnn::blobFromImage mean value is subtracted and only then pixel values are multiplied by scale.
194+
Thus, we use ``--mean="123.675 116.28 103.53"``, which is equivalent to ``[0.485, 0.456, 0.406]`` multiplied by ``255.0`` to reproduce the original image preprocessing order for PyTorch classification models:
195+
196+
```python
197+
img /= 255.0
198+
img -= [0.485, 0.456, 0.406]
199+
img /= [0.229, 0.224, 0.225]
200+
```
201+
202+
* make forward pass:
203+
204+
```cpp
205+
net.setInput(blob);
206+
Mat prob = net.forward();
207+
```
208+
209+
* process the prediction:
210+
211+
```cpp
212+
Point classIdPoint;
213+
double confidence;
214+
minMaxLoc(prob.reshape(1, 1), 0, &confidence, 0, &classIdPoint);
215+
int classId = classIdPoint.x;
216+
```
217+
218+
Here we choose the most likely object class. The ``classId`` result for our case is 335 - fox squirrel, eastern fox squirrel, Sciurus niger:
219+
220+
![ResNet50 OpenCV C++ inference output](images/opencv_resnet50_test_res_c.jpg)

0 commit comments

Comments
 (0)