Merge pull request #2229 from fannymonori:gsoc_dnn_superres

* Adding dnn based super resolution module.

* Fixed whitespace error in unit test

* Fixed errors with time measuring functions.

* Updated unit tests in dnn superres

* Deleted unnecessary indents in dnn superres

* Refactored includes in dnn superres

* Moved video upsampling functions to sample code in dnn superres.

* Replaced couts with CV_Error in dnn superres

* Moved benchmarking functionality to sample codes in dnn superres.

* Added performance test to dnn superres

* Resolve buildbot errors

* update dnn_superres

- avoid highgui dependency
- cleanup public API
- use InputArray/OutputArray
- test: avoid legacy test API

Author: fannymonori

modules/README.md

@@ -24,6 +24,8 @@ $ cmake -D OPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules -D BUILD_opencv_<r
 
 - **dnn_objdetect**: Object Detection using CNNs -- Implements compact CNN Model for object detection. Trained using Caffe but uses opencv_dnn modeule.
 
+- **dnn_superres**: Superresolution using CNNs -- Contains four trained convolutional neural networks to upscale images.
+
 - **dnns_easily_fooled**: Subvert DNNs -- This code can use the activations in a network to fool the networks into recognizing something else.
 
 - **dpm**: Deformable Part Model -- Felzenszwalb's Cascade with deformable parts object recognition code.

modules/dnn_superres/CMakeLists.txt

@@ -0,0 +1,5 @@
+set(the_description "Super Resolution using CNNs")
+
+ocv_define_module(dnn_superres opencv_core opencv_imgproc opencv_dnn
+    OPTIONAL opencv_datasets opencv_quality # samples
+)

modules/dnn_superres/README.md

@@ -0,0 +1,93 @@
+# Super Resolution using Convolutional Neural Networks
+
+This module contains several learning-based algorithms for upscaling an image.
+
+## Usage
+
+Run the following command to build this module:
+
+```make
+cmake -DOPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules -Dopencv_dnn_superres=ON <opencv_source_dir>
+```
+
+Refer to the tutorials to understand how to use this module.
+
+## Models
+
+There are four models which are trained.
+
+#### EDSR
+
+Trained models can be downloaded from [here](https://github.com/Saafke/EDSR_Tensorflow/tree/master/models).
+
+- Size of the model: ~38.5MB. This is a quantized version, so that it can be uploaded to GitHub. (Original was 150MB.)
+- This model was trained for 3 days with a batch size of 16
+- Link to implementation code: https://github.com/Saafke/EDSR_Tensorflow
+- x2, x3, x4 trained models available
+- Advantage: Highly accurate
+- Disadvantage: Slow and large filesize
+- Speed: < 3 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
+- Original paper: [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://arxiv.org/pdf/1707.02921.pdf) [1]
+
+#### ESPCN
+
+Trained models can be downloaded from [here](https://github.com/fannymonori/TF-ESPCN/tree/master/export).
+
+- Size of the model: ~100kb
+- This model was trained for ~100 iterations with a batch size of 32
+- Link to implementation code: https://github.com/fannymonori/TF-ESPCN
+- x2, x3, x4 trained models available
+- Advantage: It is tiny and fast, and still performs well.
+- Disadvantage: Perform worse visually than newer, more robust models.
+- Speed: < 0.01 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
+- Original paper: [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](<https://arxiv.org/abs/1609.05158>) [2]
+
+#### FSRCNN
+
+Trained models can be downloaded from [here](https://github.com/Saafke/FSRCNN_Tensorflow/tree/master/models).
+
+- Size of the model: ~40KB (~9kb for FSRCNN-small)
+- This model was trained for ~30 iterations with a batch size of 1
+- Link to implementation code: https://github.com/Saafke/FSRCNN_Tensorflow
+- Advantage: Fast, small and accurate
+- Disadvantage: Not state-of-the-art accuracy
+- Speed: < 0.01 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
+- Notes: FSRCNN-small has fewer parameters, thus less accurate but faster.
+- Original paper: [Accelerating the Super-Resolution Convolutional Neural Network](http://mmlab.ie.cuhk.edu.hk/projects/FSRCNN.html) [3]
+
+#### LapSRN
+
+Trained models can be downloaded from [here](https://github.com/fannymonori/TF-LapSRN/tree/master/export).
+
+- Size of the model: between 1-5Mb
+- This model was trained for ~50 iterations with a batch size of 32
+- Link to implementation code: https://github.com/fannymonori/TF-LAPSRN
+- x2, x4, x8 trained models available
+- Advantage: The model can do multi-scale super-resolution with one forward pass. It can now support 2x, 4x, 8x, and [2x, 4x] and [2x, 4x, 8x] super-resolution.
+- Disadvantage: It is slower than ESPCN and FSRCNN, and the accuracy is worse than EDSR.
+- Speed: < 0.1 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.
+- Original paper: [Deep laplacian pyramid networks for fast and accurate super-resolution](<https://arxiv.org/abs/1710.01992>) [4]
+
+### Benchmarks
+
+Comparing different algorithms. Scale x4 on monarch.png.
+
+|               | Inference time in seconds (CPU)| PSNR | SSIM |
+| ------------- |:-------------------:| ---------:|--------:|
+| ESPCN            |0.01159   | 26.5471 | 0.88116 |
+| EDSR             |3.26758     |**29.2404**  |**0.92112**  |
+| FSRCNN           | 0.01298   | 26.5646 | 0.88064 |
+| LapSRN           |0.28257    |26.7330   |0.88622  |
+| Bicubic          |0.00031 |26.0635  |0.87537  |
+| Nearest neighbor |**0.00014** |23.5628  |0.81741  |
+| Lanczos          |0.00101  |25.9115  |0.87057  |
+
+### References
+[1] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee, **"Enhanced Deep Residual Networks for Single Image Super-Resolution"**, <i> 2nd NTIRE: New Trends in Image Restoration and Enhancement workshop and challenge on image super-resolution in conjunction with **CVPR 2017**. </i> [[PDF](http://openaccess.thecvf.com/content_cvpr_2017_workshops/w12/papers/Lim_Enhanced_Deep_Residual_CVPR_2017_paper.pdf)] [[arXiv](https://arxiv.org/abs/1707.02921)] [[Slide](https://cv.snu.ac.kr/research/EDSR/Presentation_v3(release).pptx)]
+
+[2] Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A., Bishop, R., Rueckert, D. and Wang, Z., **"Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network"**, <i>Proceedings of the IEEE conference on computer vision and pattern recognition</i> **CVPR 2016**. [[PDF](http://openaccess.thecvf.com/content_cvpr_2016/papers/Shi_Real-Time_Single_Image_CVPR_2016_paper.pdf)] [[arXiv](https://arxiv.org/abs/1609.05158)]
+
+[3] Chao Dong, Chen Change Loy, Xiaoou Tang. **"Accelerating the Super-Resolution Convolutional Neural Network"**, <i> in Proceedings of European Conference on Computer Vision </i>**ECCV 2016**. [[PDF](http://personal.ie.cuhk.edu.hk/~ccloy/files/eccv_2016_accelerating.pdf)]
+[[arXiv](https://arxiv.org/abs/1608.00367)] [[Project Page](http://mmlab.ie.cuhk.edu.hk/projects/FSRCNN.html)]
+
+[4] Lai, W. S., Huang, J. B., Ahuja, N., and Yang, M. H., **"Deep laplacian pyramid networks for fast and accurate super-resolution"**, <i> In Proceedings of the IEEE conference on computer vision and pattern recognition </i>**CVPR 2017**. [[PDF](http://openaccess.thecvf.com/content_cvpr_2017/papers/Lai_Deep_Laplacian_Pyramid_CVPR_2017_paper.pdf)] [[arXiv](https://arxiv.org/abs/1710.01992)] [[Project Page](http://vllab.ucmerced.edu/wlai24/LapSRN/)]

modules/dnn_superres/include/opencv2/dnn_superres.hpp

@@ -0,0 +1,125 @@
+// This file is part of OpenCV project.
+// It is subject to the license terms in the LICENSE file found in the top-level directory
+// of this distribution and at http://opencv.org/license.html.
+
+#ifndef _OPENCV_DNN_SUPERRES_HPP_
+#define _OPENCV_DNN_SUPERRES_HPP_
+
+/** @defgroup dnn_superres DNN used for super resolution
+
+This module contains functionality for upscaling an image via convolutional neural networks.
+The following four models are implemented:
+
+- EDSR <https://arxiv.org/abs/1707.02921>
+- ESPCN <https://arxiv.org/abs/1609.05158>
+- FSRCNN <https://arxiv.org/abs/1608.00367>
+- LapSRN <https://arxiv.org/abs/1710.01992>
+
+*/
+
+#include "opencv2/core.hpp"
+#include "opencv2/dnn.hpp"
+
+namespace cv
+{
+namespace dnn_superres
+{
+
+//! @addtogroup dnn_superres
+//! @{
+
+/** @brief A class to upscale images via convolutional neural networks.
+The following four models are implemented:
+
+- edsr
+- espcn
+- fsrcnn
+- lapsrn
+ */
+
+class CV_EXPORTS DnnSuperResImpl
+{
+private:
+
+    /** @brief Net which holds the desired neural network
+     */
+    dnn::Net net;
+
+    std::string alg; //algorithm
+
+    int sc; //scale factor
+
+    void preprocess(InputArray inpImg, OutputArray outpImg);
+
+    void reconstruct_YCrCb(InputArray inpImg, InputArray origImg, OutputArray outpImg, int scale);
+
+    void reconstruct_YCrCb(InputArray inpImg, InputArray origImg, OutputArray outpImg);
+
+    void preprocess_YCrCb(InputArray inpImg, OutputArray outpImg);
+
+public:
+
+    /** @brief Empty constructor
+     */
+    DnnSuperResImpl();
+
+    /** @brief Constructor which immediately sets the desired model
+    @param algo String containing one of the desired models:
+        - __edsr__
+        - __espcn__
+        - __fsrcnn__
+        - __lapsrn__
+    @param scale Integer specifying the upscale factor
+     */
+    DnnSuperResImpl(const std::string& algo, int scale);
+
+    /** @brief Read the model from the given path
+    @param path Path to the model file.
+    */
+    void readModel(const std::string& path);
+
+    /** @brief Read the model from the given path
+    @param weights Path to the model weights file.
+    @param definition Path to the model definition file.
+    */
+    void readModel(const std::string& weights, const std::string& definition);
+
+    /** @brief Set desired model
+    @param algo String containing one of the desired models:
+        - __edsr__
+        - __espcn__
+        - __fsrcnn__
+        - __lapsrn__
+    @param scale Integer specifying the upscale factor
+     */
+    void setModel(const std::string& algo, int scale);
+
+    /** @brief Upsample via neural network
+    @param img Image to upscale
+    @param result Destination upscaled image
+     */
+    void upsample(InputArray img, OutputArray result);
+
+    /** @brief Upsample via neural network of multiple outputs
+    @param img Image to upscale
+    @param imgs_new Destination upscaled images
+    @param scale_factors Scaling factors of the output nodes
+    @param node_names Names of the output nodes in the neural network
+    */
+    void upsampleMultioutput(InputArray img, std::vector<Mat> &imgs_new, const std::vector<int>& scale_factors, const std::vector<String>& node_names);
+
+    /** @brief Returns the scale factor of the model:
+    @return Current scale factor.
+    */
+    int getScale();
+
+    /** @brief Returns the scale factor of the model:
+    @return Current algorithm.
+    */
+    std::string getAlgorithm();
+};
+
+//! @} dnn_superres
+
+}} // cv::dnn_superres::
+#endif

modules/dnn_superres/perf/perf_dnn_superres.cpp

@@ -0,0 +1,48 @@
+// This file is part of OpenCV project.
+// It is subject to the license terms in the LICENSE file found in the top-level directory
+// of this distribution and at http://opencv.org/license.html.
+
+#include "perf_precomp.hpp"
+
+using namespace std;
+using namespace cv;
+using namespace perf;
+
+namespace opencv_test { namespace {
+
+typedef perf::TestBaseWithParam<tuple<tuple<string,string,int>, string> > dnn_superres;
+
+#define MODEL testing::Values(tuple<string,string,int> {"espcn","ESPCN_x2.pb",2}, \
+                                tuple<string,string,int> {"lapsrn","LapSRN_x4.pb",4})
+#define IMAGES testing::Values("cv/dnn_superres/butterfly.png", "cv/shared/baboon.png", "cv/shared/lena.png")
+
+const string TEST_DIR = "cv/dnn_superres";
+
+PERF_TEST_P(dnn_superres, upsample, testing::Combine(MODEL, IMAGES))
+{
+    tuple<string,string,int> model = get<0>( GetParam() );
+    string image_name = get<1>( GetParam() );
+
+    string model_name = get<0>(model);
+    string model_filename = get<1>(model);
+    int scale = get<2>(model);
+
+    string model_path = getDataPath( TEST_DIR + "/" + model_filename );
+    string image_path = getDataPath( image_name );
+
+    DnnSuperResImpl sr;
+    sr.readModel(model_path);
+    sr.setModel(model_name, scale);
+
+    Mat img = imread(image_path);
+
+    Mat img_new(img.rows * scale, img.cols * scale, CV_8UC3);
+
+    declare.in(img, WARMUP_RNG).out(img_new).iterations(10);
+
+    TEST_CYCLE() { sr.upsample(img, img_new); }
+
+    SANITY_CHECK_NOTHING();
+}
+
+}}

modules/dnn_superres/perf/perf_main.cpp

@@ -0,0 +1,7 @@
+// This file is part of OpenCV project.
+// It is subject to the license terms in the LICENSE file found in the top-level directory
+// of this distribution and at http://opencv.org/license.html.
+
+#include "perf_precomp.hpp"
+
+CV_PERF_TEST_MAIN( dnn_superres )

modules/dnn_superres/perf/perf_precomp.hpp

@@ -0,0 +1,15 @@
+// This file is part of OpenCV project.
+// It is subject to the license terms in the LICENSE file found in the top-level directory
+// of this distribution and at http://opencv.org/license.html.
+
+#ifndef __OPENCV_PERF_PRECOMP_HPP__
+#define __OPENCV_PERF_PRECOMP_HPP__
+
+#include "opencv2/ts.hpp"
+#include "opencv2/dnn_superres.hpp"
+
+namespace opencv_test {
+    using namespace cv::dnn_superres;
+}
+
+#endif