[GSoC22] Data Augmentation Module in OpenCV (imgaug) #3335

ZhaoChuyang · 2022-08-24T03:29:44Z

PR for GSoC'22 project on Efficient Data Augmentation Module in OpenCV for DL Training

I implemented data augmentation methods based on basic image processing and have tested the performance of these already implemented methods in python enviroment. The following table demonstrates the running time (in seconds) comparison between OpenCV-Aug module and torchvision transforms on a subset of ImageNet:

single method:

method	config	dataset	opencv	torchvision
resize	size: (200, 200)	imagenet-320	0.388	1.745
center crop	size: (200, 200)	imagenet-320	0.001	0.017
pad	padding: (100, 100, 100, 100)	imagenet-320	0.141	0.553
random crop	size: (200, 200)	imagenet-320	0.001	0.028
random resized crop	size: (500, 500)	imagenet-320	0.368	3.806
random flip	default	imagenet-320	0.046	0.519

compose multiple methods:

method	config	num_augs	opencv	pytorch
RandomCrop + RandomFlip + Pad	RandomCrop: size(300, 300) Pad: padding(100, 100, 100, 100)	3	0.481	1.240
Resize + Pad + RandomFlip + CenterCrop	Resize: size(400, 400) Pad: padding(100, 100, 100, 100) CenterCrop: size(200, 200)	5	0.636	5.486

Besides augmentation methods for pure images, augmentation methods for detection task and segmentation task is also added, which requires processing the target labels of corresponding tasks.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

kaingwade

Please add the short-form license and the documentation for the functionalites.

kaingwade · 2022-08-26T07:53:49Z

modules/aug/samples/opencv_aug_demo.py

+
+    image = compose(image)
+
+    plt.imshow(image)


Can you call cv2.imshow() here to show the result?

OK, I will change the python sample in next commit.

kaingwade · 2022-08-26T08:14:25Z

modules/aug/src/transforms.cpp

+//    CV_EXPORTS_W void randomCropV1(InputOutputArray _src, const Size& sz, const Vec4i& padding, bool pad_if_need, int fill, int padding_mode){
+//        Mat src = _src.getMat();
+//
+//        if(padding != Vec4i()){
+//            copyMakeBorder(src, src, padding[0], padding[1], padding[2], padding[3], padding_mode, fill);
+//        }
+//
+//        // NOTE: make sure src.rows == src.size().height and src.cols = src.size().width
+//        // pad the height if needed
+//        if(pad_if_need && src.rows < sz.height){
+//            Vec4i _padding = {sz.height - src.rows, sz.height - src.rows, 0, 0};
+//            copyMakeBorder(src, src, _padding[0], _padding[1], _padding[2], _padding[3], padding_mode, fill);
+//        }
+//        // pad the width if needed
+//        if(pad_if_need && src.cols < sz.width){
+//            Vec4i _padding = {0, 0, sz.width - src.cols, sz.width - src.cols};
+//            copyMakeBorder(src, src, _padding[0], _padding[1], _padding[2], _padding[3], padding_mode, fill);
+//        }
+//
+//        int x, y;
+//        getRandomCropParams(src.rows, src.cols, sz.height, sz.width, &x, &y);
+//        Mat cropped(src, Rect(x, y, sz.width, sz.height));
+//        (*(Mat*)_src.getObj()) = cropped;
+//    }


Please remove code which are not finished, experimental, for debug and the like.

ZhaoChuyang · 2022-08-29T12:22:52Z

Hi @kaingwade, thanks for the advice. I have added the license and add documentation for imgaug module. Junk codes in header files have also been cleaned.

… range of AdjustHue and ColorJitter

… samples

asenyaev · 2022-09-30T16:59:40Z

Hello @ZhaoChuyang!

I have added support for opencv_test_imgaug in CI. Could you push the latest commit again to re-run CI for this PR?

ZhaoChuyang · 2022-10-01T06:27:21Z

Hi @asenyaev, I have commited the latest changes.

asmorkalov · 2022-10-18T06:48:43Z

modules/imgaug/include/opencv2/imgaug/rng.hpp

+        extern uint64 state;
+
+        //! Random number generator for data augmentation module
+        extern cv::RNG rng;


Global variables with direct access is very bad idea:

C++ does not define initialization order. rng and state may be initialized in any order as soon as global variables in user's code. It becomes even worse, if library is linked statically.

It's not clear how to use this in multi-threaded environment.

I propose 2 options:

Use standard OpenCV theRNG. It uses thread local seed.

Provide own theRNG-like function. It returns reference to RNG object that could be re-initialized.

asmorkalov · 2022-10-18T06:53:10Z

modules/imgaug/include/opencv2/imgaug/transforms_det.hpp

+                 */
+                CV_WRAP explicit Convert(int code);
+
+                /** @brief Apply data augmentation method on source image and its annotation.


Please specify the method.

asmorkalov · 2022-10-18T06:53:45Z

modules/imgaug/include/opencv2/imgaug/transforms_det.hpp

+            };
+
+            //! Convert the color space of the given image
+            class CV_EXPORTS_W Convert: public Transform{


I propose to rename to ColorConvert. Convert is too generic.

asmorkalov · 2022-10-18T06:56:26Z

modules/imgaug/misc/python/pyopencv_imgaug.hpp

+template<> struct pyopencvVecConverter<Ptr<cv::imgaug::Transform> >
+{
+    static bool to(PyObject* obj, std::vector<cv::Ptr<cv::imgaug::Transform> >& value, const ArgInfo& info)
+    {
+        return pyopencv_to_generic_vec(obj, value, info);
+    }
+
+};
+
+template<> struct pyopencvVecConverter<Ptr<cv::imgaug::det::Transform> >
+{
+    static bool to(PyObject* obj, std::vector<cv::Ptr<cv::imgaug::det::Transform> >& value, const ArgInfo& info)
+    {
+        return pyopencv_to_generic_vec(obj, value, info);
+    }
+
+};


Why do you need custom bindings for it?

asmorkalov · 2022-10-18T07:06:37Z

modules/imgaug/include/opencv2/imgaug/functional.hpp

+     * Brightness factor should be >= 0. When brightness factor is larger than 1, the output image will be brighter than original.
+     * When brightness factor is less than 1, the output image will be darker than original.
+     */
+    void adjustBrightness(Mat& img, double brightness_factor);


It makes sense to use InputArray and OutputArray for all functions in this header. Rationale:

Generic OpenCV interface.

UMat support, if InputArray is promoted from Transform classes directly without getMat() call.

asmorkalov · 2022-10-18T07:08:25Z

modules/imgaug/misc/python/pyopencv_imgaug.hpp

+template<> struct PyOpenCV_Converter<unsigned long long>
+{
+    static bool to(PyObject* obj, unsigned long long& value, const ArgInfo& info){
+        if(!obj || obj == Py_None)
+            return true;
+        if(PyLong_Check(obj)){
+            value = PyLong_AsUnsignedLongLong(obj);
+        }else{
+            return false;
+        }
+        return value != (unsigned int)-1 || !PyErr_Occurred();
+    }
+};


I have not found long log or unsigned long long usage in interface. Most probably the manual binding is not required.

asmorkalov · 2022-10-18T07:15:51Z

modules/imgaug/src/functional.cpp

+        std::vector<Mat> gray_arrays = {gray, gray, gray};
+        merge(gray_arrays, gray);


it chould be cvtColor with COLOR_GRAY2BGR.

asmorkalov · 2022-10-18T10:33:12Z

modules/imgaug/src/functional.cpp

+        std::vector<Mat> new_channels;
+        for(int i=0; i < num_channels; i++){
+            Mat& channel = channels[i];
+            Scalar avg = mean(channel);


The function cv::mean calculates the mean value M of array elements, independently for each channel, and return it. No need to allocate array and iterate over channels the next arithmetic steps could be dome for all channels together without loop.

asmorkalov · 2022-10-18T10:38:06Z

General notes on test code:

There is no need to store cropped/flipped/converted images in opencv_extra, if test checks single transformation. It's better to call reference function from OpenCV itself. The test should check new logic, but not OpenCV primitives behavior.
Do not use ts->set_failed_test_info. It old API. Just GTest EXPECT_XXX and ASSERT_XXX are simpler and it very clean which of condition fails.

asmorkalov · 2022-12-15T07:40:11Z

@ZhaoChuyang Friendly reminder.

ZhaoChuyang · 2022-12-15T07:54:53Z

Hi, sorry for the delay, I have been caught up in a DDL. I'll fix them ASAP.

LaurentBerger · 2024-04-28T18:34:32Z

What's new about this module?

add imgaug module

030f427

ZhaoChuyang changed the title ~~add imgaug module~~ [GSoC22] Data Augmentation Module in OpenCV (imgaug) Aug 24, 2022

kaingwade reviewed Aug 26, 2022

View reviewed changes

asmorkalov added the GSoC label Aug 29, 2022

ZhaoChuyang added 2 commits August 29, 2022 20:14

Merge branch 'opencv:4.x' into imgaug

59ecd1a

add license and documentation for imgaug module

06a7d45

ZhaoChuyang marked this pull request as draft August 29, 2022 12:27

ZhaoChuyang marked this pull request as ready for review August 29, 2022 13:47

ZhaoChuyang added 19 commits September 4, 2022 15:32

add docs and aug methods for detection module

35830e9

Change test sample of Normalize; Remove blank lines; Change the input…

6733574

… range of AdjustHue and ColorJitter

add docs for det, modify constructors

8300ece

create source file for fucntional; add doc for functional.hpp; modify…

c3d1ad6

… samples

add assert to check params for det methods

437a206

fix bugs

fc44148

fix bug variable length array compile failed on windows platform

dbedb1b

add tutorial, fix bugs in det module

671020f

add python toggle for tutorial

08bf6fc

remove trailing whitepaces

c8c8940

add test for det module

2ed2063

fix warnings

8d7807a

fix warnings

c93037d

fix warnings

fdfc491

fix warnings

be06b22

modify documentation

9380050

rename images

dfee089

add tutorial

fb35463

reduce image size

fb9a5d7

asenyaev mentioned this pull request Sep 30, 2022

Add imgaug tests for contrib workflows opencv/ci-gha-workflow#65

Merged

modify demo

b44ba61

ZhaoChuyang mentioned this pull request Oct 2, 2022

[GSoC]bugfix: errors raised when cv::Vec used as default arguments opencv/opencv#22253

Closed

6 tasks

asmorkalov reviewed Oct 18, 2022

View reviewed changes

vpisarev mentioned this pull request Feb 12, 2024

New shiny Imgproc module for OpenCV 5.0 opencv/opencv#25012

Open

LaurentBerger mentioned this pull request May 22, 2024

Support for lowpass/anti-alias enabled resizing operations to support preprocessing operations for models trained with torchvision opencv/opencv#25620

Open

		std::vector<Mat> gray_arrays = {gray, gray, gray};
		merge(gray_arrays, gray);

[GSoC22] Data Augmentation Module in OpenCV (imgaug) #3335

Are you sure you want to change the base?

[GSoC22] Data Augmentation Module in OpenCV (imgaug) #3335

Uh oh!

Conversation

ZhaoChuyang commented Aug 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

kaingwade left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZhaoChuyang commented Aug 29, 2022

Uh oh!

asenyaev commented Sep 30, 2022

Uh oh!

ZhaoChuyang commented Oct 1, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asmorkalov commented Oct 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asmorkalov commented Dec 15, 2022

Uh oh!

ZhaoChuyang commented Dec 15, 2022

Uh oh!

LaurentBerger commented Apr 28, 2024

Uh oh!

Uh oh!

ZhaoChuyang commented Aug 24, 2022 •

edited

Loading

asmorkalov commented Oct 18, 2022 •

edited

Loading