OpenCV DNN: Reuse ADE as a new DNN core engine

[dmatveev]

ADE is a pretty mature library at this point which provides all graph structure needs, support basic graph compiler (pass) infrastructure, and extensible to infinity to support all basic needs.

Related: #9

Learning ADE

• Repo itself: https://github.com/opencv/ade
• Tutorial: https://github.com/opencv/ade/tree/master/tutorial
  • Tiny graph compiler example with ND operations: https://github.com/opencv/ade/blob/master/tutorial/05_backend.cpp
  • Nested graph example: https://github.com/opencv/ade/blob/master/tutorial/06_hierarchy.cpp

ADE in action

• G-API core intermediate representation: https://github.com/opencv/opencv/blob/4.x/modules/gapi/src/compiler/gmodel.hpp
• G-API hetero intermediate representation: https://github.com/opencv/opencv/blob/4.x/modules/gapi/src/compiler/gislandmodel.hpp
• An example pass in graph compiler, organizes a graph of operation into graph of subgraphs: https://github.com/opencv/opencv/blob/4.x/modules/gapi/src/compiler/passes/exec.cpp
• An example of backend-defined passes in graph compiler: https://github.com/opencv/opencv/blob/4.x/modules/gapi/src/backends/fluid/gfluidbackend.cpp#L1406
• An example of shape inference pass in graph compiler: https://github.com/opencv/opencv/blob/4.x/modules/gapi/src/compiler/passes/meta.cpp#L36 (also called in runtime on reshape)

cc: @vpisarev

[fengyuentau]

What about graph fusion? Finding a subgraph pattern, then replace this subgraph with a single node.

[dmatveev]

There is graph pattern matching /substitution algorithm available in G-API. So-called "G-API Transformations"

• Tests (to understand the API, note at the API level it works for expressions):
  • https://github.com/opencv/opencv/blob/4.x/modules/gapi/test/gapi_transform_tests.cpp
  • https://github.com/opencv/opencv/blob/4.x/modules/gapi/test/internal/gapi_int_perform_substitution_test.cpp
• Implementation (ADE-based):
  • https://github.com/opencv/opencv/blob/4.x/modules/gapi/src/compiler/passes/pattern_matching.cpp
  • https://github.com/opencv/opencv/blob/4.x/modules/gapi/src/compiler/passes/perform_substitution.cpp

Note the current implementation is tied to G-API graph metadata, but can be generalized to work atop of abstract-level ADE.

[dmatveev]

hmm, as G-API has so many functionality out of the box, maybe it is G-API not ADE should be the DNN's internal engine? What do you think @vpisarev ?

[dmatveev]

BTW there were attempts already... Maybe now is the time? opencv/opencv#17236

If you represent DNN layers as G-API Operations, then you get:

1. Kernel/backend infrastructure for free
2. Heterogeneity for free
3. Pattern matching/substitution for free
4. Extensibility for free.