Threaded Regular Executor

[dmatveev]

Original PR: opencv/opencv#24845

What we have now

1. One threaded executor maps to one GCompiled
2. Number of threads is defined as a GCompileArg

What we need

1. Share thread-pool between a number of GCompileds
2. Number of threads should be number of cores by default? Or how do we chose it? Can we add worker threads dynamically when needed?

How it can be tuned dynamically?

1. Estimate theoretical parallelism per graph as number N
2. Initialize thread pool for N threads by default
3. If a new graph comes in
   1. How do we know if these graph should run in parallel?
   2. A number of worker threads is extended to std::max(hardware_concurrency(), pool.N + new_graph.N)

Known graph execution challenges

1. Using main thread (where .apply()/operator() was called from) as a compute thread (reduce overhead)
2. Consider TBB is the backend for the implementation - to stick different parallel runtime consumers to the same thread arena.
3. Asynchronous execution on HW: worker threads shouldn't be blocked by calls which can be asynchronous.
4. Prioritization: if multiple graphs share the same pool, a prioritization problem may occur.