High-Order Constructs

[dmatveev]

Currently (in v4.9) G-API lacks high-order constructs which become inherently important in the modern AI pipelines. Examples include Stable Diffusion and Whisper. The goal of this discussion is to document the possible API with examples and pros/cons.

Example Flows

Stable Diffusion

Classic Stable Diffusion 1.5 pipeline employs a text encoder, UNet, and VAE decoder networks. The problem here with UNet which is executed in a loop; also there's a Scheduler which refines UNet results after every iteration. In pseudocode, the flow looks like this:

embeddings := text_encoder(input_prompt);
noise := random([2,64,64]);
state := noise;
for (i : iters):
  new_state := unet(embeddings, state, i);
  state     := scheduler(new_state, i);
result := vae_decoder(new_state);

(NB: Is vae_decoder applied to output of unet but not scheduler at the final iteration?)
Here new_state is a single tensor.

Whisper

OpenVINO-based implementation of OpenAI Whisper pipeline employs kv-cache for the Whisper decoder: 1, 2. With this approach, the Whisper pipeline for a 30s audo input may look like:

features := encode(samples);
tokens   := [];
(next_token, cache) := decode(features, []);
while (next_token != EOS):
  append(tokens, next_token);
  (next_token, cache) := decode(features, cache); 

BTW, this pattern may be seen in many modern encoder-decoder scenarios.

G-API Operators Definition

Technically, G-API is a very high-level form of SSA: Every new Operation, when applied to Data, produces new Data which seen as immutable.
With the exception of Stateful kernels, G-API graph form may be seen as a pure functional programming language.

In really pure functional languages, there's no /statements/ as in imperative languages like C, but there are /expressions/ which always have a value. This property may be important as G-API tracks the graph structure purely based on value/producer_of(value) relationships (and is, in fact, defined on vectors of computation's input and output values).

The following subchapters will cover the minimum required basis to support the above scenarios.

1. IF: https://github.com/orgs/g-api-org/discussions/17 (NOT seen in the above examples, but a quite common this).
2. MAP: TBD
3. FOLD: TBD

Challenges

1. infer<Generic> may be problematic to pass inputs/outputs around. We do distinguish input and output tensor dictionaries, the output one is in fact used to populate the information about outputs of the network.
   • Example: https://github.com/opencv/opencv/blob/4.x/modules/gapi/misc/python/samples/gaze_estimation.py#L239-L257