[REQ] Automatic Device Selection

[MarcoRavich]

Hi there,
during my experiments in video inferencing on OpenVINO, I found this interesting Intel document that claims:

The Automatic Device Selection mode, or AUTO for short, uses a “virtual” or a “proxy” device, which does not bind to a specific type of hardware, but rather selects the processing unit for inference automatically.

How AUTO Works

To put it simply, when loading the model to the first device on the list fails, AUTO will try to load it to the next device in line, until one of them succeeds. What is important, AUTO starts inference with the CPU of the system by default unless there is model cached for the best suited device, as it provides very low latency and can start inference with no additional delays. While the CPU is performing inference, AUTO continues to load the model to the device best suited for the purpose and transfers the task to it when ready. This way, the devices which are much slower in compiling models, GPU being the best example, do not impact inference at its initial stages. For example, if you use a CPU and a GPU, the first-inference latency of AUTO will be better than that of using GPU alone.

Hope that inspires !

[RyanMetcalfeInt8]

Hi @MarcoRavich,

Sorry for delayed response! I've only dabbled a little bit with 'AUTO' mode -- it is something I'd be interesting in experimenting more with! It would be pretty neat to be able to implicitly start executing models on CPU while we load models to the GPU or NPU, for example. My main concern would be memory consumption -- i.e. whether OpenVINO is able to share weights across the devices in the AUTO device list, or if it needs to have a dedicated set of weights loaded for each device.

Anyway, as always, thanks for the ideas. I'll convert this to a discussion for now.