[deepspeed inference source code] use autotp combined with kernel injection

[frankxyy]

In engine.py script, I found the code above. It seems that for input with kernel injection switch on, the process of autotp will not be executed. Is there a possibility that the two: autotp and kernel injection can both be utilized?

[RezaYazdaniAminabadi]

Hi @frankxyy,

There is TP for when we use kernel-injection, it is just that the tensor sharding is done in a slightly different way and in different part of the code. If you are interested, you can follow the logic from here: https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/module_inject/replace_module.py#L354
Thanks,
Reza