Positional Embedding in DeepSpeed Transformer Kernel

[sarvghotra]

Hi,

HuggingFace's BertLayer has an option to apply positional embedding in its BertSelfAttention (here), but I couldn't find it in DeepSpeed's Transformer kernel (here). In other words, I was trying to find DeepSpeed's equivalent for Hugginface's position_embedding_type (link) argument in their BertConfig but had no luck. Could you please help with this?

Context: I am trying to implement Swin Transformer by reusing DeepSpeed's Kernel code, but Swin has relative positional embegging in it which I couldn't find in the kernel's code.