Question about residual connection

[anxietymonger]

Got here from this article.

Firstly thank you so much about your work, giving insight for time series analysis beginners like me.

However, in the definition of TransformerEncoder, I found in forward() function, the residual connection is defined like this:

ff_layer = self.ff_normalize(x[0] + ff_layer)

According to the original paper (Attention is all you need), the residual connection should be defined in this way:

ff_layer = self.ff_normalize(attn_layer + ff_layer)

Any possibility that you could provide more explanation about the intuition behind this discrepancy?

[WilliamsTim]

You are correct in that the result of the feed forward layer is supposed to be directly added back to the result of attention layer like that. And honestly, it would be easiest to pass in a single input instead of a tuple keep adding to that. If the input was say x. Then, It would just be x += attn_layer then x += ff_layer. As for the explanation for why it’s wrong is just that this whole thing is an improper implementation of a transformer.