Longformer, Big Bird implementation

[CCranney]

I actually started a Big Bird implementation already (see src/attention_smithy/attention/BigBirdAttention.py). It has a full test suite and technically performs as it should. BUT I realized during real-life application that my variation of their approach - direct indexing rather than using the gather function - does not play well with large data samples. Thus, the entire point of the implementation was rendered moot.

I'd like to rewrite this at some point to use the gather function, but that would require extensive rewrites. If anyone has any thoughts, let me know.

Longformer employs similar principles, so co-development would probably be a good idea.

[CCranney]

Big Bird and Longformer both have a distinction in choosing "global" tokens that attend to all other tokens (and gets attended to in turn). It was unclear to me how these were chosen, and in my coding efforts have assumed that specific individual tokens are given global status, selected in advance by the user. However, I have wondered if global token selection could be learned by the model itself. Something to consider in future implementation efforts.

[CCranney]

I would also be interested to see if these implementations could be mixed with Hyena operators, possibly mutually reinforcing each other.

[CCranney]

Longformer (for encoders, so far) was implemented in #44.