Custom Numeric (ie, NOT Position) Embeddings

[CCranney]

We've implemented custom position embeddings for sinusoidal and ALiBi numeric embeddings - applying their basic premises to a numeric dataset rather than position. An example would be using ALiBi to encode time rather than position, which AttentionSmithy can do. However, we have not implemented that for learned or rotary embeddings.

I'm thinking it would be a good idea to make this a general pattern - each position embedding actually has 2 implementations, one for position and one for custom. That way it applies to any numeric embedding somebody may have in mind.

[CCranney]

Custom rotary embedding was implemented in #38. Learned embeddings would be very niche - unlike alibi, sinusoidal or rotary, you couldn't really do continuous values, so it would have to be binned values. I compromised by implementing a "continuous values embedding" or CVE implementation, which I derived from the following paper. It's a way of doing learned embeddings for continuous values. Overall, I consider these implementations to fulfill this discussion, so am closing it.