Optimizing for Hardware Specifications

[CCranney]

AttentionSmithy was made primarily for flexibility and readability. It was designed with experimentation in mind, such that explorative extensions would be easy to make and that any design choice would be easy to include or remove in the final model, all with human readability prioritized. Similar packages exist (such as Transformer Engine, see here), but they focus primarily on optimization for hardware rather than experimentative discovery, extension, and readability.

I would like to get the best of both worlds. Hardware optimization is not my current area of expertise, but it is something I could see AttentionSmithy excelling at in the future.