Really interesting article with interactive examples on visualizing matmul and Transformer structure

[KerfuffleV2]

Maybe everyone else has already seen this, but if not it seems like an amazing resource for understanding some of the math and structure behind these models: https://pytorch.org/blog/inside-the-matrix/

The beginning starts with some background on the math, but if that's not interesting enough, scroll down a bit to the Inside an Attention Head section.

Also GPT2 attention head explorer: https://bhosmer.github.io/mm/examples/attngpt2/index.html

[ggerganov]

Thanks for sharing this! I had missed it and from a quick look, the animations are very cool! Will definitely check it out