LLM support, attention head and transformer architecture support for hls4ml

[danlg]

The transformer architecture (https://arxiv.org/pdf/1706.03762) has been instrumental to scale sequence neural networks.
The transformer architecture is the fundamental building block of all LLMs. The trend of open sourcing LLM or reducing the number of parameters is strong. So the support of transformer architecture and attention head would be a great addition to hls4ml, which power and latency gains expected.

But the status and feature pages doesn't list it (https://fastmachinelearning.org/hls4ml/status.html)

Given the rationale above, I don't understand why the community has not yet engaged on this work, nor that it is listed in the discussions.

• In your opinion, what are the technical barriers to overcome to implement it in hls4ml ?

[ZHAODONG-LYU]

There is a paper in arXiv (https://arxiv.org/pdf/2402.01047), adding support in hls4ml. They said transformer code will be made in the near future.