Support IBM MoLM. Possible nextgen llm architecture

[Jipok]

Reddit: https://www.reddit.com/r/LocalLLaMA/comments/16ikyma/mitibm_watson_ai_lab_releases_molm_suite_with/
Paper: https://arxiv.org/abs/2306.04640
GitHub: https://github.com/ibm/moduleformer
Models: MoLM-700M-4B MoLM-350M-4B MoLM-700M-8B

MoLM-350M-4B is a MoE-based language model. It has 4 billion parameters, but each input token only activates 350M parameters. Thus, it's computationally equivalent to a 350M dense model.