Generative AI tools, such as GPT, are trained with a goal of minimizing their entropy. This framework is adopted by language models to best enhance their efficiency. In this project, we explore how we can use entropy and information in language models and how we can optimize it for generative tasks. We also investigate the dynamics of these entropy techniques to see if it helps optimize our models. In addition, we examine various methods that help reduce the loss in language models, aiming to improve their performance and understand the implementation of entropy reduction frameworks.

Note: The results are limited due to the non-complex model architecture. This project serves as a framework for larger applications.