UnderstandingLLMs

A hands-on collection of Jupyter notebooks exploring key concepts, architectures, and optimizations in Large Language Models (LLMs).

---

Notebooks

1. LLM from Scratch - Part 1

• Build a basic LLM using PyTorch.
• Covers data preprocessing, model architecture, training loop, and inference.

---

2. LLM from Scratch - Part 2

• Enhances the previous notebook with modern techniques:
  • RMSNorm
  • Gated (SwiGLU) FFN
  • Rotary Positional Embeddings (RoPE)
• Includes gradient accumulation and mixed precision training.

---

3. Rotary Positional Embeddings (RoPE)

• Deep dive into RoPE in PyTorch.
• Compares different RoPE implementations.

---