RoBERTa Implementation

This repository contains a PyTorch-based implementation of the RoBERTa transformer model and tokenizer from the ground up. Additionally, I include a normal implementation of RoBERTa.

The dataset used is a political survey which includes 13 class labels, and each observation can have multiple labels.

What's Inside

• RoBERTa and Tokenizer Full
  Full implementation of the RoBERTa model, including the tokenizer, encoding, embedding, attention, and training loop, all built using PyTorch. Meant for learning how transformer architectures function at a lower level.

• RoBERTa Implementation
  Script for initializing and using RoBERTa.