Welcome to the Implemented Papers repository! This is where I re-create research papers in deep learning and other numerical fields. I'm doing this to improve my understanding of the latest techniques and to build a handy collection of reusable tools and code.
Implemented Papers is designed to translate innovative research papers into functional code. By re-implementing these papers, I aim to:
- Enhance my understanding of advanced deep learning techniques.
- Create modular, reusable code that serves as both a learning tool and a practical resource.
- Build a personal framework with classes and utilities that reflect modern research.
Note: Although I'm coding everything from scratch to deepen my understanding and experience, I occasionally use coding assistants such as ChatGPT and GitHub Copilot. These tools help ensure that the documentation for functions in the framework remains consistent and well-organized.
- Re-Implementations: Code for various deep learning papers and numerical methods.
- Modular Framework: A growing collection of classes and functions that can be used across different projects.
- Examples: Example scripts demonstrating the use of implemented methods.
Here is a list of the papers I've implemented, with links to their respective directories:
- NICE: NON-LINEAR INDEPENDENT COMPONENTS ESTIMATION, Laurent Dinh, David Krueger, Yoshua Bengio ; 2014; arXiv
NICE proposes a framework for learning complex high-dimensional densities using invertible transformations. It maps data to a latent space with independent variables, ensuring easy computation of the Jacobian and its inverse. This approach, based on deep neural networks, allows efficient generative modeling with strong results on image datasets.
I re-implemented the NICE model and reproduced the experiments on the MNIST dataset. Additionally, I conducted a personal experiment using a simple sinusoidal 2D distribution as a proof of concept, demonstrating the model's effectiveness in a custom scenario.
- DENSITY ESTIMATION USING REAL NVP, Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio; 2017; arXiv
RealNVP (Real-valued Non-Volume Preserving transformations) builds upon the invertible transformation framework introduced in NICE. By using affine coupling layers, RealNVP allows for the exact computation of log-determinants of the Jacobian, facilitating scalable density estimation for high-dimensional data. The model is especially successful in generative modeling tasks on image datasets, producing high-quality samples with minimal computational overhead.
I re-implemented the RealNVP model and reproduced the experiment on the CelebA dataset. Similar to the NICE paper, I conducted a personal experiment using a sinusoidal 2D distribution to further demonstrate RealNVP's capacity in modeling complex distributions effectively.
- GLOW: GENERATIVE FLOW WITH INVERTIBLE 1x1 CONVOLUTIONS, Diederik P. Kingma, Prafulla Dhariwal; 2018; arXiv
GLOW (Generative Flow with Invertible 1x1 Convolutions) further develops the invertible flow framework by introducing invertible 1x1 convolutions. These convolutions act as a generalization of the permutation operation in previous flow models, allowing for more expressive transformations and better capturing dependencies between dimensions. GLOW achieves state-of-the-art results in density estimation and image generation, demonstrating its ability to generate high-fidelity and diverse images.
I re-implemented the GLOW model and reproduced the experiments (conditional and unconditional) on the CIFAR-10 dataset.
- AUTO-ENCODING VARIATIONAL BAYES, Diederik P. Kingma, Max Welling; 2013; arXiv
Auto-Encoding Variational Bayes (VAE) introduces a deep generative model that combines variational inference with deep learning. VAE aims to learn a latent variable model by maximizing a lower bound on the log marginal likelihood. This approach allows for efficient and scalable learning of complex distributions, making it a popular choice for generative modeling tasks.
I re-implemented the VAE model and reproduced the experiments on the MNIST dataset using a 20-D latent space. I also compare the score of the Adagrad optimizer (originally used in the paper) with the score of the Adam optimizer. Finally I explore the learned latent space by linear interpolation and decoding.
- PHYSICS INFORMED DEEP LEARNING Maziar Raissi, Paris Perdikaris, George Em Karniadakis; 2017;
PINNs offer a novel framework for incorporating known physical laws, described by nonlinear partial differential equations (PDEs), into neural network architectures. By integrating physics directly into the learning process, PINNs enable data-driven solutions for high-dimensional problems while simultaneously satisfying governing equations. This approach offers a flexible and scalable method for solving complex PDEs by combining data-driven learning with the structure of physical laws.
-
(Part I): Data-driven Solutions of Nonlinear Partial Differential Equations, arXiv
The first part is focused on approximating the solution of a a-priori known PDE.
I re-implemented the continuous time approach for solving PDEs. Specifically, I have implemented and extended two key examples:
- Burgers' Equation (1D): This example is based on the original problem presented in the paper, and the results closely match the exact solutions.
- Heat Equation (1D): A personal experiment that applies the PINN framework to the heat equation, with an error analysis comparing the model’s predictions to the exact analytical solution.
-
(Part II): Data-driven Discovery of Nonlinear Partial Differential Equations, arXiv
[To be implemented]
- ENERGY CONSERVING EXPLICIT LOCAL TIME STEPPING FOR SECOND-ORDER WAVE EQUATIONS, Julien Diaz, Marcus J. Grote; DOI
This paper addresses stability constraints imposed by locally refined meshes on explicit time-stepping methods for simulating time-dependent wave phenomena. By introducing local time-stepping (LTS) methods, the authors enable the use of arbitrarily small time steps in regions with small mesh elements, resulting in an explicit, parallel, and energy-conserving numerical scheme. Starting from the standard second-order leap-frog scheme, the paper develops methods of arbitrary accuracy, validated through numerical experiments.
I implemented the LTS method for the 1D wave equation using finite element methods (FEM) and a Leap-Frog scheme. The experiments include:
- Exploring the stability criterion for the classical Leap-Frog scheme.
- Comparing it with the stability criterion when using the LTS Leap-Frog scheme.
Numerical results demonstrate the effectiveness of LTS in enhancing stability without significantly increasing computational workload. The implementation and detailed analysis are documented in my repository: MATH0024 - Local Time Stepping Leap-Frog for the wave equation in 1D. Figures and insights from the experiments are provided to illustrate the benefits of LTS.
... To be continued :-D
This project is licensed under the MIT License. See the LICENSE file for details.
I plan to do some documentation for the framework. There are many papers I want to explore in this repo, here for my future self of a set of interesting papers :
- Learning to learn by gradient descent by gradient descent
- Attention Is All You Need - A lot of the building blocks are already on my computer so it should be pretty simple. Also in the MOzART project.
- Deep Residual Learning for Image Recognition
- GUIDED IMAGE GENERATION WITH CONDITIONAL INVERTIBLE NEURAL NETWORKS
- Neural Ordinary Differential Equations
- Something on active learning (i still have to look for papers)