Annotated Research Paper Implementations: Transformers, StyleGAN, Stable Diffusion, DDPM/DDIM, LayerNorm, Nucleus Sampling and more

This is a collection of simple PyTorch implementations of neural networks and related algorithms. These implementations are documented with explanations, and the website renders these as side-by-side formatted notes. We believe these would help you understand these algorithms better.

We are actively maintaining this repo and adding new implementations. for updates.

Translations

English (original)

Chinese (translated)

Japanese (translated)

Paper Implementations

✨ Transformers

• JAX implementation
• Multi-headed attention
• Triton Flash Attention
• Transformer building blocks
• Transformer XL
• Relative multi-headed attention
• Rotary Positional Embeddings (RoPE)
• Attention with Linear Biases (ALiBi)
• RETRO
• Compressive Transformer
• GPT Architecture
• GLU Variants
• kNN-LM: Generalization through Memorization
• Feedback Transformer
• Switch Transformer
• Fast Weights Transformer
• FNet
• Attention Free Transformer
• Masked Language Model
• MLP-Mixer: An all-MLP Architecture for Vision
• Pay Attention to MLPs (gMLP)
• Vision Transformer (ViT)
• Primer EZ
• Hourglass

✨ Low-Rank Adaptation (LoRA)

✨ Eleuther GPT-NeoX

• Generate on a 48GB GPU
• Finetune on two 48GB GPUs
• LLM.int8()

✨ Diffusion models

• Denoising Diffusion Probabilistic Models (DDPM)
• Denoising Diffusion Implicit Models (DDIM)
• Latent Diffusion Models
• Stable Diffusion

✨ Generative Adversarial Networks

• Original GAN
• GAN with deep convolutional network
• Cycle GAN
• Wasserstein GAN
• Wasserstein GAN with Gradient Penalty
• StyleGAN 2

✨ Recurrent Highway Networks

✨ LSTM

✨ HyperNetworks - HyperLSTM

✨ ResNet

✨ ConvMixer

✨ Capsule Networks

✨ U-Net

✨ Sketch RNN

✨ Graph Neural Networks

• Graph Attention Networks (GAT)
• Graph Attention Networks v2 (GATv2)

✨ Reinforcement Learning

• Proximal Policy Optimization with Generalized Advantage Estimation
• Deep Q Networks with with Dueling Network, Prioritized Replay and Double Q Network.

✨ Counterfactual Regret Minimization (CFR)

Solving games with incomplete information such as poker with CFR.

• Kuhn Poker

✨ Optimizers

• Adam
• AMSGrad
• Adam Optimizer with warmup
• Noam Optimizer
• Rectified Adam Optimizer
• AdaBelief Optimizer
• Sophia-G Optimizer

✨ Normalization Layers

• Batch Normalization
• Layer Normalization
• Instance Normalization
• Group Normalization
• Weight Standardization
• Batch-Channel Normalization
• DeepNorm

✨ Distillation

✨ Adaptive Computation

• PonderNet

✨ Uncertainty

• Evidential Deep Learning to Quantify Classification Uncertainty

✨ Activations

• Fuzzy Tiling Activations

✨ Language Model Sampling Techniques

• Greedy Sampling
• Temperature Sampling
• Top-k Sampling
• Nucleus Sampling

✨ Scalable Training/Inference

• Zero3 memory optimizations

Installation

pip install labml-nn