4. Types of Generative Models

Introduction

Overview

Model Type	Strength	Weakness	Examples
GANs	Sharp, realistic images	Mode collapse, hard to train	StyleGAN, CycleGAN, BigGAN
VAEs	Stable training, latent space	Blurry outputs	β-VAE, VQ-VAE
Autoregressive	High-quality sequential data	Slow generation (sequential)	GPT, PixelCNN, WaveNet
Flow-Based	Exact likelihood, invertible	Computationally expensive	RealNVP, Glow, Normalizing Flows
Diffusion	State-of-the-art image quality	Slow sampling (many steps)	DDPM, Stable Diffusion, DALL-E 2
Transformer	Parallel training, long context	Quadratic attention cost	GPT-4, Claude, Gemini

Autoregressive Models

Generate data one element at a time, conditioning each new element on previously generated ones. The probability of a sequence is the product of conditional probabilities: P(x) = Π P(x_t | x_<t). GPT models are autoregressive.

Diffusion Models

Two processes: forward (gradually add Gaussian noise to data) and reverse (learn to denoise). At inference, start from pure noise and iteratively denoise to generate data. This approach powers all major text-to-image systems.

# Simplified diffusion training
def train_step(x_0):
    t = random_timestep()
    noise = torch.randn_like(x_0)
    x_t = sqrt(α_bar_t) * x_0 + sqrt(1 - α_bar_t) * noise
    noise_pred = model(x_t, t)
    return MSE(noise_pred, noise)

# Sampling (reverse process)
def sample(model, steps=50):
    x = torch.randn(3, 64, 64)
    for t in reversed(range(steps)):
        noise_pred = model(x, t)
        x = denoise_step(x, noise_pred, t)
    return x

Which Model to Choose?

The choice depends on the task: images → diffusion or GANs; text → autoregressive transformers; structured data → VAEs or flow-based; real-time → lightweight autoregressive or GANs.

Practice Task: Create a comparison table of 5 generative model types across dimensions: training stability, inference speed, output quality, and ease of implementation. For each, identify one ideal use case and one weak area. Run a pre-trained diffusion model (e.g., Stable Diffusion via Hugging Face) and a pre-trained GPT model. Compare their outputs qualitatively.