2. Evolution of Generative Models

Introduction

Timeline of Generative AI

• 1950s-1980s: Early language models (n-grams, HMMs), Markov chains for text generation
• 1990s-2000s: HMMs and GMMs for speech, pattern-based generation, statistical MT
• 2013: Variational Autoencoders (VAEs) introduced by Kingma & Welling
• 2014: Generative Adversarial Networks (GANs) introduced by Ian Goodfellow
• 2017: Transformer architecture published in "Attention Is All You Need" (Vaswani et al.)
• 2018: GPT-1 (117M params) — first large-scale generative pre-trained transformer
• 2019: GPT-2 (1.5B params) — demonstrated impressive text generation; initially withheld due to safety concerns
• 2020: GPT-3 (175B params) — few-shot learning breakthrough; DALL-E for text-to-image
• 2021: Stable Diffusion, Midjourney, DALL-E 2 — photorealistic image generation goes mainstream
• 2022: ChatGPT reaches 100M users in 2 months; generative AI enters mainstream consciousness
• 2023: GPT-4, Claude, Gemini, Llama 2 — multimodal capabilities, longer context, improved reasoning
• 2024-2026: AI agents, video generation (Sora, Veo), real-time generation, open-source models (Llama 3, Mistral)

Key Breakthroughs

• GANs (2014) — First framework for generating realistic synthetic data through adversarial training
• Transformer (2017) — Replaced RNNs with attention; enabled parallel training and much larger models
• Diffusion Models (2020) — Surpassed GANs for image generation with better diversity and fidelity
• RLHF (2022) — Aligned language models with human preferences through reinforcement learning
• Multimodal (2023-2024) — Models that understand and generate across text, image, audio, and video

Scaling Laws

Research shows that model performance improves predictably with increases in model size, dataset size, and compute. This empirical finding drove the "bigger is better" race, though recent focus has shifted to efficiency and alignment.