3. ML Foundations for Generative AI

Introduction

Supervised Learning

Models learn from labeled data (input → output). For generative AI, supervised learning is used in fine-tuning — training a pre-trained model on specific (prompt, response) pairs to improve instruction following.

# Example: Fine-tuning a transformer for summarization
# Pseudo-code (simplified)
for batch in supervised_dataset:
    inputs = tokenize(batch["article"])
    targets = tokenize(batch["summary"])
    loss = cross_entropy(model(inputs), targets)
    optimizer.step(loss)

Unsupervised Learning

Models find patterns in unlabeled data. Most pre-training of generative models is unsupervised — GPT predicts the next token, BERT predicts masked tokens. This allows training on vast amounts of raw text from the internet.

Reinforcement Learning from Human Feedback (RLHF)

RLHF aligns generative models with human preferences through three stages:

1. SFT — Supervised fine-tuning on high-quality demonstrations
2. Reward Modeling — Train a reward model on human comparisons of outputs
3. RL Optimization — Use PPO to optimize the generative model against the reward model

# PPO update (simplified)
for step in rl_steps:
    outputs = model.generate(prompts)
    rewards = reward_model(outputs)
    # KL penalty to prevent too much divergence
    loss = ppo_loss(outputs, rewards, ref_logprobs)
    optimizer.step(loss)

Transfer Learning

Pre-train a large model on a broad dataset, then fine-tune on a specific task. This is the dominant paradigm in generative AI — train once, adapt everywhere.

Fine-tuning Strategies

• Full Fine-tuning — Update all parameters (expensive)
• LoRA (Low-Rank Adaptation) — Train small rank matrices injected into layers (efficient)
• Adapter Layers — Add small trainable modules between frozen layers
• Prompt Tuning — Learn soft prompt embeddings while keeping model frozen