5. Probability & Estimation for Generative AI

Probability & Estimation

Probability Distributions

Generative models learn probability distributions over data. A distribution P(x) assigns a probability to each possible data point x. The goal is to approximate the true data distribution with a model distribution P_θ(x).

• Discrete Distributions — Bernoulli (binary), Categorical (multi-class), Poisson (counts)
• Continuous Distributions — Normal/Gaussian, Uniform, Beta, Exponential
• Mixture Models — Combine multiple distributions (e.g., Gaussian Mixture Models)

Probability Density Functions (PDF)

For continuous data, the PDF f(x) describes the relative likelihood. The integral over a range gives the probability. The Gaussian PDF is the most common: f(x) = 1 / (σ√(2π)) · exp(-(x-μ)² / (2σ²)).

Maximum Likelihood Estimation (MLE)

MLE finds parameters θ that maximize the probability of observed data. This is the core objective for most generative models. Equivalently, minimize negative log-likelihood (NLL).

# MLE for Gaussian distribution
def mle_gaussian(data):
    μ = mean(data)
    σ = std(data)
    return μ, σ
    # These μ, σ maximize P(data | μ, σ)

# In neural networks:
# loss = -log P_θ(data)  # minimize NLL

KL Divergence

KL divergence measures how one distribution P differs from another Q: D_KL(P || Q) = Σ P(x) log(P(x)/Q(x)). It's asymmetric and non-negative. Used in VAEs (KL loss) and distillation.

Bayesian Inference

Bayes' theorem updates beliefs given evidence: P(θ|D) = P(D|θ)·P(θ) / P(D). Posterior = Likelihood × Prior / Evidence. Used in Bayesian neural networks and some generative approaches.