Probability assigns a number between 0 and 1 to how likely an event is. A probability of 0 means the event cannot happen; 1 means it is certain; anything in between reflects uncertainty. Machine learning is essentially applied probability — every model output, every confidence score, every evaluation metric sits on top of probability ideas. This chapter covers the core rules that govern probabilities and then builds the catalog of distributions you will meet again and again in AI.
The addition rule tells us how to find the probability that at least one of two events occurs — that is, the probability of A or B.
Two events are mutually exclusive if they cannot both happen at the same time. A die roll cannot land on both 2 and 5, so those are mutually exclusive. For such events, the probability of A or B is simply the sum:
P(A or B) = P(A) + P(B)If both events can happen together (like drawing a card that is both a heart and a face card), we must avoid double-counting the overlap. The rule gains a correction term:
P(A or B) = P(A) + P(B) - P(A and B)Subtracting the joint probability undoes the double count of the region where both events occur.
The multiplication rule handles the probability that both events occur — A and B.
Two events are independent when knowing one happens gives you no information about the other. Coin flips are independent: the result of the first flip does not change the odds of the second. For independent events:
P(A and B) = P(A) * P(B)When events are dependent, the probability of the second depends on the first. Drawing two cards without replacement is the classic case — after removing one card, the deck changes. Here we multiply by the conditional probability:
P(A and B) = P(A) * P(B given A)This formula is really the definition of conditional probability rearranged, and it will reappear when we study Bayes' theorem in the next chapter.
Every random variable has a probability distribution, and there are three functions we use to describe it.
Distributions are mathematical models of how probabilities are spread across values. Choosing the right one for your data matters because models, simulations, and statistical tests all assume an underlying distribution. Below we tour the most important families, from the simplest discrete ones to the heavy-tailed giants.
The Bernoulli distribution models a single experiment with exactly two outcomes: success (1) or failure (0). Flipping a coin once, testing whether one email is spam, or checking whether one customer churns are all Bernoulli trials. It has a single parameter p, the probability of success:
P(X=1) = p
P(X=0) = 1 - pThe Binomial distribution counts the number of successes in n independent Bernoulli trials, each with the same success probability p. Tossing a fair coin 10 times and counting heads follows a Binomial(10, 0.5) distribution. Its PMF is:
P(X = k) = C(n, k) * p^k * (1 - p)^(n - k)where C(n, k) (read “n choose k”) counts how many ways k successes can be arranged among n trials. The Binomial has mean n*p and variance n*p*(1−p).
The Poisson distribution models the number of times an event occurs in a fixed interval of time or space, when events happen independently at a constant average rate. Examples: emails arriving in an hour, car accidents on a road each day, or website clicks per minute. Its single parameter is the average rate λ, and its PMF is:
P(X = k) = (e^(-lambda) * lambda^k) / k!Poisson data has the special property that its mean and variance are both equal to λ. It is the natural model for count data in many AI applications, such as anomaly detection on rare events.
The Normal distribution is the most famous distribution in statistics. Its PDF forms the familiar bell curve — symmetric about the mean μ, with a spread controlled by the standard deviation σ:
f(x) = (1 / (sigma * sqrt(2*pi))) * e^(-(x - mu)^2 / (2 * sigma^2))Heights, blood pressure, measurement errors, and countless natural measurements approximate a Normal distribution. The Empirical Rule is a practical takeaway: about 68% of values fall within 1 standard deviation of the mean, 95% within 2, and 99.7% within 3. Many ML algorithms explicitly or implicitly assume normally distributed data, which is why log transformations are so common.
The standard normal distribution is the special Normal with mean 0 and standard deviation 1. Any Normal value can be converted into a Z-score, the number of standard deviations it sits above or below the mean:
Z = (x - mu) / sigmaA Z-score of 2 means the value is 2 standard deviations above the mean. Z-scores put variables with different units onto a common scale, which is why they power standardizing features before feeding them into machine learning models — many algorithms behave badly when features have wildly different scales.
The Uniform distribution gives every value in a range an equal chance. Rolling a fair die is the discrete version; a random number chosen between 0 and 1 is the continuous version. Its PDF is flat across the interval, so the density is simply 1 divided by the width of the range. Uniform randomness is the raw material for everything from shuffling data to initializing neural-network weights.
A variable is log-normally distributed when the logarithm of that variable follows a Normal distribution. Income, stock prices, and house prices are classic examples — they cannot go below zero, and they have a long right tail where a few values are enormous. Log-normal data looks skewed on its original scale but symmetric once you take logs. This explains why transforming such features (log1p) so often improves machine learning models.
A power law describes relationships where one quantity varies as a power of another: roughly, P(X > x) is proportional to x−α. Power laws produce extreme imbalance — a small number of observations account for most of the total. Word frequencies in language, sizes of cities, and numbers of followers on social platforms all approximate power laws. They have heavy tails: extremely large values occur far more often than they would under a Normal distribution, so the mean can be misleading.
The Pareto distribution is the most famous power law. It is named after Vilfredo Pareto, who noticed that about 80% of the wealth in Italy belonged to roughly 20% of the population — the origin of the “80/20 rule.” A Pareto-distributed variable has a minimum value xm and a shape parameter α. As α gets smaller, the tail gets fatter and extreme values become more common. Understanding Pareto-style distributions helps you recognize when averages are meaningless and when rare-but-huge events dominate your data.
The Central Limit Theorem (CLT) is one of the most powerful ideas in statistics. It says: if you take many samples from a population, compute the mean of each sample, and plot those sample means, the distribution of the sample means will look approximately Normal — no matter what the underlying population looks like — as long as the sample size is large enough (commonly n ≥ 30).
Two consequences follow. First, the sample means cluster around the true population mean. Second, their spread shrinks as the sample size grows: the standard deviation of the sample means is σ/√n, where σ is the population standard deviation. This is the mathematical engine behind confidence intervals, the Z-test, and nearly every claim that a sample statistic can stand in for a population value. The CLT is why the Normal distribution is everywhere in statistics — it is the distribution that averages naturally converge to.
import numpy as np
import matplotlib.pyplot as plt
# Population: heavily skewed (exponential)
pop = np.random.exponential(scale=2.0, size=100000)
sample_means = []
for _ in range(10000):
sample_means.append(np.mean(np.random.choice(pop, size=50)))
plt.hist(sample_means, bins=40)
plt.title("Distribution of Sample Means (n=50)")
plt.show()