Inference is about learning about a population from a sample, and the tools we use are called estimators. An estimate comes in two flavors:
A good estimator is unbiased (on average it hits the true value, which is why sample variance divides by n−1) and consistent (it gets closer as the sample grows). The Central Limit Theorem from the previous chapter guarantees that the sample mean is both.
Hypothesis testing is a formal procedure for deciding whether the evidence in your data supports a claim. You start with two competing statements:
The mechanism is a proof by contradiction, in statistical clothing. You assume the null is true, then ask: if the null were true, how likely would we be to see a sample as extreme as the one we actually got? If that probability is tiny, the null looks implausible and you reject it, concluding the evidence favors the alternative. If the probability is not tiny, you fail to reject the null — which is not proof that the null is true, only that the evidence is not strong enough to discard it.
The test produces a test statistic (a number computed from your sample), and you compare it against a known distribution under the null to get a p-value.
The p-value is the probability of observing a result as extreme as, or more extreme than, the one you got — assuming the null hypothesis is actually true. A small p-value means your data would be very unlikely under the null, which is evidence against it.
We compare the p-value to a threshold α (commonly 0.05). If p < α, we say the result is statistically significant and reject the null. If p ≥ α, we fail to reject it.
The Z-test compares a sample mean to a known value when the population standard deviation σ is known and the sample size is reasonably large. Because the CLT makes the sample mean approximately Normal, we can build the test statistic:
Z = (xbar - mu0) / (sigma / sqrt(n))where x̄ is the sample mean, μ₀ is the hypothesized mean under the null, σ is the population standard deviation, and n is the sample size. The Z statistic measures how many standard errors the sample mean sits away from the hypothesized value. We then compare Z to the critical values of the standard Normal distribution (roughly ±1.96 for a two-sided test at α = 0.05). If |Z| exceeds the critical value, we reject the null.
In practice σ is almost never known, which is why the t-test — coming next — is used far more often in real work.
The Student t-distribution is the Normal distribution's cousin for small samples. When you have to estimate the population standard deviation from the sample itself, the extra uncertainty widens the tails of the distribution. The t-distribution looks like a bell curve but with heavier tails, so extreme values are a bit more likely than under the Normal. Its exact shape depends on the degrees of freedom (typically n−1 for a one-sample problem).
As the degrees of freedom grow, the t-distribution approaches the standard Normal. At n = 30 or so the two are nearly indistinguishable — one more place the “large enough sample” rule of thumb comes from.
The t-test is used when the population standard deviation is unknown and must be estimated from the sample. The t-statistic is:
t = (xbar - mu0) / (s / sqrt(n))where s is the sample standard deviation and the other terms match the Z-test. The recipe mirrors the Z-test exactly, except we compare t to the t-distribution with n−1 degrees of freedom.
There are three common variants:
from scipy import stats
group_a = [78, 82, 85, 79, 90, 88]
group_b = [70, 74, 68, 72, 75, 71]
t_stat, p_value = stats.ttest_ind(group_a, group_b)
print("t:", t_stat, "p:", p_value)The choice between them is driven by what you know and how big your sample is:
The logic and interpretation are identical; the t-test just accounts for the extra uncertainty that comes with estimating σ. For almost all real datasets, the t-test is the safe default.
Hypothesis tests can go wrong in two ways:
There is always a trade-off. Tighten α to avoid false positives and you make false negatives more likely (power drops). Loosen it and you catch more real effects but cry wolf more often. In machine learning, these errors show up everywhere: a spam filter flagging a normal email is a Type 1 error; letting spam through is Type 2.
Bayes' theorem updates a belief when new evidence arrives. It turns the conditional-probability idea from the probability chapter into a formula:
P(A | B) = (P(B | A) * P(A)) / P(B)In plain terms:
Classic example: suppose a disease affects 1% of people (prior), a test is 99% accurate at detecting it when present, and has a 5% false-positive rate. If someone tests positive, the chance they truly have the disease is far below 99% — Bayes' theorem shows how to correctly combine the prior with the evidence. Bayes' theorem is the foundation of Naive Bayes classifiers, Bayesian inference, and a great deal of modern AI thinking.
A confidence interval gives a range of values that is likely to contain the true population parameter. A 95% confidence interval for the mean takes the form:
CI = xbar +/- z* (sigma / sqrt(n))where z* is the critical value (about 1.96 for 95%), and the term z* × σ/√n is the margin of error. Correctly read, it means: if you repeated the whole sampling-and-interval process many times, about 95% of the intervals you built would contain the true mean. It is not a statement that the true mean has a 95% chance of sitting inside this particular interval.
Three things shrink the margin of error: a larger sample (n in the denominator), a smaller population spread σ, or a lower confidence level. When a news poll reports “±3%” that is the margin of error in action.
The chi-square test works on categorical data. It compares the counts you actually observed in each category against the counts you would expect if some claim were true. The test statistic sums up how far observed counts stray from expected ones:
chi2 = sum((observed - expected)^2 / expected)A large chi-square statistic — meaning big gaps between observed and expected — gives a small p-value and evidence that the observed pattern is not pure chance.
The goodness-of-fit test checks whether a sample matches a claimed distribution. For example: you roll a die 600 times and want to know if it is fair. If fair, each face should appear about 100 times. The chi-square goodness-of-fit test compares your 600 observed counts against those 100-per-face expectations and decides whether the deviations are too large to be coincidence. The degrees of freedom are the number of categories minus 1.
Analysis of Variance (ANOVA) extends the two-group t-test to three or more groups. It answers: “do the means of several groups differ, or are the differences just noise?” The null hypothesis is that all group means are equal; the alternative is that at least one differs.
The F-statistic that drives the test is a ratio:
F = (between-group variance) / (within-group variance)If the variation between groups is large relative to the variation within groups, the groups are genuinely different and F is large, producing a small p-value.
ANOVA rests on assumptions you should check before trusting the result:
When assumptions break badly, non-parametric alternatives like the Kruskal-Wallis test come into play. ANOVA only tells you that some group differs — follow-up pairwise tests (with corrections for multiple comparisons) identify which ones.
ANOVA comes in several flavors depending on how many factors you study:
ANOVA's name comes from how it works: it partitions the total variation in the data into additive pieces. The total sum of squares (variation of every point around the grand mean) splits into:
SST = SSB + SSWDividing each sum of squares by its degrees of freedom yields mean squares, and the F-ratio is MSB/MSW. This decomposition — separating signal (between-group differences) from noise (within-group variation) — is the same idea behind regression's R-squared and is worth internalizing.