Linear Regression

Linear regression is the doorway to predictive modeling. It is simple enough to understand completely, yet it introduces nearly every idea you will reuse for the rest of your ML career: a cost function, a training loop, overfitting, and evaluation metrics. Master it and the harder models become variations on a theme.

Simple Linear Regression Introduction

Simple linear regression predicts a continuous target y from a single feature x by drawing a straight line through the scatter plot. The line captures the general trend — for example, "each additional hour studied raises the expected score by about six points." The model is a model-based learner (recall Chapter 18): the entire dataset is condensed into two numbers, the slope and intercept.

Understanding Simple Linear Regression Equations

The model's prediction for input x is:

y_hat = w*x + b

Here w is the slope (how many units y moves per unit of x) and b is the intercept (the predicted value when x is zero). Each training example contributes a residual, the vertical gap between the actual value y and the prediction y_hat:

residual = y - y_hat

Training is the process of choosing w and b so that the residuals across all examples are, taken together, as small as possible.

The Cost Function

We need a single number that summarizes how bad a given (w, b) pair is. The standard choice is the mean squared error (MSE):

J(w, b) = (1 / n) * sum( (y_i - (w*x_i + b))**2 )

Why square the errors? Three reasons stand out:

• Squaring removes the sign, so a big over-prediction and a big under-prediction are punished equally.
• It punishes large errors far more than small ones, because the square grows quickly.
• The squared-error function is smooth and convex, which guarantees the math we are about to do will always find the best line.

Plotting J against w for a fixed b gives a smooth U-shaped bowl. The bottom of that bowl is the slope of the best-fit line.

The Convergence (Gradient Descent) Algorithm

Instead of trying every possible w, gradient descent walks downhill on the cost surface. At any point, the gradient tells us the direction of steepest ascent; we step in the opposite direction to descend:

1. Start with arbitrary w and b (often zero).
2. Compute the gradient: the derivative of J with respect to w and b.
3. Update: w = w - learning_rate * dJ/dw and b = b - learning_rate * dJ/db.
4. Repeat until the cost stops changing meaningfully.

import numpy as np

def gradient_descent(X, y, lr=0.01, epochs=1000):
    w = b = 0.0
    n = len(X)
    for _ in range(epochs):
        pred = w * X + b
        dw = (-2 / n) * np.sum(X * (y - pred))
        db = (-2 / n) * np.sum(y - pred)
        w -= lr * dw
        b -= lr * db
    return w, b

Gradient Descent Part Two: Learning Rate and Convergence

Two knobs control whether descent succeeds. The learning rate (usually written as alpha) decides how large each step is:

• Too large: the steps overshoot the bowl's bottom, the cost may oscillate or even diverge to infinity.
• Too small: training is reliable but painfully slow — the algorithm may need millions of iterations.
• Just right: cost drops steadily and flattens out.

Convergence is declared when the improvement in cost between iterations falls below a small tolerance, or after a fixed number of epochs. It helps to watch the learning curve: a plot of cost against iteration should look like a smooth descending slide, not a sawtooth. On a purely convex cost like MSE, gradient descent is guaranteed to reach the global minimum; on the bumpy surfaces of deep networks we can only guarantee a good local minimum.

Multiple Linear Regression

Real problems rarely have one feature. With d features x1, x2, ..., xd, the model becomes a hyperplane (Chapter 18):

y_hat = w1*x1 + w2*x2 + ... + wd*xd + b

In vector form this is y_hat = w·x + b. The gradient descent update looks identical to the simple case — just compute the gradient for each weight separately — but now the model is a whole family of lines, one per feature. The coefficient wj is interpreted as the change in y for a one-unit change in xj, holding all other features constant.

Performance Metrics for Regression

After training we need to score the model on unseen data. A family of related metrics all starts from the residuals.

Mean Squared Error (MSE)

The average squared residual: MSE = (1/n) * sum((y_i - y_hat_i)**2). It is the most common training loss, but its units are squared (a "score error squared"), which makes it hard to interpret directly.

Root Mean Squared Error (RMSE)

Taking the square root returns the metric to the original units: RMSE = sqrt(MSE). If house prices are in dollars, RMSE is in dollars, so it reads like "typical error size." RMSE penalizes large outliers more than small ones.

Mean Absolute Error (MAE)

The average absolute residual: MAE = (1/n) * sum(|y_i - y_hat_i|). It treats every error equally, so it is more robust when your data contains a few extreme outliers that would otherwise dominate RMSE. A rule of thumb: if outliers are meaningful, prefer RMSE; if you want a typical error that ignores freakish points, prefer MAE.

from sklearn.metrics import mean_absolute_error, mean_squared_error

mae = mean_absolute_error(y_true, y_pred)
mse = mean_squared_error(y_true, y_pred)
rmse = mean_squared_error(y_true, y_pred, squared=False)
print(mae, mse, rmse)

Overfitting and Underfitting

A straight line can only be wrong in two directions. If the true relationship is curved and the line is too rigid to follow it, the model is underfitting — high error on both training and test data. If the model were given too much freedom (say, a polynomial with hundreds of terms), it could memorize every training point while generalizing terribly; that is overfitting — tiny training error, large test error.

The fix for underfitting is more capacity: more features or a more flexible model (Chapter 20). The fix for overfitting is regularization, more data, or simpler models. The general lesson: a model's worth is judged on data it has never seen, so always hold out a test set.

Linear Regression with OLS (Ordinary Least Squares)

Gradient descent is a general hammer, but for linear regression there is a closed-form solution. OLS solves for the weights directly by setting the gradient to zero and solving the resulting equations. In matrix form:

w = (X^T X)^(-1) X^T y

where X is the design matrix with a column of ones for the intercept. OLS is exact and needs no learning rate, but computing (X^T X)^(-1) becomes expensive for large feature sets. sklearn's LinearRegression uses this style of linear algebra under the hood, so beginners get exact answers without tuning.

Simple Linear Regression Practical

Let's put everything together with hours studied against exam score:

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

X = np.array([1, 2, 3, 4, 5, 6, 7, 8]).reshape(-1, 1)
y = np.array([52, 55, 61, 65, 72, 78, 83, 90])

model = LinearRegression()
model.fit(X, y)
print(model.coef_[0], model.intercept_)   # slope, intercept
print(mean_squared_error(y, model.predict(X)))

The slope tells us the expected score gain per extra hour; the intercept is the predicted score for zero hours.

Multiple Linear Regression Practical

Now add a second feature so the model becomes a plane over the feature space:

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# features: hours, previous score ; target: final score
data = pd.DataFrame({
    "hours": [3, 5, 7, 2, 8, 4, 6, 1],
    "prev":  [60, 65, 70, 55, 80, 58, 75, 40],
    "final": [65, 72, 85, 58, 92, 66, 80, 50],
})
X = data[["hours", "prev"]]
y = data["final"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)
print("R2:", model.score(X_test, y_test))

Comparing the simple and multiple versions of the same problem teaches you what an extra feature is worth — and whether the added complexity actually reduces test error.