A straight line is a poor fit for curves, and a curve trained without discipline memorizes its training data. This chapter gives you two tools: polynomial features to bend the line, and regularization to keep the bending honest. Together they turn linear regression into a genuinely capable workhorse.
When a scatter plot shows a curve — sales rising then falling with price, say — a straight line underfits. The trick is to keep using a linear model but feed it nonlinear features. Instead of predicting from x alone, we create features x, x^2, x^3, ... and fit:
y_hat = w0 + w1*x + w2*x^2 + w3*x^3
This is still linear in the parameters — that is what makes it a linear regression — but the feature map lets the fitted curve turn. With degree 1 you get a line, degree 2 a parabola, degree 3 an S-curve. Higher degree means more bends, and more bends means more freedom to overfit.
In scikit-learn you create the engineered features with PolynomialFeatures and feed them to a linear regression:
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
X = np.linspace(0, 10, 50).reshape(-1, 1)
y = 3 * X.ravel()**2 + np.random.normal(0, 30, size=50)
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
model = LinearRegression()
model.fit(X_poly, y)
print(model.coef_) # w1, w2 for x and x**2Compare degrees 1, 3, and 10 on a validation split. You will watch training error fall while test error eventually rises — that hump is overfitting appearing before your eyes.
Polynomial features often explode feature scales (x^4 dwarfs x), so scaling must happen between transformation and fitting. A Pipeline bundles the steps into one object that can be trained and reused without leaking:
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipe = Pipeline([
("poly", PolynomialFeatures(degree=3)),
("scaler", StandardScaler()),
("model", LinearRegression()),
])
pipe.fit(X, y)Pipelines guarantee the exact same transformations are applied at training and prediction time — a silent bug killer.
Ridge adds a penalty to the cost function proportional to the sum of squared weights:
J = MSE + alpha * sum(w_j^2)
The penalty shrinks every coefficient toward zero, never exactly to zero. Large alpha pushes harder, producing a smoother, more stable fit. Ridge shines when many features are modestly useful and you want to curb extreme coefficients without discarding features.
Lasso uses the sum of absolute weights instead of squares:
J = MSE + alpha * sum(|w_j|)
That sharp-cornered penalty drives some coefficients to exactly zero, so Lasso performs automatic feature selection — it silently drops the features that do not help. ElasticNet blends both penalties:
J = MSE + alpha * (r * sum(|w_j|) + (1 - r) * sum(w_j^2))
The l1_ratio parameter r controls the mix. ElasticNet is the practical favorite when many features are correlated, because pure Lasso can pick one feature from a correlated group and discard the rest arbitrarily.
from sklearn.linear_model import Ridge, Lasso, ElasticNet
models = {
"ridge": Ridge(alpha=1.0),
"lasso": Lasso(alpha=0.1),
"elastic": ElasticNet(alpha=0.1, l1_ratio=0.5),
}A single train/test split wastes data and its answer depends on luck. Cross-validation reuses every example for both training and testing:
k equal folds; each fold becomes the test set once, the model trains on the rest. The reported score is the average across the k runs. Typical values are 5 or 10.k = n, training on everything but one point each time. Expensive but nearly unbiased; good for tiny datasets.from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipe, X, y, cv=5, scoring="r2")
print(scores.mean(), scores.std())Before any model, the raw data needs hygiene (this was Chapter 16's theme). Drop duplicate rows, handle missing values (impute with the median, or drop if sparse), remove truly nonsensical values (negative price, a person aged 200), and convert categorical columns into a numeric form.
Explore before you model: histograms and box plots reveal skewed distributions; scatter matrices expose correlations and outliers. Feature engineering then creates new columns that capture structure — a price_per_sqft ratio, the log of a heavily skewed amount, the number of bathrooms squared. Every engineered feature is a hypothesis the model can test.
Not every column deserves a weight. Filter methods score each feature against the target (f_regression, correlation), wrapper methods try subsets and keep the best, and embedded methods — like Lasso — select features while training. Fewer features mean faster training, simpler interpretation, and less noise for the model to memorize.
With clean, well-chosen features, train a small portfolio of models — Ridge, Lasso, ElasticNet, and a plain LinearRegression as baseline — under identical cross-validation so their scores are comparable. Choose the simplest model whose score is statistically indistinguishable from the best; simplicity wins.
Regularization strength (alpha) and mix (l1_ratio) are not learned from data; you must try candidates. A grid search over a few sensible values, scored with cross-validation, finds the sweet spot. In Chapter 21 we will automate this with GridSearchCV; for now, a small manual loop over alpha values is enough to see how error changes.
End to end: load a single-feature dataset, split it, fit, and report MAE/RMSE on the test set. A classic starter is study hours versus score, or temperature versus ice cream sales. The deliverable is a slope, an intercept, and a one-line scatter-plus-line plot that shows the fit.
Multiple regression quietly assumes: a linear relationship, independence of errors, constant error variance (homoscedasticity), normally distributed errors, and no severe multicollinearity between features. You can check each with a residual plot (points should scatter randomly around zero, not form a funnel), a Q-Q plot of residuals, and a correlation matrix. When an assumption is violated, log-transform a feature or add a polynomial term.
Build a Boston-housing-style workflow yourself. Steps: load data; check shape and missing values; visualize the target's distribution and its relationship to the strongest features; engineer new features; standardize; split; fit a baseline; measure error. The discipline of doing this without a tutorial is what turns knowledge into skill.
Lasso doubles as a feature selector, so train it under cross-validation and inspect which coefficients were pushed to zero. If Lasso zeroes out a column you believed in, either the feature is redundant with others or it was never predictive in the first place.
from sklearn.linear_model import LassoCV
lasso = LassoCV(cv=5, random_state=42).fit(X_train, y_train)
print(lasso.alpha_)
print(lasso.coef_)RidgeCV and ElasticNetCV scan their regularization parameters internally using cross-validation, so you hand them a grid and get back the best configuration plus a trained model. Compare the three CV-selected models' test RMSE; the differences between them tell you about the structure of your data (pure noise, correlated features, or irrelevant columns).
Training is expensive; prediction should be cheap. The joblib (or pickle) module serializes the fitted model to disk so a server can reload it later without retraining:
import joblib
joblib.dump(pipe, "model.pkl")
loaded = joblib.load("model.pkl")
loaded.predict(new_X)Save the full pipeline, not just the final estimator, so preprocessing and prediction stay consistent.
A production-ready project has a folder structure: raw and processed data directories, a src package with modules for ingestion, cleaning, feature engineering, and training; configuration files for parameters; and a saved artifact. Version everything, log key decisions, and keep the pipeline rerunnable from one command. Chapter 29 will turn this into a full template.
Once the model is saved, expose it. The common paths on AWS: wrap the pickled pipeline in a Flask or FastAPI app and run it on EC2; push the same app as a container to Amazon ECR and run it on Elastic Beanstalk; or host the serialized model on S3 and have a serverless function load it on demand. Whichever you choose, the model file is just an artifact — deployment is plumbing around it.
Technically, yes — and it usually does it badly. If you code the classes as 0 and 1, a line can fit the numeric labels, but three things go wrong:
The lesson is important: your choice of model should follow your problem type. For categorical outcomes you need a classifier with a probability output — which is exactly the subject of Chapter 21.
y = sin(x) + noise). Fit polynomial degrees 1, 3, 8, and 15, and record train vs test MSE for each. Now fit degrees 8 and 15 with Ridge (alpha = 10) and compare test errors. Write a paragraph explaining, in your own words, exactly how the penalty rescues the high-degree fits.