The previous chapter showed that a single decision tree overfits easily. The fix is one of the most powerful ideas in machine learning: instead of trusting one model, combine many. This chapter builds the three great families of ensembles — bagging (random forests), classic boosting (AdaBoost), and modern gradient boosting (including XGBoost) — and walks through a complete random forest regression project.
An ensemble is a collection of weak-ish models whose outputs are combined. Bagging and boosting are the two main recipes for building one.
Both approaches turn weak learners into strong ones. Bagging mainly reduces variance; boosting reduces both bias and variance.
A random forest is bagging applied to decision trees with one crucial twist: at every split, each tree may only consider a random subset of the features. This forces the trees to disagree even more, and averaging over more disagreement reduces variance further.
For regression, every tree outputs a number and the forest returns the mean of those numbers. For classification, each tree votes and the most popular class wins. The two dominant controls are n_estimators (number of trees) and max_depth (how deep each tree may grow).
from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor(n_estimators=100, max_depth=6, random_state=42)
rf.fit(X_train, y_train)
print(rf.score(X_test, y_test))Before writing code, classify the task. Ask three questions:
feature_importances_). They are an excellent default choice for tabular data.
Feature engineering for trees starts with data hygiene. Handle missing values (impute with the median or most frequent value), drop near-duplicate rows, and check data types — dates and strings are usually not what the model wants. Then look at the distribution of each feature with histograms to catch outliers and skewed columns. Trees are invariant to monotone transformations, so you do not need to scale or normalize anything.
Next, encode categorical columns. Ordinal categories (small/medium/large) can use label encoding; unordered categories should use one-hot encoding so the tree can split on each level individually. Then create derived features that capture structure the raw columns hide — for a car dataset, "age of the car = current year - model year" or "price per kilometre" are informative. Finally, drop redundant columns (two features that are perfectly correlated add noise, not signal).
n_estimators, max_depth and min_samples_leaf with GridSearchCV.Suppose we must predict the selling price of used cars from features like age, mileage, fuel type, transmission, and owner count. The target is continuous, so this is a regression problem, and the natural metrics are MAE and R-squared.
import pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv("cars.csv")
df["age"] = 2026 - df["year"] # derived feature
df = pd.get_dummies(df, columns=["fuel", "transmission"], drop_first=True)
df = df.fillna(df.median(numeric_only=True))
X = df.drop(columns=["price"])
y = df["price"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
rf = RandomForestRegressor(n_estimators=150, max_depth=10, min_samples_leaf=3,
random_state=42)
rf.fit(X_train, y_train)
pred = rf.predict(X_test)
print("R2 :", round(rf.score(X_test, y_test), 3))
print("MAE:", round(mean_absolute_error(y_test, pred), 2))
print(sorted(zip(X.columns, rf.feature_importances_),
key=lambda t: t[1], reverse=True)[:5])The feature importances tell you which columns actually drive price — often age and mileage dominate, confirming the intuition behind the engineered features.
max_depth and raise min_samples_leaf rather than adding more trees.
AdaBoost (Adaptive Boosting) chains many weak learners instead of averaging parallel ones. The classic weak learner is a decision stump — a tree of depth one, which asks a single question. Rather than resampling rows, AdaBoost assigns each training example a weight and adjusts those weights after every stump.
Start with equal weights (all 1/N). Train a stump on the weighted data and record its weighted error e. A stump is weak by design, so e is usually less than 0.5 but not near zero.
The stump's importance is its voting weight, computed from its error:
alpha = 0.5 * ln( (1 - e) / e )When e is small, alpha is large and that stump's vote counts heavily; when e approaches 0.5 (random guessing), alpha approaches 0.
After each stump, every training example's weight is updated so the next stump is forced to pay attention to the hard cases:
misclassified: w = w * exp(alpha)
correctly classified: w = w * exp(-alpha)Misclassified points grow heavier; correctly classified points shrink.
Divide every weight by the total so they sum to 1 and behave like a probability distribution. Then arrange the points along a line from 0 to 1, each occupying a segment proportional to its weight — these segments are the "bins" used for sampling.
To build the next stump's dataset, draw N points with replacement, where the chance of picking a point equals its bin width. Heavier points (the mistakes) are sampled more often, so each new stump sees a version of the data biased toward what the ensemble currently gets wrong.
Prediction is a weighted vote. Every stump predicts, and its vote counts alpha times. The class receiving the largest total weighted vote wins. (For multiclass problems, scikit-learn uses the SAMME variant, which extends the same idea.)
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
base = DecisionTreeClassifier(max_depth=1, random_state=42)
ada = AdaBoostClassifier(estimator=base, n_estimators=50,
learning_rate=1.0, random_state=42)
ada.fit(X_train, y_train)
print(ada.score(X_test, y_test))AdaBoost handles regression too, by reweighting points whose errors are largest:
from sklearn.ensemble import AdaBoostRegressor
ada_r = AdaBoostRegressor(n_estimators=50, learning_rate=1.0, random_state=42)
ada_r.fit(X_train, y_train)
print(ada_r.score(X_test, y_test))AdaBoost is sensitive to noise: when the data contains many outliers, the boosting weights keep chasing points that are impossible to learn, and performance degrades.
Gradient boosting drops reweighting entirely. Each new tree is fit to the residuals (the negative gradient of the loss) of the current ensemble. The recipe is simple:
n_estimators rounds.Because each tree chases the leftover error, boosting is a low-bias method. The learning rate (often 0.01 to 0.1) controls how hard each tree pushes; smaller rates need more trees but generalize better.
from sklearn.ensemble import GradientBoostingClassifier
gbc = GradientBoostingClassifier(n_estimators=100, max_depth=3,
learning_rate=0.1, random_state=42)
gbc.fit(X_train, y_train)
print(gbc.score(X_test, y_test))from sklearn.ensemble import GradientBoostingRegressor
gbr = GradientBoostingRegressor(n_estimators=100, max_depth=3,
learning_rate=0.1, random_state=42)
gbr.fit(X_train, y_train)
print(gbr.score(X_test, y_test))Gradient boosting usually beats random forests on raw accuracy, but it overfits faster if depth or the number of estimators is too large — treat validation curves as a must.
XGBoost (Extreme Gradient Boosting) is gradient boosting with serious engineering and regularization. Three ideas set it apart:
Each leaf's score is closed-form. For a leaf with gradients summing to G and Hessians summing to H, the optimal score is:
leaf score = -G / (H + lambda)The lambda term regularizes: a larger lambda shrinks leaf values and prevents huge jumps. XGBoost also handles missing values natively by learning which direction to route them, which removes a whole preprocessing step.
For regression the loss is squared error, whose Hessian is constant, so the math simplifies — but the same regularization and subsampling machinery applies, making XGBoost just as powerful for continuous targets.
import xgboost as xgb
xgb_clf = xgb.XGBClassifier(n_estimators=100, max_depth=4,
learning_rate=0.1, subsample=0.8, random_state=42)
xgb_clf.fit(X_train, y_train)
print(xgb_clf.score(X_test, y_test))xgb_reg = xgb.XGBRegressor(n_estimators=150, max_depth=4,
learning_rate=0.05, subsample=0.8, random_state=42)
xgb_reg.fit(X_train, y_train)
print(xgb_reg.score(X_test, y_test))XGBoost ships with its own plotting utilities (xgb.plot_importance) and built-in cross-validation, and it is the go-to baseline for tabular competition data.
california_housing or diabetes data) and train four models on the same split: RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor, and XGBRegressor. Compare their test R-squared and MAE side by side, then check which features each model ranks most important. Write one sentence explaining why the boosted models did or did not outperform the forest.