Every model we have seen so far — linear regression, logistic regression, support vector machines — draws a smooth mathematical boundary. Decision trees take a completely different route: instead of fitting an equation, they ask a sequence of yes/no questions until the answers lead to a decision. A tree is the machine-learning cousin of a flowchart, and because every question is human-readable, trees are among the easiest models to explain to a non-technical colleague.
A decision tree is built from three kinds of components: a root node at the top, internal nodes in the middle, and leaf nodes at the bottom. Every node except the root has one parent, and every internal node asks a single question about one feature.
Imagine predicting whether a shopper will buy a product. The root might ask "is annual income above 50,000?", a child node might ask "has this user visited the site this month?", and the leaves output "will buy" or "will not buy". To predict for a new person you simply walk down the tree, answering each question with their feature values. Because each question examines only one feature at a time, trees make no assumption that the data is linearly separable — a single tree can carve the feature space into rectangles using only horizontal and vertical boundaries.
To pick a good question, the tree needs a way to measure how mixed a group of labels is. That measure is called impurity, and the two most common impurity functions are entropy and the Gini index.
Entropy, borrowed from information theory, measures the average amount of surprise in a set. For classes with probabilities p1, p2, ..., pk, entropy is:
H = -sum( pi * log2(pi) ) for each class iIf all samples belong to one class, one pi equals 1 and its log is 0, so H = 0 — the node is perfectly pure. If two classes are split 50/50, then H = -(0.5 * -1 + 0.5 * -1) = 1 — maximum uncertainty. Entropy is measured in bits and always lands between 0 and log2(number of classes).
Gini impurity asks a simpler question: what is the probability that two items drawn at random from the node belong to different classes?
G = 1 - sum( pi^2 ) for each class iA pure node has G = 0. A 50/50 two-class node has G = 0.5. Both functions are at their minimum when the node is pure and grow as the labels become more evenly spread.
Information gain measures how much a split reduces impurity. The tree picks the question that offers the largest drop, because a big drop means the children are much cleaner than the parent. For a classification tree built on entropy:
IG = H(parent) - sum( (n_child / n_parent) * H(child) )Work through a concrete example:
The algorithm computes the information gain of every candidate question and greedily picks the winner, then repeats the process inside each child until a stopping rule fires.
DecisionTreeClassifier uses gini by default; pass criterion="entropy" to switch.A continuous feature has infinitely many possible split points, so the tree needs a practical strategy:
For an age column holding [18, 25, 34, 41, 55], the candidates are the midpoints 21.5, 29.5, 37.5 and 48. The tree evaluates all of them and picks the best. Because thresholds are chosen greedily, a deep tree on one numerical feature behaves like a staircase function made of many parallel cuts.
Trees are eager learners: give them enough depth and they will memorize the training set, noise included. That is overfitting, and two families of remedies exist.
Pre-pruning stops the tree from growing too large in the first place:
max_depth: limit the number of levels.min_samples_split: require at least this many samples before a node may split.min_samples_leaf: every leaf must hold at least this many samples.max_features: consider only a random subset of features at each split.Post-pruning lets the tree grow fully and then cuts branches back. The common approach is cost-complexity pruning, where each branch pays a penalty proportional to the leaves it adds; branches that do not improve validation performance enough are removed. In scikit-learn you can read the candidate alphas with cost_complexity_pruning_path, then train a tree for each alpha and keep the one with the best test score.
Regression trees predict a number rather than a class. A leaf's prediction is simply the mean of the target values inside that region, and the impurity measure is replaced by variance or mean squared error. Information gain becomes variance reduction:
VR = Var(parent) - sum( (n_child / n_parent) * Var(child) )The tree keeps choosing the split that most reduces the variance of the target in its children. The result is a piecewise-constant function — a staircase — which is why deep regression trees look jagged and shallow ones look smooth.
Implementing a tree in scikit-learn takes a few lines:
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
clf = DecisionTreeClassifier(random_state=42)
clf.fit(X_train, y_train)
print(clf.score(X_test, y_test))Visualize the learned rules with sklearn.tree.plot_tree, or print them as text with export_text. A rule looks like "petal width <= 0.8 → class 0", which you can read as a plain sentence.
Pre-pruning is where you steer the bias-variance trade-off. A tree with no limits memorizes the training data; a tree with max_depth=1 (a single stump) is barely better than guessing on some problems. A quick experiment:
from sklearn.tree import DecisionTreeRegressor
for depth in [1, 3, 5, None]:
tree = DecisionTreeRegressor(max_depth=depth, random_state=42)
tree.fit(X_train, y_train)
print(depth, round(tree.score(X_train, y_train), 3),
round(tree.score(X_test, y_test), 3))Typical output shows the training score climbing as depth grows while the test score peaks somewhere in the middle — that peak is your sweet spot. Use GridSearchCV over max_depth, min_samples_leaf and min_samples_split to find it systematically.
We finish with a small project. The diabetes dataset bundled with scikit-learn contains ten medical measurements per patient and a target that is a quantitative measure of disease progression one year after baseline.
from sklearn.datasets import load_diabetes
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
data = load_diabetes()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
tree = DecisionTreeRegressor(max_depth=4, min_samples_leaf=5, random_state=42)
tree.fit(X_train, y_train)
y_pred = tree.predict(X_test)
print("R2 on train:", round(tree.score(X_train, y_train), 3))
print("R2 on test :", round(tree.score(X_test, y_test), 3))
print("MAE :", round(mean_absolute_error(y_test, y_pred), 2))max_depth=3, and one post-pruned with cost-complexity pruning. Report the train and test MAE of each and explain which generalizes best. Extra credit: use export_text to print the rules of the best tree and describe, in plain English, what its first two splits mean.