← Back to Tutorials Chapter 27

Anomaly Detection

Sometimes the most valuable pattern in your data is the one that appears once. Fraud, equipment failure, network intrusion, and manufacturing defects are all rare events buried inside otherwise normal records. Anomaly detection is the discipline of finding those rare events automatically. This chapter compares three very different strategies: Isolation Forest, DBSCAN, and Local Outlier Factor.

Anomaly Detection with Isolation Forest

In-Depth Intuition

Most outlier methods describe the normal region and flag points that violate it. Isolation Forest flips the idea: it does not try to profile the normal data at all. Instead it asks how easy each point is to isolate from the rest. An anomaly is, by definition, a point that is easy to separate — there are so few similar points around it that a few random cuts trap it on its own almost immediately.

The algorithm builds many random decision trees. In each tree it repeatedly picks a random feature, picks a random threshold on that feature, and cuts the data along it until every point sits in its own leaf. Normal points live in dense regions, so cutting them away takes many splits and they end up in deep leaves. Anomalies sit in sparse regions, so a single unlucky cut often strands them, and they end up in shallow leaves. The depth at which a point is isolated becomes its anomaly score.

How Splits Isolate Anomalies

A normal point is surrounded by neighbours, so random cuts peel points off one layer at a time — it takes a long path to leave it alone.
An anomaly is far from everything, so the first or second random split usually separates it into a leaf by itself.
Shallow average path length across all trees → anomalous. Deep average path length → normal.

The final score is normalized from these path lengths; lower scores mean more anomalous. In scikit-learn, IsolationForest returns 1 for normal points and -1 for anomalies, and also exposes a raw score_samples you can threshold yourself. Isolation Forest scales to huge datasets because it never computes distances between pairs of points.

DBSCAN Clustering for Anomaly Detection

Recall from the previous chapter that DBSCAN labels points as core, border, or noise. The noise label is a free anomaly detector: points that live in low-density regions have no core point within eps and get labeled -1.

from sklearn.cluster import DBSCAN
from sklearn.preprocessing import StandardScaler

X_std = StandardScaler().fit_transform(X)

db = DBSCAN(eps=0.5, min_samples=5)
labels = db.fit_predict(X_std)
anomalies = X[labels == -1]   # rows labeled as noise

Two settings shape the outcome:

eps is the radius that defines "near". Too small and every sparse point becomes noise; too large and true anomalies get absorbed into a cluster.
min_samples is the minimum neighbourhood size that counts as dense. Raise it to be stricter about what counts as normal.

The strength of DBSCAN here is that it needs no assumption about the shape of the data — if your normal data forms a crescent, DBSCAN models the crescent and still flags points far from it.

Practical tip: DBSCAN's anomaly set changes dramatically with eps. Instead of guessing, plot a 2D projection of the data and try a few values, watching how the number of noise points changes. A sudden jump in noise between two eps values usually means you have passed the density of your real clusters.

Local Outlier Factor for Anomaly Detection

Local Outlier Factor (LOF) is a density-based method that compares each point's density with the density of its nearest neighbours. The core idea is that anomalies live in regions that are sparse relative to their surroundings.

Find the k nearest neighbours of each point.
Compute each point's reachability distance to its neighbours (a modified distance that smooths out ties inside dense clusters).
Define the local reachability density: the inverse of the average reachability distance to the neighbours.
Compute the LOF score: the average ratio of the neighbours' density to the point's own density.

A point whose density is close to its neighbours' has an LOF score near 1 and is normal. A point that is far sparser than its neighbours has a score well above 1 and is an outlier.

from sklearn.neighbors import LocalOutlierFactor

lof = LocalOutlierFactor(n_neighbors=20, contamination="auto")
outlier_mask = lof.fit_predict(X)   # -1 marks anomalies
print(lof.negative_outlier_factor_)  # raw LOF scores (negative)

LOF's great advantage is that it is truly local: a cluster at density 100 and a cluster at density 5 are both "normal" internally, and points outside both are flagged. Global methods would misread the sparser-but-valid cluster as anomalous.

Comparing the Three Approaches

Speed and scale: Isolation Forest is the fastest for large, wide datasets. DBSCAN is fast for clustering-like work but needs the eps tuned. LOF is nearest-neighbour based, so it slows down as rows grow and struggles in very high dimensions.
Assumptions: Isolation Forest assumes anomalies are few and easily cut off; DBSCAN assumes anomalies are points outside dense regions; LOF assumes anomalies are points whose neighbourhood is locally sparse.
Shapes: DBSCAN and LOF respect arbitrary cluster shapes; Isolation Forest is geometry-agnostic and works well even when features are not numeric-friendly after scaling.
Parameter sensitivity: Isolation Forest mostly needs contamination (the expected fraction of anomalies). DBSCAN hinges on eps. LOF hinges on n_neighbors.
Interpretability: DBSCAN is easy to explain ("point with no neighbours around it"); LOF is intuitive ("sparser than its neighbours"); Isolation Forest is the least visual but the most robust default.

Choosing: when in doubt on tabular data, run Isolation Forest first — it needs the least tuning and tolerates high dimensions. Use DBSCAN when you also want the clusters themselves or when your data has strong spatial structure. Use LOF when you have many points and your anomalies are subtle local deviations, not global extremes.

A Practice Exercise

Practice: load the scikit-learn make_blobs dataset with 3 well-separated clusters and 300 points, then inject 20 synthetic anomalies by shifting 20 rows far away from all cluster centers. Run IsolationForest, DBSCAN, and LocalOutlierFactor on the same data (standardize first). For each method, count how many of the 20 injected anomalies it flags, how many normal points it wrongly flags, and report both numbers side by side. Then repeat once with only 5 injected anomalies and note which method degrades most — the winner tells you which approach to trust when anomalies are extremely rare.