Support Vector Machines

Logistic regression finds a separating hyperplane; a support vector machine finds the best separating hyperplane. The word "best" has a precise meaning: the plane that sits as far as possible from the nearest training points of either class. That single idea — maximum margin — makes SVMs powerful on small, high-dimensional datasets.

Introduction to the Support Vector Machine

Imagine two groups of points that a straight boundary can separate. An infinite number of lines split them correctly. Most of those lines are fragile: a tiny nudge in a training point could push a future test point to the wrong side. The SVM principle is to choose the line with the maximum possible "clearance" — the widest empty corridor, or margin, between the two classes. A model that separates with the widest margin generalizes best, because it has the most room for error.

Soft Margin and Hard Margin

Real data is messy: classes overlap, points get mislabeled, outliers stray. A hard margin SVM insists that every training point be correctly classified and on the correct side of the margin. If the data is not perfectly separable, a hard margin has no valid solution at all.

A soft margin SVM relaxes that demand. It allows a few points to violate the margin — or even sit on the wrong side — and pays a penalty for each violation. A single knob, the cost parameter C, balances the trade-off:

• Large C: heavy penalty for violations, narrow margin, hard to misclassify training data — low bias, high variance.
• Small C: violations are tolerated, wide margin, smoother decision surface — high bias, lower variance.

In practice the soft margin version is the one you use, because real datasets are rarely perfectly separable.

SVM Math Intuition: Maximizing Margin, Support Vectors

Recall the hyperplane w·x + b = 0 and the point-to-plane distance |w·x0 + b| / ||w|| from Chapter 18. An SVM scales w and b so that the points closest to the plane satisfy |w·x + b| = 1. The width of the margin is then:

margin = 2 / ||w||

Maximizing the margin means minimizing ||w|| — the same geometry you met with Ridge, but for a different reason. The training points that sit exactly on the margin boundary are the support vectors. They alone define the position and orientation of the boundary; move one and the plane moves. All the other points could be deleted without changing the model at all. This is why SVMs remain effective with small datasets: only the support vectors matter.

Combined with the class labels, the constraint becomes: for every point, y_i * (w·x_i + b) >= 1, where y_i is +1 or −1. Points on the correct side far from the boundary easily satisfy it; support vectors satisfy it with equality.

The SVC Cost Function

Adding soft-margin slack variables xi_i turns the constraints into y_i * (w·x_i + b) >= 1 - xi_i with xi_i >= 0. The optimization problem is:

minimize (1/2) * ||w||^2 + C * sum(xi_i)

shorthand: J(w, b) = (1/2) * ||w||^2 + C * sum(xi_i). The first term widens the margin, the second term punishes margin violations, and C is the exchange rate between the two. This is the "hinge-style" objective at the heart of an SVC — technically the squared hinge loss version used by sklearn differs in detail, but this is the core idea.

Support Vector Regression

The same machinery reverses for regression. Instead of predicting a category, an SVR finds a tube of width epsilon around the fitted function and only charges error for points that fall outside the tube. Points inside the tube are "good enough," so the model ignores small residuals and focuses on the significant deviations. The result is a robust regression that is insensitive to small noise: only the points outside the tube act as support vectors that pull the function.

SVM Kernels: Linear, Polynomial, RBF

Not every problem is linearly separable. The kernel trick fixes that without adding features to your data: a kernel is a function that measures similarity between two points as if they had been mapped into a higher-dimensional space. The decision boundary in the original space can then curve.

• Linear kernel: K(a, b) = a·b. Just the plain dot product — a straight boundary. Fast, interpretable, great for text with many features.
• Polynomial kernel: K(a, b) = (gamma * a·b + coef0)^degree. Produces polynomial decision boundaries; the degree controls the curvature.
• RBF kernel: K(a, b) = exp(-gamma * ||a - b||^2). The Gaussian radial basis function measures closeness by distance. Its gamma parameter sets the radius of influence: small gamma — smooth, global boundaries; large gamma — tight, wiggly boundaries that hug the points.

The elegance of kernels is that we never compute the high-dimensional mapping — the kernel evaluates directly in input space, which is cheap.

Support Vector Classifiers

Scikit-learn's SVC handles binary and multiclass classification. With three or more classes it uses a one-vs-one tournament internally: a model per class pair, then a majority vote. The probability=True flag makes it estimate class probabilities (via Platt scaling) so you can use predict_proba.

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("svc", SVC(kernel="rbf", C=1.0, gamma="scale")),
])
pipe.fit(X_train, y_train)
print(pipe.score(X_test, y_test))

SVM Kernel Implementation

Choosing a kernel is an empirical question, so compare them under cross-validation. Keep the linear kernel as your baseline; upgrade to RBF when the boundary is clearly curved. Tune gamma and C together, since they interact.

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

param_grid = {
    "kernel": ["rbf", "poly", "linear"],
    "C": [0.1, 1.0, 10.0],
    "gamma": ["scale", 0.01, 0.1],
}
gs = GridSearchCV(SVC(), param_grid, cv=5, scoring="f1")
gs.fit(X_train_scaled, y_train)
print(gs.best_params_)

The winning kernel depends on the geometry of your data — some problems genuinely need a straight wall, others a wavy boundary.

Support Vector Regression Implementation

SVR is the regression sibling. Its knobs are kernel, C, and epsilon (the tube half-width). A larger epsilon discards more small residuals and yields a flatter, smoother fit:

from sklearn.svm import SVR
from sklearn.metrics import mean_squared_error

model = SVR(kernel="rbf", C=100.0, epsilon=0.1)
model.fit(X_train, y_train)
pred = model.predict(X_test)
print(mean_squared_error(y_test, pred))

Grid-search C and epsilon the same way you would for an SVC. SVR shares all of the SVM traits: robustness to noise, dependence on a handful of support vectors, and sensitivity to feature scale.