Logistic Regression

Despite its name, logistic regression is a classifier. It answers the question Chapter 20 left open: how do we output a probability for a category? The answer is to wrap linear regression in a squashing function so predictions stay between 0 and 1. This chapter builds the intuition, the metrics that judge classifiers, and the machinery to tune and deploy them.

Logistic Regression In-Depth Math Intuition

The Sigmoid Function

Start with the linear score z = w·x + b. It can range from negative to positive infinity, which is useless as a probability. The sigmoid (or logistic) function folds that entire line into the interval (0, 1):

sigmoid(z) = 1 / (1 + exp(-z))

• When z is large and positive, exp(-z) approaches 0 and the output approaches 1.
• When z is large and negative, the denominator blows up and the output approaches 0.
• When z = 0, the output is exactly 0.5 — the point of maximum uncertainty.

The output p = sigmoid(z) is interpreted as the probability that the example belongs to the positive class.

Log-Odds

The name "logistic" comes from the log-odds. Rearranging the sigmoid:

z = log( p / (1 - p) )

The quantity p / (1 - p) is the odds — if p = 0.75, the odds are 3 to 1. The model is literally learning a linear function of the features that predicts the natural log of the odds. That is why the coefficients read like regression coefficients: increasing a feature by one unit multiplies the odds by exp(wj). An odds ratio above 1 pushes the prediction toward class 1; below 1 pushes it toward class 0.

The Decision Boundary

To make a hard prediction, pick a threshold, normally 0.5: predict class 1 when p >= 0.5, class 0 otherwise. Because p = 0.5 corresponds to z = 0, the decision boundary in feature space is the hyperplane w·x + b = 0 — exactly the geometry from Chapter 18. Points far on one side get near-certain probabilities; points near the boundary get near-0.5.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

w, b = np.array([1.0, 2.0]), -3.0
p = sigmoid(np.dot([2.0, 1.0], w) + b)
print(p)  # probability of positive class

Performance Metrics for Classification

Accuracy alone can lie. When 95% of emails are ham, a model that always predicts ham is 95% "accurate" yet completely useless. That is why we count the four cells of a confusion matrix.

The Confusion Matrix

• True Positive (TP): predicted positive, actually positive.
• True Negative (TN): predicted negative, actually negative.
• False Positive (FP): predicted positive, actually negative (a false alarm).
• False Negative (FN): predicted negative, actually positive (a miss).

Everything else derives from these four counts.

Accuracy, Precision, Recall, F1

accuracy = (TP + TN) / total — fraction correct overall. Good for balanced problems.

precision = TP / (TP + FP) — of everything we flagged positive, how much was right? High precision means few false alarms.

recall = TP / (TP + FN) — of all true positives, how many did we catch? High recall means few misses.

F1 = 2 * precision * recall / (precision + recall) — the harmonic mean, a single number balancing the two.

Choose your metric by the cost of mistakes: fraud detection favors high recall (catch the fraud), legal spam filters favor high precision (never block a real email), and F1 gives a balanced score when both matter.

from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score

print(confusion_matrix(y_true, y_pred))
print(precision_score(y_true, y_pred), recall_score(y_true, y_pred), f1_score(y_true, y_pred))

Logistic Regression One-vs-Rest (OVR)

Logistic regression is natively binary. For C classes, the one-vs-rest strategy trains C binary models, each distinguishing "class k" against "everything else," then picks the class whose model is most confident. Scikit-learn wraps this automatically via multi_class="ovr" (the newer default for softmax-style is multinomial, but OVR remains the classic, well-understood option). OVR is simple, parallelizable, and a solid default for multiclass problems.

Logistic Regression Implementation

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42)

clf = LogisticRegression(max_iter=2000)
clf.fit(X_train, y_train)
print(clf.score(X_test, y_test))

Note max_iter: logistic regression is trained by gradient descent, so you must let it converge. If you see a convergence warning, raise the iteration cap or scale the features.

GridSearchCV Hyperparameter Search

The most important tuning dials are the regularization strength C (inverse of alpha — smaller C means stronger regularization) and the penalty type. GridSearchCV tries every combination in a grid and reports the best cross-validated score:

from sklearn.model_selection import GridSearchCV

param_grid = {"C": [0.01, 0.1, 1.0, 10.0]}
gs = GridSearchCV(LogisticRegression(max_iter=2000),
                  param_grid, cv=5, scoring="f1")
gs.fit(X_train, y_train)
print(gs.best_params_, gs.best_score_)

Grid search is thorough but explodes combinatorially as you add dimensions to the grid.

RandomizedSearchCV

When the search space grows, randomized search is more efficient. It samples a fixed number of parameter combinations from a distribution rather than exhausting a grid, giving most of the benefit of grid search at a fraction of the cost:

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform

param_dist = {"C": loguniform(0.001, 100)}
rs = RandomizedSearchCV(LogisticRegression(max_iter=2000),
                        param_dist, n_iter=50, cv=5, random_state=42)
rs.fit(X_train, y_train)
print(rs.best_params_)

Use randomized search when you have many hyperparameters or a large dataset; use grid search when the grid is small enough to enumerate.

Logistic Regression on Imbalanced Datasets

When one class dominates (99% legit transactions vs 1% fraud), a model can achieve 99% accuracy by predicting the majority class always. Countermeasures, in order of preference:

• Evaluate with precision/recall/F1 or ROC-AUC instead of accuracy.
• Tell the algorithm the classes are not equal: pass class_weight="balanced" to up-weight the minority class.
• Adjust the decision threshold downward (say, predict fraud when p >= 0.3) to catch more minority cases.
• Resample: oversample the minority with SMOTE or undersample the majority, then retrain.

ROC Curves and AUC

A Receiver Operating Characteristic (ROC) curve shows the trade-off between the true positive rate and the false positive rate as the decision threshold slides from 0 to 1. A perfect classifier produces an ROC curve that hugs the top-left corner; a random guess produces a diagonal line. The Area Under the Curve (AUC) summarizes the whole curve as a single number:

• AUC = 1.0 — perfect separation.
• AUC = 0.5 — no better than a coin flip.
• AUC = 0.8+ — strong discrimination for most real problems.

from sklearn.metrics import roc_curve, roc_auc_score

prob = clf.predict_proba(X_test)[:, 1]
fpr, tpr, _ = roc_curve(y_test, prob)
print("AUC:", roc_auc_score(y_test, prob))

AUC is threshold-independent, making it ideal for comparing models before you commit to a production threshold.