Miscellaneous

This chapter covers essential ML concepts, performance metrics, ensemble methods, and advanced techniques that complement the core algorithms. Mastering these will make you a well-rounded ML practitioner.

Performance Metrics

Classification Metrics

Accuracy = (TP + TN) / (TP + TN + FP + FN) — overall correctness. Precision = TP / (TP + FP) — how many positive predictions were correct. Recall = TP / (TP + FN) — how many actual positives were found. F1-score = 2 × (P × R) / (P + R) — harmonic mean of precision and recall.

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

y_true = [0, 1, 1, 0, 1, 0, 1, 0, 1, 1]
y_pred = [0, 1, 0, 0, 1, 0, 1, 1, 1, 0]

print(f"Accuracy:  {accuracy_score(y_true, y_pred):.3f}")
print(f"Precision: {precision_score(y_true, y_pred):.3f}")
print(f"Recall:    {recall_score(y_true, y_pred):.3f}")
print(f"F1-score:  {f1_score(y_true, y_pred):.3f}")

Regression Metrics

RMSE (Root Mean Squared Error) is the square root of MSE, giving errors in the same units as the target. MAE (Mean Absolute Error) averages the absolute differences. RMSE penalizes large errors more than MAE.

from sklearn.metrics import mean_absolute_error, mean_squared_error
import numpy as np

y_true = [2.5, 5.0, 7.2, 8.0, 9.5]
y_pred = [2.3, 5.2, 6.8, 8.3, 9.1]

mae = mean_absolute_error(y_true, y_pred)
rmse = np.sqrt(mean_squared_error(y_true, y_pred))

print(f"MAE:  {mae:.3f}")
print(f"RMSE: {rmse:.3f}")

Ensemble Methods: Boosting

Boosting combines weak learners sequentially, where each new model focuses on correcting the errors of its predecessor. AdaBoost weights misclassified points more heavily. XGBoost uses gradient boosting with regularization for state-of-the-art performance. Gradient Boosting generalizes boosting to arbitrary differentiable loss functions.

from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=200, random_state=42)

ada = AdaBoostClassifier(n_estimators=50, random_state=42)
gbt = GradientBoostingClassifier(n_estimators=50, random_state=42)

ada.fit(X, y)
gbt.fit(X, y)

print(f"AdaBoost accuracy:         {ada.score(X, y):.3f}")
print(f"Gradient Boosting accuracy: {gbt.score(X, y):.3f}")

Bagging

Bagging (Bootstrap Aggregating) trains multiple models on bootstrapped subsets of the data and averages their predictions. This reduces variance without increasing bias. Random Forest is a bagging ensemble of decision trees.

Cross Validation

K-fold cross validation splits data into k folds, trains on k-1 folds, and validates on the remaining fold. This process repeats k times, providing a reliable estimate of model performance with lower variance than a single train-test split.

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

model = RandomForestClassifier(n_estimators=50, random_state=42)
scores = cross_val_score(model, X, y, cv=5)

print(f"CV scores: {scores}")
print(f"Mean: {scores.mean():.3f} (+/- {scores.std() * 2:.3f})")

ROC Curves and AUC

The Receiver Operating Characteristic (ROC) curve plots the True Positive Rate against the False Positive Rate at various threshold settings. The Area Under the Curve (AUC) summarizes the model's ability to distinguish between classes — an AUC of 1.0 is perfect, 0.5 is random guessing.

Grid Search and Random Search

Hyperparameter tuning finds the optimal configuration for a model. Grid Search exhaustively evaluates all combinations in a parameter grid. Random Search samples random combinations, often finding good parameters faster for high-dimensional spaces.

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

param_grid = {
    'C': [0.1, 1, 10],
    'kernel': ['rbf', 'poly'],
    'gamma': ['scale', 'auto']
}

grid = GridSearchCV(SVC(), param_grid, cv=5, scoring='accuracy')
grid.fit(X, y)

print(f"Best params: {grid.best_params_}")
print(f"Best score:  {grid.best_score_:.3f}")

Additional Concepts

Scaling standardizes features to similar ranges, essential for distance-based algorithms. Association Rules (Apriori) discover frequent itemsets in transaction data. Gaussian Discriminant Analysis models class-conditional distributions. Cost functions measure prediction error. Bayes Theorem provides a framework for probabilistic inference. Adversarial ML studies vulnerabilities in models. Stacking combines diverse models via a meta-learner. Epochs are complete passes through the training data. Perceptron is the simplest neural network unit. Regularization (L1, L2) penalizes large weights to prevent overfitting. Entropy measures impurity in decision trees.