Machine Learning Foundations

Welcome to the machine learning half of this book. Everything you learned so far — statistics, probability, data cleaning, and visual exploration — exists to feed a learning algorithm. In this chapter we lay the conceptual groundwork: what machine learning is, the main families of learning techniques, and the geometry that almost every classic model relies on.

Introduction to Machine Learning

Machine learning is a branch of artificial intelligence in which a computer improves its performance on a task by learning from data, without being explicitly programmed with rules for every possible situation. Instead of telling the computer how to recognize a spam email, we show it thousands of examples of spam and legitimate mail and let it discover the patterns itself.

Formally, Tom Mitchell's classic framing describes it well: a program is said to learn from experience E with respect to some task T and performance measure P if its performance on T, as measured by P, improves with experience E. The experience is almost always a dataset of examples.

The core learning loop is simple to state and hard to perfect:

1. Collect a dataset of examples.
2. Choose a model family (a line, a tree, a neural network).
3. Define a way to measure how wrong the model is.
4. Adjust the model's internal parameters to reduce that wrongness.
5. Evaluate on data the model has never seen.

Types of Machine Learning Techniques

Learning techniques are usually grouped by how much supervision the model receives — that is, how much extra information accompanies the raw input data.

Supervised Learning

Every training example comes as a pair: an input x and a known answer y (called a label or target). The model learns a mapping from inputs to outputs. If y is a continuous number, the task is called regression (predict a house price). If y belongs to a small set of categories, the task is classification (predict spam or ham). Linear regression, logistic regression, decision trees, and support vector machines all belong here.

Unsupervised Learning

The data comes with no labels at all. The model must find structure on its own: grouping similar points into clusters (clustering), compressing the data into fewer dimensions (dimensionality reduction), or flagging points that look strange (anomaly detection). K-Means and PCA are the canonical examples.

Semi-Supervised Learning

A realistic middle ground: a large pool of unlabeled data with a small amount of labeled data. The model uses the labeled points to seed understanding, then propagates labels to nearby unlabeled points. This is common when labeling is expensive, such as in medical imaging, where a radiologist can annotate a few scans but millions exist.

Reinforcement Learning

There is no dataset of correct answers at all. An agent interacts with an environment, takes actions, and receives rewards or penalties. It learns a policy — a strategy for choosing actions — that maximizes cumulative reward. Training a robot to walk or teaching an AI to play chess uses this framework. The "supervision" is delayed: a bad move early in the game only hurts many steps later.

The Geometry of Models: Lines, Planes, and Hyperplanes

Many classic models represent their learned knowledge as a straight boundary or a flat surface. Understanding this geometry will make every later chapter (linear regression, logistic regression, SVMs) feel natural.

The Equation of a Line

In two dimensions, a line is written as y = m*x + b, where m is the slope and b is the y-intercept. Another common form is w1*x + w2*y + b = 0. Both describe the same object: the set of points where the expression on the left equals zero. The second form will generalize beautifully to higher dimensions.

The Equation of a Plane in 3D

In three dimensions, the set of points satisfying w1*x + w2*y + w3*z + b = 0 is a flat two-dimensional surface — a plane. Think of a sheet of paper tilted in mid-air. The coefficients w1, w2, w3 control the tilt of the sheet, and b shifts the sheet closer to or farther from the origin.

The Hyperplane

The same pattern continues in any number of dimensions. If each data point has d features, a hyperplane is the set of points satisfying w·x + b = 0, where w is a vector of weights and x is the feature vector. In 2D a hyperplane is a line; in 3D it is a plane; in d dimensions it is a flat object of dimension d - 1 that slices the feature space into two halves. A model that predicts class +1 on one side and class −1 on the other is, geometrically, just placing a hyperplane.

import numpy as np

# A hyperplane in 3D: w1*x + w2*y + w3*z + b = 0
w = np.array([2.0, -1.0, 0.5])
b = 3.0

def side(x):
    return np.dot(w, x) + b

point_a = np.array([1.0, 1.0, 1.0])
point_b = np.array([-2.0, 5.0, 0.0])
print(side(point_a))  # positive: one side
print(side(point_b))  # negative: other side

Distance of a Point from a Plane or Hyperplane

It is often useful to know how far a point sits from a separating hyperplane — far points are classified with high confidence, points on the boundary are borderline. For a line a*x + b*y + c = 0, the perpendicular distance of a point (x0, y0) is:

distance = |a*x0 + b*y0 + c| / sqrt(a*a + b*b)

The generalization to a hyperplane w·x + b = 0 is elegant:

distance = |w·x0 + b| / ||w||

The numerator measures how "deep" the point is on its side of the hyperplane, and the denominator ||w|| (the length of the weight vector) scales that raw score into a real distance. This exact formula drives the margin concept you will meet with support vector machines, and it explains why SVMs care about the length of w.

import numpy as np

def dist_to_hyperplane(x0, w, b):
    return abs(np.dot(w, x0) + b) / np.linalg.norm(w)

w = np.array([1.0, 1.0])
b = 0.0  # the line x + y = 0
print(dist_to_hyperplane(np.array([3.0, 4.0]), w, b))  # 4.95

Instance-Based vs Model-Based Learning

Learning systems also differ in where the learned knowledge is stored.

Instance-Based Learning

The model keeps the training examples around and answers new queries by comparing against them. A new point is classified by looking at the majority label among its k nearest stored neighbors. Training is nearly instant (just store the data), but prediction requires scanning or indexing the stored points, and the model's "knowledge" is the dataset itself. K-Nearest Neighbors is the textbook example.

Model-Based Learning

The algorithm distills the training data into a compact set of parameters — a slope and intercept, a vector of weights, a decision tree structure. Training is the expensive step; prediction afterwards is a cheap formula evaluation, and the raw data can be discarded. Linear regression is the clearest example: fifty thousand house records collapse into a handful of coefficients.

Choosing Between Them

• Instance-based shines with small datasets and no requirement for a compact, deployable artifact.
• Model-based wins when you need fast predictions, interpretable parameters, or a model small enough to ship.
• Hybrids exist: kernel-based methods behave instance-like but learn weights on top of the stored points.