Everything you have learned so far — regression, trees, ensembles — belongs to classical machine learning. In this chapter we cross into deep learning, where models contain thousands or millions of learnable parameters organized into layers. You will meet the perceptron, the smallest building block, then grow that idea into a full network and study every ingredient a trainer needs: activation functions, loss functions, optimizers, weight initialization, and regularization.
Deep learning is machine learning with neural networks that have many hidden layers. The word "deep" refers to the depth of the network, not to deep understanding. Each layer transforms its input a little, and by stacking transformations the network can represent very complex functions: recognizing a face from raw pixels, translating a sentence, or predicting the next word as you type.
A perceptron receives inputs x1, x2, ..., xn, multiplies each by a weight, sums them, adds a bias, and passes the result through a threshold. It is a single binary decision boundary.
import numpy as np
def perceptron(x, w, b):
z = np.dot(x, w) + b
return 1 if z >= 0 else 0Think of it as a vote: each input casts a weighted vote, and the bias sets the generosity of the deciding threshold.
An artificial neural network (ANN) stacks layers of neurons. The input layer receives features, hidden layers recombine them, and the output layer produces the answer. Two consecutive layers are usually fully connected: every neuron in one layer connects to every neuron in the next. Training is a loop of three steps: forward pass (compute predictions), loss (measure the error), and backward pass (adjust weights to reduce the error).
Backpropagation computes how much each weight contributed to the error and how it should change. After the loss is computed at the output, the algorithm walks backwards through the network, layer by layer, applying the chain rule. The update rule for a weight is:
weight = weight - learning_rate * d(loss)/d(weight)The gradient d(loss)/d(weight) tells us the direction of steepest increase in the loss; subtracting it moves the weight toward lower loss.
Backpropagation rests entirely on the chain rule. If a = f(u) and u = g(x), then da/dx = da/du * du/dx. A network composes many functions, so the gradient of the loss with respect to an early weight is a long product of small local derivatives. That long product is exactly where training can go wrong.
The sigmoid squashes its input into (0, 1) but its derivative is tiny away from zero — at most 0.25 in the middle and near zero at the tails. In a deep network the chain rule multiplies many such small numbers, so gradients shrink exponentially and early layers stop learning entirely. This is the vanishing gradient problem, and it is the main reason sigmoids disappeared from hidden layers.
sigmoid(z) = 1 / (1 + exp(-z)). Smooth and bounded, good for output probabilities in binary classification, but suffers from vanishing gradients and outputs that are not zero-centered.
tanh(z) = (exp(z) - exp(-z)) / (exp(z) + exp(-z)). Zero-centered version of sigmoid, output in (-1, 1), usually better than sigmoid for hidden layers, but still suffers vanishing gradients.
relu(z) = max(0, z). Cheap to compute, gradient is 1 for positive inputs and 0 for negatives, which fights vanishing gradients and gives sparse activation. Downside: dead neurons when many weights push inputs negative.
Leaky ReLU lets a small negative slope a * z (typically a = 0.01) through, so neurons can never fully die. Parametric ReLU (PReLU) makes a a learned parameter instead of a fixed constant.
ELU is smooth at zero and saturates to a * (exp(z) - 1) for negative inputs. The smooth curve can improve gradient flow and robustness compared with ReLU, at a small compute cost.
For multiclass classification the output layer uses softmax, which converts scores into positive numbers that sum to 1, so they can be read as class probabilities.
The two terms are often used interchangeably, but a useful distinction is: a loss function measures the error of one training example, while a cost function averages the losses over the whole batch or dataset. Gradient descent minimizes the cost, which is built from many individual losses.
Plain stochastic gradient descent updates weights using the gradient from a single example. It is noisy but can escape shallow local minima; the noise slows convergence near the end.
Split the dataset into small batches (for example 32 or 64 samples) and update after each batch. This balances the speed of full-batch gradient descent with the noise of single-sample SGD, and it is the standard way of training.
Momentum keeps a running average of past gradients, so the update behaves like a ball rolling downhill. It damps oscillations and accelerates movement through consistent slopes.
Adagrad scales the learning rate per weight by the accumulated squared gradients, giving rare parameters bigger steps. Its learning rate decays over time and can become too small.
RMSProp replaces the ever-growing accumulation with an exponential moving average, so the per-parameter learning rate keeps adapting without shrinking to zero.
Adam combines momentum (a running mean of gradients) with RMSProp (a running mean of squared gradients), plus bias corrections for the first steps. It is fast, robust, and the default choice for most deep learning work.
from tensorflow import keras
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])The mirror image of vanishing gradients: if the products in the chain rule grow, weights explode into huge values and training diverges. It is handled with gradient clipping, smaller learning rates, careful weight initialization, and architectures like LSTM that keep gradients bounded.
Dropout randomly switches off a fraction of neurons during each training pass. The surviving neurons must learn more robust features, and the effect is like training many thinned networks and averaging them. During prediction dropout is disabled so all neurons vote.
model.add(keras.layers.Dense(64, activation="relu"))
model.add(keras.layers.Dropout(0.3))