Convolutional Neural Networks

Fully connected networks work fine on small, structured inputs, but they struggle with images: a 256 by 256 color photo has almost 200,000 numbers per example, and treating every pixel as an independent feature throws away the spatial structure. Convolutional neural networks (CNNs) are built to exploit that structure. This chapter explains how CNNs see images and why they became the backbone of computer vision.

CNN Introduction

A CNN processes images in a stack of stages: it detects edges, then shapes, then parts of objects, and finally whole objects. It does this with three kinds of layers — convolution layers that find features, pooling layers that shrink the image while keeping the important content, and fully connected layers that make the final decision. The network learns which features matter without anyone hand-designing them.

The Human Brain vs CNN Analogy

Neuroscientists found that cells in the early visual cortex respond to simple patterns such as oriented edges, while higher areas respond to more complex things like faces. A CNN mirrors this hierarchy: early layers learn tiny edge-like filters, and later layers combine them into recognizable concepts. The resemblance is loose but gives an excellent mental model of why the architecture works.

Everything You Need to Know About Images

For a computer, an image is a grid of numbers. A grayscale image is a single grid where each value is the brightness of a pixel, usually 0 (black) to 255 (white). A color image uses three channels — red, green, and blue — so it is really three grids stacked together, and a pixel's color is the combination of its three channel values.

• Pixel: one cell of the grid; its numeric value is intensity.
• Channel: one grid of the image; grayscale has 1 channel, RGB has 3.
• Shape: (height, width, channels); e.g. a 32×32 RGB image is (32, 32, 3).
• Normalization: dividing pixels by 255 rescales them into [0, 1], which stabilizes training.

The Convolution Operation in CNN

A convolution slides a small grid of weights called a kernel (for example 3×3) across the image. At each position it multiplies the overlapping pixels by the kernel values and sums the products to produce one output number. The result is a feature map that is large wherever the kernel's pattern matched.

# one step of a 3x3 convolution on a 3x3 patch
patch = [[0, 128, 90],
         [20, 60, 200],
         [255, 10, 5]]
kernel = [[1, 0, -1],
          [1, 0, -1],
          [1, 0, -1]]
out = sum(p * k for row_p, row_k in zip(patch, kernel)
              for p, k in zip(row_p, row_k))

That kernel is a vertical-edge detector: it subtracts the right side of the patch from the left side, so it responds strongly to vertical boundaries. During training, many kernels are learned automatically.

Padding in CNN

Every convolution shrinks the image by kernel_size - 1 rows and columns, because the kernel cannot reach past the edges. Padding adds a border of zeros around the image before convolving. Valid padding uses no border (image shrinks), while same padding adds enough zeros so the output has the same height and width as the input. Same padding lets us stack many layers without the image disappearing.

CNN vs ANN Operations

An ANN layer connects every neuron to every neuron of the previous layer, so a single fully connected layer over an image has millions of weights and ignores spatial order. A CNN layer instead uses local connectivity (each neuron sees only a small window), weight sharing (the same kernel slides across the whole image), and spatial structure. This cuts the parameter count dramatically and makes the model translation-tolerant — a cat stays a cat whether it is on the left or the right of the photo.

Max, Min and Average Pooling

Pooling downsamples the feature map by sliding a small window and replacing each window with a summary statistic:

• Max pooling: keeps the largest value. It highlights the strongest feature response and adds a small amount of translation invariance.
• Min pooling: keeps the smallest value; rarely used, but can emphasize suppressed regions in specialized pipelines.
• Average pooling: keeps the mean. It is smoother and used when the overall level of activation matters, for example just before the classifier.

from tensorflow import keras
model = keras.Sequential([
    keras.layers.Conv2D(32, (3, 3), activation="relu", padding="same"),
    keras.layers.MaxPooling2D((2, 2)),
])

Flattening and Fully Connected Layers

After several convolution and pooling stages, the output is still a multi-channel grid. To classify, we flatten it into a single long vector and feed it into one or two fully connected (dense) layers. The dense layers recombine the learned spatial features into a final prediction, typically ending in a softmax layer for multiclass classification.

A CNN Example with RGB Images

Imagine classifying a 32×32 RGB image into 10 classes:

1. Input shape (32, 32, 3) — three color channels.
2. Convolution layer with 32 kernels of size 3×3, same padding, ReLU → feature maps of shape (32, 32, 32).
3. Max pooling 2×2 → (16, 16, 32).
4. Convolution layer with 64 kernels → (16, 16, 64).
5. Max pooling 2×2 → (8, 8, 64).
6. Flatten → 8 * 8 * 64 = 4096 values.
7. Dense layer with ReLU, then a dense softmax layer with 10 outputs.

model = keras.Sequential([
    keras.layers.Input(shape=(32, 32, 3)),
    keras.layers.Conv2D(32, (3, 3), activation="relu", padding="same"),
    keras.layers.MaxPooling2D((2, 2)),
    keras.layers.Conv2D(64, (3, 3), activation="relu", padding="same"),
    keras.layers.MaxPooling2D((2, 2)),
    keras.layers.Flatten(),
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dense(10, activation="softmax"),
])
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])

This architecture is small enough to train on a laptop and accurate enough on the CIFAR-10 dataset to demonstrate everything a CNN does.