7. Transformers in Generative AI

Transformers in Generative AI

The Transformer Revolution

The Transformer architecture (Vaswani et al., 2017) replaced recurrent neural networks by relying entirely on attention mechanisms. It enables parallel processing of sequences and scales to billions of parameters, forming the backbone of all major LLMs.

Figure: Key components of the Transformer encoder block.

Input Embeddings

Each input token is mapped to a high-dimensional vector (e.g., 768d for BERT-base, 12288d for GPT-4). Embeddings are learned during training and capture semantic relationships between tokens.

Positional Encoding

Since attention has no inherent notion of position, positional encodings are added to input embeddings. Original transformers used sinusoidal functions; learned positional embeddings are also common.

Multi-Head Attention

The core innovation. Queries (Q), Keys (K), and Values (V) are linear projections of the input. Attention weights are computed as: Attention(Q,K,V) = softmax(QKᵀ/√d)V. Multiple heads learn different relationship patterns.

# Scaled dot-product attention (single head)
def attention(Q, K, V):
    d_k = Q.shape[-1]
    scores = Q @ K.T / sqrt(d_k)
    weights = softmax(scores, dim=-1)
    return weights @ V

# Multi-head: run attention h times in parallel, concat results
def multi_head(Q, K, V, h=8):
    heads = [attention(Q_i, K_i, V_i) for i in range(h)]
    return concat(heads)

Key Transformer Concepts

Residual Connections — Skip connections that add input to output of each sublayer, enabling deep networks
Layer Normalization — Normalize across feature dimension, stabilizes training
Feed Forward Network — Two linear layers with ReLU/GELU activation applied per position
Causal Masking — In decoder, mask future tokens so position t can only attend to positions ≤ t

Decoder-Only Architecture (GPT)

GPT uses only the decoder stack with causal masking. It's autoregressive — each token is predicted from all previous tokens. The decoder consists of masked multi-head attention, then cross-attention (to encoder, if present), then FFN.

Practice Task: Implement a single layer of multi-head attention from scratch in Python with NumPy/PyTorch. Test it on a simple sequence (e.g., 5 tokens with random embeddings). Visualize the attention weights for one head as a heatmap. Explain what patterns the attention head might be capturing.