Attention & Transformers

LSTM and GRU made sequence models practical, but they still read one token at a time and compress the entire source into one final state. The transformer threw that away: it processes all tokens in parallel and lets every token directly look at every other token through a mechanism called attention. Transformers now power translation, search, and the large language models behind modern assistants. This chapter builds your intuition for them from the ground up.

In-Depth Intuition of Encoder-Decoder Sequence-to-Sequence Architectures

A sequence-to-sequence model (seq2seq) has two networks. The encoder reads the entire source sentence and compresses it into a final hidden state. The decoder then generates the target sentence word by word, feeding each produced word back as the input for the next step. This is how classic machine translation worked: read the French sentence fully, then write the English one.

Problems with Encoder-Decoder

The bottleneck is the final state. Every ounce of meaning from a long source sentence must fit into one vector, and the decoder has no other way to consult the source. Information from the beginning of a long sentence gets diluted by the time the decoder starts writing. Attention was invented to fix exactly this: instead of one compressed summary, let the decoder look back at every encoder position, every step.

In-Depth Architecture Explanation of the Attention Mechanism

Attention computes a weighted average over encoder outputs. The decoder has a current state called the query; each encoder position contributes a key (its "address"); and each carries a value (its content). The model scores how well the query matches each key, turns those scores into probabilities with softmax, and uses them to weigh the values. The result is a context vector rich in the parts of the source that matter for this output step.

scores = query . keys            # dot-product similarity
weights = softmax(scores)        # probabilities summing to 1
context = sum(weights * values)  # weighted blend of source

A Plan of Action for Learning Transformers

Transformers are intimidating only because they stack many ideas. Learn them in this order: (1) self-attention, because it is the engine; (2) multi-head attention, which runs several attention passes at once; (3) positional encoding, which gives order to a model that otherwise sees a bag of words; (4) the feed-forward layer and residual structure; (5) layer normalization; (6) how encoder and decoder assemble these blocks; (7) the mask in the decoder that prevents peeking at future words.

What and Why to Use Transformers

Transformers are used because they fix the weaknesses of RNNs. They process the whole sequence in parallel, so training on GPUs is dramatically faster; they give every token a direct connection to every other token, so long-range dependencies are not crushed by distance; and the same architecture scales to enormous sizes, which is why pretrained transformers dominate NLP. Their cost is quadratic in sequence length, because every pair of tokens must be compared.

Understanding the Basic Architecture of Transformers

A transformer is built from stacked blocks. The encoder block contains a multi-head self-attention sublayer followed by a feed-forward network, each wrapped with residual connections and layer normalization. The decoder block contains those same two sublayers plus an extra cross-attention sublayer that lets the decoder consult the encoder output. Both sides convert tokens into vectors through an embedding plus positional encoding.

How the Self-Attention Layer Works

In self-attention, every token in the sequence plays all three roles. For each token we compute a query, a key, and a value by multiplying the token's vector by three learned matrices W_q, W_k, W_v. Then each token scores its query against every key, normalizes with softmax, and gathers a weighted sum of all values. A token's output therefore depends on the whole sentence — the model literally reads the full context before answering.

Q = X @ W_q;  K = X @ W_k;  V = X @ W_v
attention = softmax(Q @ K.T / sqrt(d_k)) @ V

Dividing by sqrt(d_k) keeps the dot products from growing huge and pushing softmax into very sharp distributions.

Multi-Head Attention

One attention pass finds one kind of relationship: "this word depends on that word". A sentence contains many relationships at once — grammar, coreference, meaning. Multi-head attention runs several attention computations in parallel with different learned projections, then concatenates their outputs and projects them back together. Each head can specialize in a different pattern, giving the model much richer context.

The Feed-Forward Neural Network with Multi-Head Attention

After attention mixes information across positions, the feed-forward sublayer processes each position independently: the same two dense layers with a ReLU in between, applied to every token vector. The pattern repeats block after block: attention shares information across the sequence, then the feed-forward layer thinks locally about each token.

In-Depth Intuition of Positional Encoding

Attention is a weighted average, and a weighted average does not care about order — "dog bites man" and "man bites dog" would look identical to a bag of tokens. Positional encoding injects order by adding a vector that encodes each token's position to its embedding. The classic scheme uses sine and cosine functions of different frequencies, so every position gets a unique pattern and nearby positions get similar patterns. Modern variants learn the position vectors or use relative distances instead.

Layer Normalization

Layer normalization rescales the activations of one layer across all its features so they have zero mean and unit variance, then applies learned scale and shift parameters. It stabilizes training and lets networks use larger learning rates. In a transformer it is applied inside each sublayer, after the residual addition.

def layer_norm(x):
    mean = x.mean(axis=-1, keepdims=True)
    var = x.var(axis=-1, keepdims=True)
    xhat = (x - mean) / np.sqrt(var + 1e-6)
    return gamma * xhat + beta

Layer Normalization Examples

Suppose a hidden state is [1.0, -1.0, 3.0]. The mean is 1.0 and the variance is 2.67, so the normalized values are roughly [0.0, -1.22, 1.22] before the learned scale and shift are applied. Because normalization happens per layer per sample, batch size does not affect it — which matters for transformers, where sequences have different lengths.

The Complete Encoder Transformer Architecture

1. Tokens become embeddings; positional encodings are added.
2. Multi-head self-attention mixes information across all positions.
3. A residual connection adds the input back, then layer normalization.
4. A feed-forward network processes each position, again followed by residual plus layer normalization.
5. Steps 2–4 repeat for N stacked encoder blocks.

Decoder Transformer Plan of Action

The decoder is like the encoder with two differences: self-attention is masked so a position only sees earlier positions (it must generate left to right), and an extra cross-attention sublayer lets the decoder query the encoder's output. Build it in this order: masked self-attention, then encoder–decoder attention, then feed-forward, each with residuals and normalization.

Masked Multi-Head Attention in the Decoder

When generating word 5, the model must not see words 6, 7, and 8. The decoder enforces this by masking: the attention scores for future positions are set to an enormous negative number before softmax, so their weights collapse to zero. The model learns to predict the next token using only what came before — exactly the condition it faces at inference time.

Encoder-Decoder Multi-Head Attention

This is where the decoder connects to the encoder. The decoder's hidden states provide the queries; the encoder's outputs provide the keys and values. So each decoder position looks at the entire source sentence and gathers the parts it needs for the next word — the original attention idea, now inside every decoder block.

The Final Decoder Linear and Softmax Layer

After the last decoder block, a final linear layer projects each position's vector to the vocabulary size, and softmax turns those scores into a probability distribution over every possible next word. The model samples from that distribution to choose the next token, appends it, and repeats until it generates the end-of-sentence marker.