LSTM & GRU Networks

The plain RNN remembers recent words well but forgets distant ones because its gradients vanish across time. Long short-term memory (LSTM) networks solve this by giving the network an explicit, controllable memory, and gated recurrent units (GRU) do the same with fewer moving parts. This chapter dissects both architectures, then builds a character-level text generator from scratch and wraps it in a web app.

Why LSTM RNNs

The vanilla RNN update h_t = activation(W @ [h_prev, x_t]) overwrites its memory at every step, so old information is inevitably washed out. LSTM instead keeps a separate cell state that information can flow through almost untouched — and three gates decide when to write, read, and clear it. Because the cell state is updated by addition rather than overwriting, gradients can travel backward through it without shrinking, which is why LSTM trains reliably on long sequences.

The LSTM Architecture

An LSTM cell maintains two vectors: the cell state C_t, its long-term memory, and the hidden state h_t, which is the visible output. Three gates — forget, input, and output — each use a sigmoid to produce a number between 0 and 1 that acts like a tap on the information flow.

The Forget Gate

The forget gate looks at the previous hidden state and the current input, and decides how much of the old cell state to keep:

f_t = sigmoid(W_f @ [h_prev, x_t] + b_f)
C_t = f_t * C_prev   # multiply old memory by the forget tap

A forget value of 1 preserves the memory; 0 wipes it. In a language model this is how the network drops a plural subject once it has seen the matching verb.

The Input Gate and Candidate Memory

Where the forget gate erases, the input gate writes. It works in two pieces: a sigmoid gate decides how strongly to accept new information, and a tanh layer produces the candidate values C_tilde. The two are multiplied and added to the cell state:

i_t = sigmoid(W_i @ [h_prev, x_t] + b_i)
C_tilde = tanh(W_c @ [h_prev, x_t] + b_c)
C_t = C_t + i_t * C_tilde

The Output Gate

The output gate decides what the cell reveals as the hidden state. The cell state is pushed through tanh to compress it into (-1, 1), then scaled by the gate:

o_t = sigmoid(W_o @ [h_prev, x_t] + b_o)
h_t = o_t * tanh(C_t)

So h_t is the "reported" memory, and the next step uses it as the input context.

The Training Process in LSTM

LSTMs train with backpropagation through time exactly like plain RNNs, but the additive cell-state updates keep the gradient signal strong across many steps. Practical training still uses the same machinery: an optimizer (Adam is standard), a loss function, mini-batches, and monitoring on a validation split to stop before overfitting.

Variants of LSTM

• Peephole LSTM: lets the gates look at the cell state directly, not just the hidden state.
• Bidirectional LSTM: two LSTM passes, one forward and one backward, whose outputs are merged — perfect for tasks where both contexts matter (like classification of a whole sentence).
• Stacked LSTM: multiple LSTM layers on top of each other, letting the network learn hierarchical abstractions.
• GRU: a simplified cousin that merges the forget and input gates; covered next.

The GRU RNN Complete In-Depth Intuition

The GRU removes the separate cell state and uses only the hidden state, controlled by two gates. The update gate decides how much of the past to carry forward, and the reset gate decides how much of the past to forget when building the new candidate. Fewer operations mean fewer parameters and faster training, at a small cost in expressiveness on very long sequences.

z_t = sigmoid(W_z @ [h_prev, x_t] + b_z)   # update gate
r_t = sigmoid(W_r @ [h_prev, x_t] + b_r)   # reset gate
h_tilde = tanh(W_h @ [r_t * h_prev, x_t] + b_h)
h_t = (1 - z_t) * h_prev + z_t * h_tilde

A Text Generation Problem Statement

Our project: generate new text in the style of a classic public-domain book. The model reads characters one at a time and predicts the next character. Because the training data is small, the network trains in minutes on a CPU — the ideal scale for learning the full pipeline by hand.

Data Collection and Data Processing

1. Load a plain-text book (for example from Project Gutenberg) and lowercase it.
2. Build a character-to-index mapping and index-to-character mapping over the distinct characters.
3. Create training sequences of fixed length: each input is maxlen characters, and the label is the character that follows them.
4. Convert labels to one-hot vectors so softmax cross-entropy can measure the error.

text = open("book.txt").read().lower()
chars = sorted(set(text))
char2idx = {c: i for i, c in enumerate(chars)}
idx2char = {i: c for i, c in enumerate(chars)}

seqs, labels = [], []
for i in range(0, len(text) - maxlen, 1):
    seqs.append([char2idx[c] for c in text[i:i+maxlen]])
    labels.append(char2idx[text[i+maxlen]])

LSTM Neural Network Model Training

model = keras.Sequential([
    keras.layers.Input(shape=(maxlen,)),
    keras.layers.Embedding(len(chars), 64),
    keras.layers.LSTM(128),
    keras.layers.Dense(len(chars), activation="softmax"),
])
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")
model.fit(X, y, epochs=50, batch_size=128)

Each epoch the loss tells us how many bits of uncertainty remain about the next character. With 50+ characters to choose from, a loss near 3.9 is random guessing, and anything meaningfully below that means the model is learning real patterns.

Predictions from the LSTM Model

To generate, seed the model with a warm-up string, predict a probability distribution over the next character, sample from it (with a temperature knob), append the character, and roll the window forward.

def sample(preds, temperature=0.7):
    preds = np.log(preds) / temperature
    exp = np.exp(preds - preds.max())
    return idx2char[np.random.choice(len(exp), p=exp / exp.sum())]

seed = list(text[:maxlen])
for _ in range(500):
    x = np.array([[char2idx[c] for c in seed[-maxlen:]]])
    probs = model.predict(x, verbose=0)[0]
    nxt = sample(probs)
    seed.append(nxt)
print("".join(seed[maxlen:]))

Streamlit Web App Integration with the Trained LSTM Model

Save the model with model.save and stash the character mappings with joblib. The app loads them, lets the user type a seed phrase and pick a temperature, then streams generated text:

import streamlit as st
st.title("Shakespeare-ish Text Generator")
seed = st.text_input("Seed phrase", "to be or not to be")
temp = st.slider("Creativity (temperature)", 0.2, 1.5, 0.7)
if st.button("Generate"):
    st.code(generate(model, seed, temp, n=500))

Deploy to Streamlit Community Cloud with requirements.txt containing tensorflow, streamlit, numpy, and joblib.

GRU RNN Variant Practical Implementation

Swapping the layer is a one-line change, and it is the fastest way to feel the difference between the two architectures:

model.add(keras.layers.GRU(128))  # instead of keras.layers.LSTM(128)

Train the same text generator with GRU and compare: training runs faster, and the generated text quality is usually comparable on this dataset.

Bidirectional RNN Architecture and Intuition

A unidirectional RNN reads left to right, so it cannot know about words that appear later. A bidirectional RNN runs two passes — one forward, one backward — and merges their outputs, so every position is informed by both sides of the sequence. That is ideal for sentiment or named-entity recognition over full sentences, but it is useless for streaming prediction, because the backward pass needs the whole sequence first.

model.add(keras.layers.Bidirectional(keras.layers.LSTM(64)))