Recurrent Neural Networks

An ANN treats every input as independent, which is fine for tabular data but wrong for language: the word "broke" means different things depending on the words before it. Recurrent neural networks (RNNs) were the first deep learning architecture to give models a memory. This chapter explains how that memory works, why it is fragile, and then walks through a complete sentiment-analysis project from raw movie reviews to a deployed web app.

Introduction to NLP in Deep Learning

Classical NLP pipelines relied on hand-built features such as TF-IDF vectors. Deep learning changed the game by learning representations: instead of counting words, a network learns to embed words into dense vectors where similar words sit close together. RNNs extended this further by processing a sequence step by step, keeping a hidden state that summarizes everything seen so far.

Understanding RNN Architecture — RNN vs ANN

In an ANN every input feeds through layers and produces an output; the weights are shared across examples but the network has no notion of one example following another. An RNN processes one token at a time, and at each step it combines the current token with the previous hidden state to produce a new hidden state. Crucially, the same weights are reused at every time step, so the network can handle sequences of any length with a fixed set of parameters.

h_t = activation(W_h @ h_prev + W_x @ x_t + b)

The hidden state h_t is the RNN's memory: it travels forward through the sequence, carrying context from earlier words into later decisions.

Forward Propagation with Time in RNN Training

During the forward pass, the RNN walks through the sequence from first token to last. At each step it computes a hidden state, and for a classification task it uses the final hidden state (or a pooled version) to make the prediction. Because the same weight matrices are applied at every step, the whole sequence is processed by one small set of parameters.

Backward Propagation with Time in RNN Training

Errors flow backward through time: the loss depends on the last hidden state, which depends on the second-to-last, and so on all the way to the first token. Backpropagation through time (BPTT) unrolls the RNN into a long feed-forward network and applies the chain rule across every step. This is where the trouble begins, because gradients must now be multiplied across many time steps.

Problems with RNNs

• Vanishing gradients: repeated multiplication of small derivatives makes early time steps stop learning, so long-range context is forgotten.
• Exploding gradients: occasionally the products grow instead of shrink, destabilizing training (mitigated with gradient clipping).
• Sequential bottleneck: each step depends on the previous one, so RNNs cannot be parallelized the way CNNs can.
• Long memory is hard: plain RNNs struggle with dependencies spanning dozens of tokens — the motivation for LSTM and GRU in the next chapter.

A Sentiment Classification Problem Statement

Our project: classify IMDb movie reviews as positive or negative based on their text. This is binary text classification, and the review length varies wildly, which makes it an honest test of sequence models. We will build an RNN with an embedding layer, train it on a fixed-length padded version of the reviews, and expose it through a Streamlit app.

Getting Started with Word Embedding Layers

An embedding layer is a lookup table: it maps each integer word index to a dense vector of learned values. Similar words end up with similar vectors because the training process pushes them together when they appear in similar contexts. The embedding dimension (for example 32, 64, or 128) is a hyperparameter that controls how rich the learned representation can be.

Implementing Word Embeddings with Keras/TensorFlow

from tensorflow import keras

embedding = keras.layers.Embedding(
    input_dim=vocab_size,   # number of distinct words kept
    output_dim=64,          # length of each word vector
    input_length=max_len    # fixed sequence length after padding
)

Keras handles the lookup internally. The embedding weights are trained like any other parameter, so the network learns word representations that are useful for this specific task.

Loading and Understanding the IMDB Dataset and Feature Engineering

The IMDB dataset ships inside Keras as integer-encoded reviews. Every review is a list of word indices, and num_words caps the vocabulary to the most frequent words. Because reviews differ in length, we need to convert them into a rectangular batch: pad_sequences truncates long reviews and appends zeros to short ones so every review has exactly max_len tokens.

from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing.sequence import pad_sequences

(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=10000)
X_train = pad_sequences(X_train, maxlen=200, padding="post", truncating="post")
X_test = pad_sequences(X_test, maxlen=200, padding="post", truncating="post")

Training a Simple RNN with an Embedding Layer

model = keras.Sequential([
    keras.layers.Embedding(10000, 64, input_length=200),
    keras.layers.SimpleRNN(64),
    keras.layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(X_train, y_train, epochs=5, batch_size=64, validation_split=0.2)

Five epochs is enough to see the pattern: training accuracy climbs quickly while validation accuracy rises more slowly, and soon the model stops improving — a sign of the plain RNN's limited capacity.

Predictions from a Trained Simple RNN

New text must be encoded exactly like the training data. You need the same word-to-index mapping the dataset used, then padding to the same maxlen. After that, model.predict returns a probability in [0, 1], where values above 0.5 read as positive sentiment.

import numpy as np
from tensorflow.keras.preprocessing.text import one_hot

def predict_review(model, text, maxlen=200):
    tokens = [word_index.get(w, 2) for w in text.lower().split()]
    padded = pad_sequences([tokens], maxlen=maxlen, padding="post")
    p = float(model.predict(padded)[0, 0])
    return "Positive" if p >= 0.5 else "Negative", p

An End-to-End Streamlit Web App Integrated with the RNN and Deployment

Wrap the model behind a simple interface so anyone can paste a review and read the verdict:

import streamlit as st
st.title("IMDb Sentiment Analyzer")
review = st.text_area("Paste a movie review", height=200)
if st.button("Analyze"):
    label, p = predict_review(model, review)
    st.metric("Sentiment", label, f"{p:.1%} confidence")

Deploy the same way as the previous project: pin tensorflow, streamlit, and numpy in requirements.txt, push to GitHub, and connect the repository to Streamlit Community Cloud. Note that loading a Keras model takes a few seconds on the free tier, so show a spinner while it loads.