← Back to Tutorials Chapter 31

NLP: Text Representation

A Roadmap to Learning NLP

Natural language processing lets computers work with human language. A sensible learning path moves from basics to power:

1. Text basics: how to clean, split, and normalise text.
2. Representations: turning words into numbers a model can compute with (this chapter).
3. Classic models: applying machine learning on top of representations for tasks like spam detection.
4. Sequence models: recurrent networks and transformers that respect word order.
5. Applications: translation, summarisation, chatbots, and question answering.

The chapter you are reading sits firmly at step two: the vocabulary of text representations every NLP project builds on.

Practical Use Cases of NLP

• Spam filtering: classifying emails as spam or legitimate.
• Sentiment analysis: deciding whether a review is positive or negative.
• Named entity extraction: pulling people, places, and organisations out of news articles.
• Machine translation: converting text between languages.
• Chatbots and virtual assistants: understanding requests and generating replies.
• Search and recommendations: matching queries to documents by meaning.

Tokenization and Basic Terminology

Tokenization splits a piece of text into smaller units called tokens. A token is usually a word, but it can also be a punctuation mark, a number, or even a sub-word piece. Tokenization is the first step of almost every NLP pipeline because everything after it assumes text has been cut into processable chunks.

• Corpus: the whole collection of documents.
• Document: one unit of text in the corpus, such as a review or an email.
• Vocabulary: the set of distinct tokens that appear across the corpus.
• Token: one element produced by splitting a document.

Tokenization in Practice

The NLTK toolkit provides ready-made tokenizers. Word tokenization splits on word boundaries; sentence tokenization splits on sentence boundaries:

from nltk.tokenize import word_tokenize, sent_tokenize

text = "Hello world! Machine learning is fun."
print(word_tokenize(text))
print(sent_tokenize(text))

Notice the output keeps punctuation as its own token (!, .). If your downstream task does not care about punctuation, you will remove it during preprocessing.

Preprocessing: Stemming with NLTK

Words like "running", "runs", and "ran" all mean the same thing. Stemming chops word endings with simple rules to reduce them to a base form, called a stem. It is fast but crude; the stem is not always a real word:

from nltk.stem import PorterStemmer

stemmer = PorterStemmer()
for w in ["running", "runs", "happily", "eating"]:
    print(w, "->", stemmer.stem(w))

Stemming reduces vocabulary size so the model sees related words as one feature, but it can over-merge words with different meanings.

Preprocessing: Lemmatization with NLTK

Lemmatization also reduces words to a base form, but it uses vocabulary knowledge and grammar to return a proper dictionary word, the lemma. "Running" becomes "run", and "better" becomes "good". The trade-off is speed: lemmatization is slower than stemming.

from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize("running", pos="v"))  # run
print(lemmatizer.lemmatize("better", pos="a"))   # good

Passing the part of speech helps the lemmatizer choose correctly, which is why lemmatization and POS tagging are often used together.

Preprocessing: Stop Words

Stop words are the small, high-frequency words that carry little meaning on their own: "the", "a", "is", "of", "and". They appear in almost every document, so they add noise to a bag of words model without helping separate classes. NLTK ships a standard list:

from nltk.corpus import stopwords

tokens = word_tokenize("The cat is on the mat and it looks happy.")
cleaned = [t for t in tokens
           if t.lower() not in stopwords.words("english")]
print(cleaned)

Removing stop words shrinks the vocabulary and speeds up training. Note that for some tasks, such as sarcasm detection, the small words carry the signal, so removal is not always wise.

Part-of-Speech Tagging with NLTK

Part-of-speech (POS) tagging assigns a grammatical category to each token: noun, verb, adjective, and so on. The tagger uses surrounding context, so "book" is tagged as a verb in "book a flight" but a noun in "read a book":

from nltk import pos_tag

tokens = word_tokenize("The quick brown fox jumps over the lazy dog")
print(pos_tag(tokens))

POS tags are useful downstream: lemmatizers take them as input, and parsers use them to understand sentence structure.

Named Entity Recognition

Named entity recognition (NER) finds and classifies real-world entities in text: names of people, organisations, locations, dates, and amounts. This is the technology behind extracting a conference date from an email or pulling company names from the news:

from nltk import ne_chunk

sentence = "Snehal works at Microsoft in Seattle."
tokens = word_tokenize(sentence)
chunked = ne_chunk(pos_tag(tokens))
print(chunked)

The output is a tree whose leaves name the entities and their types, such as ORGANIZATION for Microsoft and GPE (geo-political entity) for Seattle.

What Comes After Preprocessing

Preprocessing produces clean tokens, but machine learning models compute with numbers, not words. The bridge between the two is text representation: a rule for converting tokens into vectors. The rest of this chapter surveys the classic representations, from simple counting methods to learned word embeddings.

One-Hot Encoding

The Intuition

One-hot encoding gives every word in the vocabulary its own dimension. Each word is represented by a vector of zeros with a single 1 in the position corresponding to that word. For a vocabulary of four words ["cat", "dog", "run", "sleep"]:

cat   -> [1, 0, 0, 0]
dog   -> [0, 1, 0, 0]
run   -> [0, 0, 1, 0]
sleep -> [0, 0, 0, 1]

Advantages and Disadvantages

• Advantages: trivial to understand, no training required, and words are fully distinct.
• Disadvantages: the vector size grows with the vocabulary (imagine a million dimensions for a real corpus), every pair of vectors is equally far apart so no meaning is captured, and "cat" is no closer to "dog" than to "run".

One-hot encoding teaches an important lesson: representing words as independent symbols throws away all relationship information.

Bag of Words

The Intuition

The bag of words (BOW) model represents a document by counting how often each vocabulary word appears in it, ignoring order. A document becomes a vector where position i holds the count of vocabulary word i. "The cat chased the dog" becomes a count vector over the vocabulary.

Advantages and Disadvantages

• Advantages: easy to build, captures word frequency, and works well with linear models for simple classification.
• Disadvantages: ignores word order entirely ("dog bites man" and "man bites dog" look identical), creates large sparse vectors, and treats common words as if they were as informative as rare ones.

BOW Implementation with NLTK

from sklearn.feature_extraction.text import CountVectorizer

docs = ["the cat chased the dog", "the dog slept"]
vectorizer = CountVectorizer()
matrix = vectorizer.fit_transform(docs)
print(matrix.toarray())
print(vectorizer.get_feature_names_out())

N-Grams

The Concept

A single word misses context. An N-gram is a contiguous sequence of N tokens. Unigrams are single words, bigrams are word pairs ("not good", "loved it"), and trigrams are triples. N-grams let the model see short patterns while staying cheap to build.

N-Gram BOW Implementation with NLTK

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(ngram_range=(1, 2))
matrix = vectorizer.fit_transform(["not good at all", "really good movie"])
print(vectorizer.get_feature_names_out())

Notice the feature list now contains phrases such as "not good" and "good at". These capture negation and word combinations that single words miss.

TF-IDF

The Intuition

Bag of words over-rewards frequent words. TF-IDF weights each term by two factors: how often it appears in the document (term frequency, TF) and how rare it is across the whole corpus (inverse document frequency, IDF). A word that is common in one document but rare elsewhere gets a high weight, which makes it a good discriminator:

TF-IDF = TF x log( total_documents / documents_containing_term )

Advantages and Disadvantages

• Advantages: down-weights filler words, highlights informative terms, and usually beats plain BOW for classification.
• Disadvantages: still ignores word order, still produces large sparse vectors, and the counts are relative to one corpus, so the weights are not portable to a different corpus.

TF-IDF Implementation in Python

from sklearn.feature_extraction.text import TfidfVectorizer

docs = [
    "the movie was great and the acting was great",
    "the plot was confusing and boring",
]
vectorizer = TfidfVectorizer()
matrix = vectorizer.fit_transform(docs)
print(matrix.toarray())

Word Embeddings

Counting methods treat every word as an isolated symbol. Word embeddings learn dense, low-dimensional vectors where similar words end up close together. With a good embedding, "king" sits near "queen", and "Paris" is to "France" as "Rome" is to "Italy" in the vector space. This captures meaning, which counting methods cannot.

Word2Vec Intuition

Word2Vec learns embeddings by turning a side task into a learning signal: use a word to predict its neighbours. Words that appear in similar contexts ("I drank ___", "___, she said") get pushed together in vector space. Two architectures do this in opposite directions: CBOW and Skip-Gram.

CBOW: Continuous Bag of Words

CBOW predicts the target word from its surrounding context words. Given the sentence "the cat ___ on the mat", CBOW takes ["the", "cat", "on", "the", "mat"] and tries to predict the missing "sat". The model is a tiny neural network: context vectors are averaged, projected through a hidden layer, and the output is a probability over the vocabulary.

CBOW is fast to train and works well for frequent words, because it smooths over many context examples.

Skip-Gram in Depth

Skip-Gram reverses the task: given the target word, predict the surrounding context words. For "sat", it tries to predict "the", "cat", "on", "the", "mat" as neighbours. The intuition is that a word's meaning is encoded by being able to produce its own context.

Because each target word gets its own training example for every context word, Skip-Gram trains slower than CBOW but learns better vectors for rare words. The hidden layer weights themselves become the word vectors once training finishes.

Advantages of Word2Vec

• Dense vectors, typically 100-300 dimensions, instead of vocabulary-sized ones.
• Captures semantic similarity: related words cluster together.
• Supports arithmetic: king - man + woman lands close to queen.
• Learned from raw text without any human labelling.

Average Word2Vec

Word2Vec produces a vector per word, but documents and reviews need one vector each. Average Word2Vec (AvgWord2Vec) averages the vectors of all words in a document to produce a single document vector. It is the simplest bridge between word vectors and text classification:

import numpy as np

def avg_word2vec(tokens, model):
    vectors = [model.wv[w] for w in tokens if w in model.wv]
    if not vectors:
        return np.zeros(model.vector_size)
    return np.mean(vectors, axis=0)

The averaging loses word order and subtle emphasis, but it is fast, robust, and a strong baseline before you reach for more complex sequence models.

Word2Vec Implementation with Gensim

Gensim provides a practical Word2Vec. Train on your own corpus of tokenized sentences:

from gensim.models import Word2Vec

sentences = [
    ["the", "cat", "chased", "the", "dog"],
    ["the", "dog", "slept", "on", "the", "mat"],
]
model = Word2Vec(sentences, vector_size=100, window=5,
                 min_count=1, sg=0, epochs=10)
print(model.wv.similarity("cat", "dog"))
print(model.wv.most_similar("dog"))

The sg flag chooses the architecture: 0 for CBOW and 1 for Skip-Gram. Once trained, model.wv holds the learned vectors you can average for document-level tasks, exactly as the next chapter will do.