Data Cleaning & Feature Engineering

Why Cleaning Comes First

Real datasets are messy. They arrive with missing cells, impossible values, duplicated rows, inconsistent labels, and severe class imbalances. The old saying “garbage in, garbage out” is literal in machine learning: a model is only as good as the data it trains on, and most of a data scientist's working hours are spent preparing data rather than fitting models. This chapter covers the highest-impact cleaning and feature-engineering moves.

Handling Missing Values

Missing data appears as NaN or None, and every modelling library either errors out or silently misbehaves on it, so you must decide what to do. The three main strategies:

• Drop: remove rows or columns with missing values. Use dropna() when the missing rows are few or when a column is missing almost everything. But dropping throws away information — be careful not to delete the rare rows you actually need.
• Impute with a constant or statistic: fill missing values with the mean, median, mode, or a fixed value. Medians are safer than means when outliers exist. Simple and effective for many models.
• Model-based imputation: estimate missing values from the other columns (k-nearest-neighbors imputation or iterative imputation). More expensive but preserves relationships.

Before choosing, always ask why the data is missing. If values are missing because of a systematic bias (e.g., only low-income respondents refuse to state income), the missingness itself may carry signal — a technique called missing-not-at-random handling, where you keep a “was missing” indicator column.

import pandas as pd

df = pd.DataFrame({"age": [25, None, 34, 41, None],
                   "city": ["Delhi", "Pune", None, "Delhi", "Pune"]})

# Fill numeric column with its median
df["age"] = df["age"].fillna(df["age"].median())
# Fill categorical column with the mode
df["city"] = df["city"].fillna(df["city"].mode()[0])
print(df)

Handling Imbalanced Datasets

An imbalanced dataset has one class vastly outnumbered by another — think fraud detection (a fraction of 1% are frauds) or rare-disease screening. A naive model that always predicts the majority class can score 99% accuracy while failing completely at its real job. This is why accuracy is a poor metric on imbalanced data; precision, recall, F1, and ROC-AUC tell the true story.

Practical strategies, from easiest to most involved:

• Resampling: oversample the minority class (randomly duplicate its rows) or undersample the majority class (drop rows randomly). Undersampling can discard useful data; naive oversampling can overfit.
• Class weights: many algorithms accept a class_weight parameter that penalizes mistakes on the minority class more heavily — no data changes needed.
• Evaluation by the right metric: switch from accuracy to precision/recall and inspect the confusion matrix.
• Synthetic generation: create new minority samples rather than duplicating existing ones — SMOTE, next.

Handling Imbalanced Datasets with SMOTE

SMOTE (Synthetic Minority Over-sampling Technique) fixes naive oversampling by creating new minority-class rows instead of copying old ones. The algorithm works in feature space:

1. Pick a minority-class sample.
2. Find its k nearest neighbors from the same class.
3. Choose one neighbor at random and draw a synthetic point along the straight line between the two samples (or slightly interpolate).

Because each synthetic point is a blend of two real ones, SMOTE adds variety to the feature space instead of memorizing duplicates. In practice it is usually combined with light undersampling of the majority class (SMOTE-ENN or SMOTE-Tomek) to clean noisy borderline samples.

from imblearn.over_sampling import SMOTE
from sklearn.model_selection import train_test_split

X = df.drop("target", axis=1)
y = df["target"]
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)

smote = SMOTE(random_state=42)
X_train_sm, y_train_sm = smote.fit_resample(X_train, y_train)

Handling Outliers Using Python

Outliers are extreme values that sit far from the rest of the data. Some are errors (a negative age), while others are genuine rare events (a billionaire in an income study). Deciding which kind you have is a judgement call that should be informed by your domain, not done blindly.

Three common detection methods:

• Z-score: flag values more than 3 standard deviations from the mean. Works fine for roughly Normal data.
• IQR rule: flag values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR. Robust to skewed data.
• Isolation Forest or Local Outlier Factor: model-based approaches that work on many columns at once (we cover these in the anomaly-detection chapter).

import numpy as np

values = np.array([10, 12, 11, 13, 12, 11, 98])
q1, q3 = np.percentile(values, [25, 75])
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
outliers = values[(values < lower) | (values > upper)]
print("Flagged outliers:", outliers)

After detection you can cap them (clip values to a percentile like the 1st and 99th), transform them away (log scale tames the right tail), or remove them — but before deleting anything, make sure the rows are truly bad and not simply surprising.

Data Encoding: Turning Text into Numbers

Machine learning models consume numbers, so categorical columns must be converted. Which encoding you pick changes the model's assumptions and its performance, so it deserves real thought.

Nominal (One-Hot) Encoding

One-hot encoding (OHE) creates one binary column per category: a 1 in exactly one of them for each row, and 0 elsewhere. City values like {Delhi, Pune, Mumbai} become three columns. This makes sense for nominal categories where no ordering exists — the model should treat “Delhi” as categorically different from “Pune,” not as “more” or “less.”

df_encoded = pd.get_dummies(df, columns=["city"])
print(df_encoded.head())

Two caveats: with high-cardinality columns (hundreds of cities), one-hot encoding explodes the feature count — and the model has to spend capacity on meaningless combinations. Also, since each category becomes an independent binary column, OHE throws away the information that different cities are different degrees of similar.

Label and Ordinal Encoding

Label encoding replaces each category with a plain integer (Delhi→0, Pune→1, Mumbai→2). It is compact but dangerous for nominal data: the model may interpret 1 < 2 as “Pune is between Delhi and Mumbai,” an ordering that does not exist. Use label encoding only for the target variable in classification or when the category truly has no other sensible treatment.

Ordinal encoding is the honest version of the same idea for ordinal categories — those with a natural, meaningful order. Ratings like {Low, Medium, High} map cleanly to {0, 1, 2} because “Medium” genuinely sits between “Low” and “High.” As long as the order is real, ordinal encoding is compact, preserves the ranking, and often beats one-hot encoding for tree-based models.

from sklearn.preprocessing import OrdinalEncoder

encoder = OrdinalEncoder(categories=[["Low", "Medium", "High"]])
df["priority"] = encoder.fit_transform(df[["priority"]])

Target-Guided Ordinal Encoding

Plain ordinal encoding assigns ranks 0, 1, 2 regardless of how the categories relate to your target. Target-guided encoding (also called target encoding) instead maps each category to a number derived from the target — typically the mean of the target among rows in that category. If “High” priority customers churn at 80% while “Low” churn at 10%, those numbers become the encoded values, teaching the model the real relationship.

There is one sharp risk: data leakage. If you encode a category using the target mean computed over the whole dataset — including the test rows — the model sees target information it should not, and your test performance will be flattering and fake. The standard fix is a smoothing + out-of-fold approach:

• Add a smoothing weight so categories with few rows shrink toward the global target mean.
• Compute each fold's encoding from the other folds only, never from the fold being validated.

def smoothed_target_encoding(df, col, target, prior, alpha=10):
    agg = df.groupby(col)[target].agg(["mean", "count"])
    code = (agg["count"] * agg["mean"] + alpha * prior) / (agg["count"] + alpha)
    return df[col].map(code)