Dimensionality Reduction

Dimensionality reduction reduces the number of features in a dataset while preserving as much information as possible. This helps combat the curse of dimensionality, reduces overfitting, speeds up training, and improves model interpretability.

Feature Selection

Feature selection chooses a subset of the original features without transforming them. It is preferred when interpretability is important and when you want to eliminate irrelevant or redundant features.

Filter Methods

Filter methods rank features by statistical scores independent of any ML model. Common approaches include correlation coefficients, chi-squared tests, mutual information, and variance thresholds. They are fast but may miss feature interactions.

from sklearn.feature_selection import VarianceThreshold, SelectKBest, chi2
import numpy as np

X = np.array([[0, 2, 3], [1, 1, 3], [0, 3, 2], [1, 2, 2]])
y = np.array([0, 1, 0, 1])

# Variance Threshold: remove low-variance features
selector = VarianceThreshold(threshold=0.5)
X_high_var = selector.fit_transform(X)
print(f"After VarianceThreshold: {X_high_var}")

# SelectKBest with chi-squared
kbest = SelectKBest(score_func=chi2, k=2)
X_selected = kbest.fit_transform(X, y)
print(f"Selected features: {kbest.get_support(indices=True)}")

Wrapper Methods

Wrapper methods evaluate feature subsets by training and testing a model on each subset. They are computationally expensive but often produce better feature sets by considering feature interactions.

Backward Elimination

Backward elimination starts with all features, removes the least significant one (highest p-value), and iterates until all remaining features are statistically significant. It is a greedy approach that considers feature importance in context.

import statsmodels.api as sm

# Assuming X has shape (n_samples, n_features)
# X = sm.add_constant(X)  # adds intercept
# model = sm.OLS(y, X).fit()
# while max(model.pvalues) > 0.05:
#     remove feature with highest p-value
#     refit model

Forward Construction

Forward construction starts with no features and adds the most statistically significant feature one at a time. It stops when no remaining feature significantly improves the model. It is the reverse of backward elimination.

Embedded Methods

Embedded methods perform feature selection during model training. Lasso regression (L1 regularization) naturally shrinks some coefficients to zero, effectively selecting features. Tree-based models also provide feature importance scores.

from sklearn.linear_model import Lasso
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=100, n_features=20, noise=0.1, random_state=42)

lasso = Lasso(alpha=0.1)
lasso.fit(X, y)

selected = np.sum(lasso.coef_ != 0)
print(f"Features with non-zero coefficients: {selected}")
print(f"Coefficients: {lasso.coef_}")

Feature Extraction

Principal Component Analysis (PCA)

PCA is the most popular feature extraction technique. It transforms the original features into a new set of uncorrelated variables called principal components, each capturing the maximum remaining variance in the data. The first component captures the most variance, followed by the second, and so on.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

X = np.random.rand(100, 10)  # 100 samples, 10 features

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

pca = PCA(n_components=0.95)  # retain 95% variance
X_pca = pca.fit_transform(X_scaled)

print(f"Original dimensions: {X.shape[1]}")
print(f"Reduced dimensions: {X_pca.shape[1]}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Cumulative explained variance: {np.cumsum(pca.explained_variance_ratio_)}")

PCA Explained Variance

The explained variance ratio tells you how much information (variance) each principal component captures. A common practice is to choose enough components to explain 90-95% of the total variance. The scree plot visualizes this diminishing return as more components are added.

Correlation-based Filters

Correlation-based feature selection evaluates subsets of features based on their correlation with the target and inter-correlation among features. Good feature subsets have high correlation with the target and low correlation with each other.