Regression

Regression is a supervised learning technique used to predict continuous numerical values. It models the relationship between one or more independent variables and a dependent variable by fitting a line (or curve) to the data.

Linear Regression Concept

Linear regression assumes a linear relationship between the input features and the target variable. The goal is to find the line (or hyperplane in higher dimensions) that best fits the data points by minimizing the error between predicted and actual values.

Simple Linear Regression

Simple linear regression models the relationship between a single feature x and the target y using the equation y = mx + b, where m is the slope and b is the intercept. The model learns these parameters from the training data.

import numpy as np
from sklearn.linear_model import LinearRegression

# Sample data: hours studied vs exam score
X = np.array([[1], [2], [3], [4], [5], [6], [7], [8]])
y = np.array([52, 55, 61, 65, 72, 78, 83, 90])

model = LinearRegression()
model.fit(X, y)

print(f"Slope (m): {model.coef_[0]:.2f}")
print(f"Intercept (b): {model.intercept_:.2f}")

# Predict score for 6.5 hours
pred = model.predict([[6.5]])
print(f"Predicted score: {pred[0]:.2f}")

Multiple Linear Regression

Multiple linear regression extends simple regression to multiple features. The model learns a weight for each feature, producing predictions as a weighted sum of the inputs plus an intercept term.

from sklearn.linear_model import LinearRegression

# Multiple features: [hours studied, previous score, attendance %]
X = [[3, 60, 85], [5, 65, 90], [7, 70, 88], [2, 55, 78], [8, 80, 95]]
y = [65, 72, 85, 58, 92]

model = LinearRegression()
model.fit(X, y)

print(f"Coefficients: {model.coef_}")
print(f"Intercept: {model.intercept_:.2f}")

# Predict for a new student
pred = model.predict([[6, 75, 92]])
print(f"Predicted score: {pred[0]:.2f}")

Polynomial Regression

When data follows a curved pattern, polynomial regression adds polynomial terms (squared, cubed, etc.) to the feature set, allowing the linear model to fit nonlinear relationships.

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
import numpy as np

X = np.array([1, 2, 3, 4, 5, 6, 7, 8]).reshape(-1, 1)
y = np.array([5, 15, 30, 55, 85, 125, 175, 235])

poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)

model = LinearRegression()
model.fit(X_poly, y)

print(f"Coefficients: {model.coef_}")
print(f"R-squared: {model.score(X_poly, y):.3f}")

Evaluating Regression Models

R-squared (R²)

R-squared measures the proportion of variance in the target variable explained by the model. It ranges from 0 to 1, where higher values indicate better fit. However, R² always increases with more features, so adjusted R² is preferred for multiple regression.

Mean Squared Error (MSE)

MSE calculates the average of the squared differences between predicted and actual values. Because errors are squared, larger errors are penalized more heavily, making MSE sensitive to outliers.

Scikit-learn LinearRegression API

The LinearRegression class in scikit-learn uses ordinary least squares (OLS) to find the optimal coefficients. It provides methods like fit(), predict(), and score() for training, prediction, and evaluation.

from sklearn.metrics import mean_squared_error, r2_score

y_true = [3.0, 4.5, 5.8, 7.2, 8.9]
y_pred = [2.8, 4.7, 5.5, 7.0, 9.2]

mse = mean_squared_error(y_true, y_pred)
r2 = r2_score(y_true, y_pred)

print(f"MSE: {mse:.3f}")
print(f"R-squared: {r2:.3f}")

When to Use Regression

Regression is suitable when your target variable is continuous and you need to forecast values, understand relationships between variables, or quantify the impact of features on the outcome. Common applications include price prediction, sales forecasting, and trend analysis.