Classification

Classification is a supervised learning task where the goal is to predict a discrete label or category from a set of input features. It is one of the most widely used ML techniques, powering spam detection, medical diagnosis, and image recognition systems.

Logistic Regression

Despite its name, logistic regression is used for binary classification. It applies the sigmoid function to a linear combination of inputs to output a probability between 0 and 1. A threshold (typically 0.5) converts this probability into a class label.

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=200, n_features=4, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = LogisticRegression()
model.fit(X_train, y_train)

print(f"Accuracy: {model.score(X_test, y_test):.3f}")
print(f"Coefficients: {model.coef_}")

K-Nearest Neighbors (KNN)

KNN classifies a data point based on the majority class among its k nearest neighbors in the feature space. It is a non-parametric, lazy learning algorithm that stores all training data and makes predictions on the fly.

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print(f"KNN Accuracy: {knn.score(X_test, y_test):.3f}")

Naive Bayes

Naive Bayes applies Bayes' theorem with the "naive" assumption that all features are independent. Despite this simplifying assumption, it performs well on many real-world problems, especially text classification and spam filtering.

Decision Trees

Decision trees split data recursively based on feature values, creating a tree-like structure where each leaf node represents a class label. The splits are chosen to maximize information gain or minimize impurity (e.g., Gini impurity or entropy).

from sklearn.tree import DecisionTreeClassifier

dt = DecisionTreeClassifier(max_depth=3, random_state=42)
dt.fit(X_train, y_train)
print(f"Decision Tree Accuracy: {dt.score(X_test, y_test):.3f}")

Support Vector Machine (SVM)

SVM finds the hyperplane that best separates classes by maximizing the margin between the closest points (support vectors). Kernel functions allow SVM to create nonlinear decision boundaries by projecting data into higher-dimensional space.

from sklearn.svm import SVC

svm = SVC(kernel='rbf', C=1.0, gamma='scale')
svm.fit(X_train, y_train)
print(f"SVM Accuracy: {svm.score(X_test, y_test):.3f}")

Random Forest

Random forest builds multiple decision trees on bootstrapped subsets of data and averages their predictions. This ensemble method reduces overfitting and generally achieves better accuracy than a single decision tree.

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42)
rf.fit(X_train, y_train)
print(f"Random Forest Accuracy: {rf.score(X_test, y_test):.3f}")

Confusion Matrix & Evaluation Metrics

A confusion matrix summarizes classification performance by comparing predicted labels against actual labels. It contains four quadrants: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).

from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score

y_true = [0, 1, 1, 0, 1, 0, 1, 0]
y_pred = [0, 1, 0, 0, 1, 0, 1, 1]

cm = confusion_matrix(y_true, y_pred)
print("Confusion Matrix:")
print(cm)

precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)

print(f"Precision: {precision:.3f}")
print(f"Recall: {recall:.3f}")
print(f"F1-score: {f1:.3f}")

Precision, Recall, and F1-Score

Precision = TP / (TP + FP) — how many selected items are relevant. Recall = TP / (TP + FN) — how many relevant items are selected. The F1-score is the harmonic mean of precision and recall, providing a balanced metric when classes are imbalanced.

SGD Classifier

Stochastic Gradient Descent (SGD) Classifier is a linear classifier optimized via SGD. It is efficient on large datasets and supports various loss functions including hinge loss (SVM) and log loss (logistic regression).

from sklearn.linear_model import SGDClassifier

sgd = SGDClassifier(loss='hinge', max_iter=1000, tol=1e-3, random_state=42)
sgd.fit(X_train, y_train)
print(f"SGD Classifier Accuracy: {sgd.score(X_test, y_test):.3f}")