End-to-End ML Projects

From Notebook to Production

Until now you have trained models in notebooks where every step runs top to bottom in one session. A production machine learning project is different: it is a repeatable software system that others can run, extend, and deploy. This chapter walks through a complete project, from an empty repository to a working prediction service on a cloud platform.

GitHub and Code Setup

Start by creating a new repository on GitHub rather than building on your laptop in isolation.

1. Create a repository on GitHub named after the project (for example, student-performance-project).
2. Clone it locally so you work inside a version-controlled folder.
3. Create a virtual environment for the project and install the core packages: pip install numpy pandas scikit-learn matplotlib seaborn flask.
4. Add a requirements.txt so anyone can reproduce your environment, and a .gitignore to keep datasets and environment files out of Git.

git clone https://github.com/you/student-performance-project.git
cd student-performance-project
python -m venv venv
venv\Scripts\activate
pip install numpy pandas scikit-learn matplotlib seaborn flask

Commit early and often. Every commit is a checkpoint you can return to when a later change goes wrong.

Project Structure, Logging and Exception Handling

Organise the code into modules so each responsibility lives in its own file:

project/
  src/           # source code
    exception.py # custom exception class
    logger.py    # logging setup
    components/  # data ingestion, transformation, model training
    pipelines/   # training and prediction pipelines
    utils.py     # shared helpers
  app.py         # Flask prediction web app
  requirements.txt
  setup.py

Logging replaces print(). Logs carry timestamps and levels (INFO, WARNING, ERROR), which makes debugging far easier when the project runs unattended:

import logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s",
    handlers=[logging.FileHandler("run.log"), logging.StreamHandler()],
)
logging.info("Training pipeline started")

Exception handling ensures failures carry context. Define a custom exception class that captures the original traceback so you can see exactly where a job failed and why:

import sys

class ProjectException(Exception):
    def __init__(self, message, error_detail=sys):
        super().__init__(message)
        self.message = f"{message} | {sys.exc_info()}"

Wrap pipeline sections in try-except blocks that log the failure and re-raise, so a failed stage is never silent.

Problem Statement, EDA and Model Training

Before any code, write the problem down. For a running example, imagine predicting student exam scores from attributes like study hours, parent education, and previous grades. The problem statement answers three questions: what are we predicting, why is it useful, and how will we measure success.

Exploratory data analysis (EDA) is the inspection phase. Load the data with Pandas, check the shape and missing values, and visualise distributions with Matplotlib and Seaborn:

import pandas as pd
import seaborn as sns

df = pd.read_csv("data/student.csv")
print(df.info())
print(df.isnull().sum())
sns.histplot(df["scores"])

EDA answers whether features are numeric or categorical, how they correlate, and whether outliers exist. The insights here dictate the preprocessing choices later, so never skip this step. A quick benchmark model (a linear regression or random forest with default settings) tells you a reasonable baseline that fancier pipelines must beat.

Data Ingestion Implementation

Data ingestion is the component responsible for locating raw data and making it available to the rest of the pipeline. It separates the "where is the data" concern from everything else, so swapping a CSV for a database later touches only this module.

import pandas as pd
from dataclasses import dataclass

@dataclass
class DataIngestionConfig:
    raw_path: str = "data/raw.csv"
    train_path: str = "data/train.csv"
    test_path: str = "data/test.csv"

class DataIngestion:
    def __init__(self, config):
        self.config = config

    def initiate_ingestion(self):
        df = pd.read_csv(self.config.raw_path)
        train = df.sample(frac=0.8, random_state=42)
        test = df.drop(train.index)
        train.to_csv(self.config.train_path, index=False)
        test.to_csv(self.config.test_path, index=False)
        return self.config.train_path, self.config.test_path

A dataclass holds the configuration, and the component class performs the work. Notice the split is reproducible because of the fixed random seed.

Data Transformation with Pipelines

Raw data almost never matches what a model expects. Transformation encodes categorical columns, scales numerical columns, and fills missing values. scikit-learn Pipeline objects bundle these steps into one reusable unit:

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

num_pipe = Pipeline([("scaler", StandardScaler())])
cat_pipe = Pipeline([("encoder", OneHotEncoder())])

preprocessor = ColumnTransformer([
    ("num", num_pipe, numeric_features),
    ("cat", cat_pipe, categorical_features),
])

Fit the preprocessor only on the training data, then apply it to train and test. This prevents data leakage, where the test set quietly influences the learned statistics and inflates your reported accuracy.

Model Trainer Implementation

The model trainer loads the transformed data, trains one or more algorithms, and saves the winning model so the prediction pipeline can reuse it without retraining. Keep the trainer focused: fit, evaluate, persist.

from sklearn.ensemble import RandomForestRegressor
import joblib

class ModelTrainer:
    def __init__(self, model_path="artifacts/model.joblib"):
        self.model_path = model_path

    def train(self, X_train, y_train, X_test, y_test):
        model = RandomForestRegressor(n_estimators=200, random_state=42)
        model.fit(X_train, y_train)
        score = model.score(X_test, y_test)
        joblib.dump(model, self.model_path)
        return score

Models are persisted with joblib or pickle. The saved artifact is what the web application loads at startup to make predictions.

Hyperparameter Tuning

Default settings rarely give the best model. Tuning searches the space of hyperparameters for a better combination. Two practical approaches:

• Grid search (GridSearchCV) tries every combination of a candidate grid; thorough but slow.
• Random search (RandomizedSearchCV) samples combinations randomly; faster and often just as good for many parameters.

from sklearn.model_selection import RandomizedSearchCV

param_grid = {
    "n_estimators": [100, 300, 500],
    "max_depth": [5, 10, None],
}
search = RandomizedSearchCV(
    RandomForestRegressor(random_state=42), param_grid, cv=5
)
search.fit(X_train, y_train)
best = search.best_estimator_
print(search.best_params_)

Use cross-validation inside the search so the score reflects out-of-sample behaviour rather than memorised training data.

Building the Prediction Pipeline

The prediction pipeline packages the trained model into a service. The simplest option is a small web application with Flask that accepts a request, transforms the input with the saved preprocessor, and returns the prediction:

from flask import Flask, request, jsonify
import joblib

app = Flask(__name__)
model = joblib.load("artifacts/model.joblib")
preprocessor = joblib.load("artifacts/preprocessor.joblib")

@app.route("/predict", methods=["POST"])
def predict():
    data = request.get_json()
    features = preprocessor.transform([data["features"]])
    result = model.predict(features)
    return jsonify({"prediction": float(result[0])})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Test locally with a JSON request before deployment. A small sanity test that the endpoint returns a number is worth the five minutes it takes.

Deploying on AWS Elastic Beanstalk

AWS Elastic Beanstalk is the fastest path to a live web app because it manages the servers for you. Beanstalk inspects your code, creates an environment, and runs the app while you stay focused on code, not infrastructure.

1. Write a requirements.txt and make sure app.py exposes a Flask app (Beanstalk looks for application by convention).
2. Install the EB CLI: pip install awsebcli.
3. Initialize and deploy: eb init then eb create.
4. Monitor the health endpoint until it turns green, then open the app with eb open.

Beanstalk is ideal for first deployments because it hides networking, load balancing, and scaling details. It runs your code on an EC2 instance under the hood, and you can always graduate to controlling that instance directly.

Deploying on EC2 with ECR

For more control, run your model yourself on an EC2 instance, shipping the container through Amazon ECR (Elastic Container Registry). The steps follow a clear pattern:

1. Build a Docker image locally as shown in the previous chapter.
2. Authenticate Docker to ECR: aws ecr get-login-password | docker login --username AWS --password-stdin <registry-url>.
3. Create a repository, tag the image with its URI, and push it.
4. Launch an EC2 instance, install Docker on it, pull the image, and run it with docker run -p 8080:8080.
5. Open port 8080 in the instance's security group so the public can reach the service.

docker tag my-app:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app

This approach suits projects that need custom packages, GPU drivers, or unusual runtimes, because the instance is entirely yours.

Deploying on Azure with Containers

Azure offers a comparable path. The container is pushed to Azure Container Registry (ACR), then hosted on a service such as App Service or Container Apps:

1. Create a registry: az acr create --name myregistry --resource-group mygroup --sku Basic.
2. Build and push the image: az acr build --registry myregistry --image my-app:v1 ..
3. Create an App Service plan and a web app for containers, then deploy the image from ACR.

az group create --name mygroup --location eastus
az appservice plan create --name myplan --resource-group mygroup --is-linux
az webapp create --resource-group mygroup --plan myplan \
    --name mymodel-app --deployment-container-image-name \
    myregistry.azurecr.io/my-app:v1

The idea is the same on every cloud: build once, push the container to a registry, and let the platform pull and run it. The skills you learned for Docker carry straight over, which is why packaging matters more than any single cloud vendor.