3. Stages of ML Systems

ML pipeline stages: data ingestion, preprocessing, feature store, training, registry, serving, monitoring

Every ML system can be understood as three major stages. Each stage has its own infrastructure requirements, failure modes, and design patterns.

Stage 1: Data Pipeline

The data pipeline is responsible for getting raw data from sources into a usable form for training and serving.

Ingestion: Collect data from event streams (Kafka), databases, APIs, or file uploads.
Validation: Schema validation, anomaly detection, data quality checks.
Preprocessing: Cleaning, normalization, feature engineering, splitting (train/val/test).
Storage: Raw data in data lake (S3), processed features in feature store (offline + online).
Versioning: Dataset versioning for reproducibility (DVC, LakeFS).

# Data pipeline failure modes
- Schema change in source (new field, removed field, type change)
- Missing values increase beyond threshold
- Late-arriving data breaks training-serving consistency
- Data quality issue (duplicates, outliers) degrades model silently

Stage 2: Training Pipeline

Transforms features and labels into a trained model artifact.

Feature loading: Query offline feature store for training dataset.
Training: Single-node or distributed training across GPUs.
Evaluation: Compute metrics on validation and test sets.
Registration: Log metrics, parameters, and model artifact to model registry.
Promotion: If metrics exceed champion model, stage for deployment.

# Training pipeline success criteria
- Training completes within SLA (e.g., under 4 hours)
- Validation metrics meet thresholds (e.g., AUC > 0.85)
- No data leakage between train and val sets
- Model artifact size within deployment budget

Stage 3: Serving Pipeline

Exposes the trained model for inference requests.

Model loading: Load model weights from registry into memory.
Feature serving: Fetch real-time features from online feature store.
Inference: Run model prediction with cached or live features.
Post-processing: Apply business rules, thresholds, calibration.
Monitoring: Log latency, predictions, features for drift analysis.

Exercise: Draw the three-stage pipeline for a spam detection ML system. For each stage, list: the specific technologies you would use, the data formats flowing through, what could go wrong, and how you would detect failure.