8. Offline vs Online Pipelines

Batch (Offline) Pipeline

Processes data in scheduled batches. Used for training, feature backfill, reporting, and non-real-time predictions.

# Batch training pipeline (simplified)
Schedule: Daily at 2 AM
1. Load yesterday's feature data from Feature Store (offline)
2. Load ground truth labels from data warehouse
3. Drop NaN rows, validate schema
4. Train model with hyperparameters from previous best run
5. Evaluate on validation set
6. If metrics improve: register model in registry, trigger deployment

Real-Time (Online) Pipeline

Processes data as it arrives with sub-second latency. Used for inference, feature computation, and immediate actions.

Lambda Architecture

Processes data through both batch and speed layers, merging results in a serving layer.

# Lambda Architecture
All incoming data splits into two paths:

Batch Layer:
  Data → Batch processing → Pre-computed views (serving layer)
  (complete, accurate, high latency)

Speed Layer:
  Data → Stream processing → Real-time views (serving layer)
  (approximate, low latency)

Serving Layer merges batch + speed views for query

Pros: Combines accuracy of batch with timeliness of streaming. Cons: Code duplication between batch and speed layers.

Kappa Architecture

Simplified: everything is a stream. Batch is treated as a special case of streaming with a larger window.

# Kappa Architecture
Data → Event Stream (Kafka) → Stream Processor → Serving Layer
No separate batch layer.
Batch jobs = stream processing with historical replay (reprocess from topic start)

Pros: Single codebase, simpler operations. Cons: Harder to handle large historical backfills efficiently.

Choosing Your Approach