5. Core Components

Production ML systems are built from reusable components. Mastering each component's design considerations is essential.

Feature Store

A centralized repository for features shared across training and serving.

• Offline store: Parquet/AVRO in S3 for batch training. Supports point-in-time queries to avoid data leakage.
• Online store: Redis / DynamoDB for sub-millisecond feature lookup during inference.
• Feature serving API: gRPC or REST endpoint returning features for entity keys (user_id, product_id).
• Consistency: Eventual consistency is typically acceptable (features are minutes stale). Strong consistency for critical features.

Model Registry

A versioned catalog of models with metadata:

• Model binary / weights / ONNX file
• Training metrics (AUC, precision, recall)
• Hyperparameters and training configuration
• Dataset version used for training
• Deployment status (staging, production, archived)
• Approval gate history

Inference API

The interface between the ML system and its consumers.

# Inference API design
POST /predict
{
  "features": {"amount": 299.99, "merchant_id": "amzn", ...},
  "request_id": "req_abc",
  "model_version": "v3.2"  # optional, defaults to champion
}

Response:
{
  "prediction": 0.12,
  "model_version": "v3.2",
  "latency_ms": 142,
  "explanation": {"amount": 0.05, "merchant_risk": 0.07}
}

Feedback Loop

The mechanism for collecting ground truth and improving the system over time.

• Explicit feedback: User ratings, thumbs up/down, corrections.
• Implicit feedback: Clicks, conversions, dwell time.
• Outcome labels: Transaction confirmed fraud/legit, recommendation clicked/purchased.
• Delay handling: Ground truth may arrive hours or days after prediction. Store predictions until labels are available.

# Feedback store schema
request_id (PK) | features | prediction | ground_truth | label_time
req_abc         | {...}    | 0.12       | 0.0          | 2026-06-15T10:35:00Z
req_def         | {...}    | 0.87       | 1.0          | 2026-06-15T11:00:00Z