Traditional system design focuses on deterministic behavior, request handling, data storage, and reliability. ML system design adds data pipelines, model training and serving, feature stores, drift detection, and the uncertainty that comes with probabilistic models. ML systems must handle silent degradation (model drifts) in addition to traditional failure modes (server crashes).
Use batch when predictions can be pre-computed and the application can tolerate minutes/hours of staleness (e.g., daily recommendations, weekly churn scoring). Use real-time when the prediction depends on the latest data and users expect instant results (e.g., fraud detection, real-time search ranking, chatbots). Many systems use both — batch for cold-start, real-time for hot requests.
Training-serving skew occurs when feature distributions differ between training and serving. Mitigations: use a feature store with shared computation logic, log features at serving time and compare distributions, validate that the same preprocessing code runs in both paths, and monitor feature drift in production.
A feature store is a centralized repository for ML features shared across training and serving. You need one when: multiple models share features, you need low-latency online feature lookups, you want point-in-time correct training data (avoiding data leakage), or you want feature sharing across teams. Start with an offline feature store (Parquet files + SQL); add an online store (Redis) when latency requirements demand it.
Through monitoring at two levels: (1) system metrics — latency, throughput, error rates, cache hit ratios; (2) model metrics — prediction distribution shifts (data drift), accuracy changes when ground truth arrives (concept drift), feature distribution changes, and business metrics (CTR, conversion, revenue). Use tools like Evidently AI, WhyLabs, or NannyML for automated drift detection.
| Factor | Large LLM | Smaller Specialized Model |
|---|---|---|
| Accuracy | High on diverse tasks | Comparable or better on specific task |
| Latency | Higher (500ms—2s) | Lower (10—100ms) |
| Cost | Higher (GPU, API fees) | Lower (CPU, smaller instances) |
| Maintenance | Prompt engineering, RAG setup | Standard ML pipeline |
| Data needs | Prompt examples, retrieval corpus | Labeled training data |
Use an LLM when you need generalization across many tasks. Use a smaller model for a specific, well-defined task with latency/cost constraints.
Route a portion of traffic to the new model (challenger) and the rest to the current model (champion). Measure business metrics (CTR, conversion, revenue) and model metrics (latency, error rate, prediction distribution). Use statistical significance tests before declaring the challenger the winner. Shadow deployments run the challenger silently before A/B testing.
Start simple: feature store = S3 + Parquet, model registry = filesystem + metadata CSV, serving = Flask/FastAPI on a single instance. Add complexity only when needed: online feature store (Redis) when latency matters, model registry (MLflow) when you have multiple models, inference server (Triton) when GPU optimization is needed, monitoring (Prometheus) when you need alerts.