21. Interview Prep

AI system design interviews assess your ability to architect production-grade ML systems. The focus is on trade-offs, data pipelines, scale, and practical engineering decisions — not just model accuracy.

The AI System Design Interview Framework

Clarify the problem: What is the ML task? What are the inputs/outputs? What are the constraints? (latency, throughput, cost, data volume)
Define success metrics: Model metrics (AUC, F1) and business metrics (CTR, conversion, revenue).
Data pipeline design: Data sources, ingestion, preprocessing, feature engineering, feature store.
Model architecture: Model choice (DNN, GBDT, LLM), training strategy, evaluation.
Serving infrastructure: API design, caching, batch vs real-time, autoscaling, GPU vs CPU.
Monitoring & feedback: Drift detection, retraining triggers, A/B testing, explainability.
Failure modes & trade-offs: What breaks? How do you degrade gracefully? What trade-offs did you make?

Common AI System Design Questions

Question	Key Areas to Cover
Design YouTube recommendations	Two-stage retrieval + ranking, user embeddings, real-time feedback, cold start, watch time prediction
Design a fraud detection system	Real-time inference, feature store, ensemble models, latency budget, imbalanced data, explainability for compliance
Design ChatGPT (a conversational AI system)	LLM serving (vLLM, continuous batching), context management, prompt caching, safety guardrails, rate limiting
Design an image search system	Embedding generation, vector DB (HNSW), hybrid search, indexing pipeline, real-time updates
Design a content moderation pipeline	Multi-modal (text + image), cascade: fast filter → slow deep model, human review queue, feedback loop
Design a personalized news feed	Candidate generation (follows, topics, trending), ranking (CTR model), real-time features, diversity, A/B testing

Scaling Estimation Practice

# Practice calculating these for any system you design
Users: MAU, DAU, concurrent users
Requests: QPS average and peak
Data storage: daily, monthly, yearly growth
Bandwidth: incoming and outgoing
Training data: volume, update frequency
Model size: parameters, memory footprint (FP32 vs FP16 vs INT8)
Inference cost: $ per prediction, $ per month

Must-Know Trade-offs for Interviews

Batch vs real-time: Freshness vs complexity. Use batch for training, real-time for serving.
Model complexity vs latency: Distill, quantize, prune to meet latency SLAs.
Consistency vs availability: For feature stores. Eventual consistency is usually acceptable.
Global model vs personalized: One model for all vs per-user models. Hybrid: global model + user embedding.
Cloud vs on-prem: Cost, latency, data sovereignty, GPU availability.
Cache everything vs selective caching: Cache hit ratio vs memory cost.

How to Present Your Design

Start with a high-level block diagram (5-7 boxes).
Walk through a single request end-to-end.
Deep-dive into the most interesting component (usually feature pipeline or model serving).
Discuss failure modes ("What happens if Redis goes down?")
Summarize trade-offs and what you would prioritize next.

Exercise: Prepare and practice answering this question: "Design an AI system that generates personalized workout and meal plans for 10M users." Use the 7-step framework. Record your answer (voice or written) and check: did you cover data pipeline, model architecture, serving, monitoring, failure modes, and trade-offs? Time yourself — aim for 35—40 minutes.