Design a production recommendation system for an e-commerce platform with 50M users and 5M products. Homepage recommendations, product page similar items, checkout cross-sells.
Requirements
Functional: Personalized home page recommendations, similar products on product page, frequently-bought-together on checkout.
Non-functional: p99 < 200ms homepage, 10K QPS peak, 99.95% uptime, recommendations update within 1 hour of user action.
Scale Estimation
MAU: 50M → DAU: ~20M (40%)
Homepage loads/day: 40M (2 per user)
Peak QPS: 10,000 (including holiday surge)
Products per recommendation: 20
Features per user: ~200, per product: ~100
Feature vector size: ~300 × 8 bytes = 2.4KB
Architecture: Two-Stage Retrieval + Ranking
Stage 1: Retrieval (candidate generation) — Retrieve 500 candidate products per user from multiple sources:
Stage 2: Ranking — Score 500 candidates with a deep neural network:
User embedding (128-dim, stored in Redis)
Product embedding (128-dim, stored in Redis)
Cross features (user-product interactions)
Context features (time of day, device type)
Output: top-20 ranked products
Stage 3: Post-processing — Business rules, diversity, personalization boost.
Caching Strategy
Cache
Key
TTL
Store
Homepage recs
user_id
1 hour
Redis
Similar products
product_id
24 hours
Redis
User embeddings
user_id
1 hour
Redis
Product embeddings
product_id
24 hours
Redis
Trending (per category)
category + region
5 minutes
Redis
Data Pipeline
# Offline training (daily)
1. Load 30 days of interaction data (clicks, purchases, cart adds)
2. Compute user and product embeddings (Matrix Factorization, 128-dim)
3. Train ranking model (DNN with cross features)
4. Evaluate → register if metrics improve
# Online feature computation (streaming)
Real-time events → Kafka → Flink → update embeddings
Invalidate caches for affected users/products
# Batch feature computation (hourly)
Popularity scores, "frequently bought together" pairs, price elasticity
Key Trade-offs
Personalization vs cold-start: Blend trending items for new users, increase personalization as data accumulates.
Freshness vs stability: Micro-batch embedding updates every 5 minutes instead of real-time to avoid thrashing.
Diversity vs relevance: Apply diversity penalty (MMR — Maximum Marginal Relevance) to avoid showing 20 products from same brand.
Exercise: Extend this system with a real-time session-based component. The user's current browsing session (last 5 clicks) should instantly adjust recommendations without waiting for daily retraining. Describe: session vector computation, storage (which DB?), blending with long-term embeddings, and the additional latency budget.