9. Caching Strategies

Caching is critical for ML system performance. There are three distinct caching opportunities, each with different design considerations.

1. Feature Cache

Features are computed once and cached for reuse across requests.

Cache key: entity_id + feature_name
Cache store: Redis, Memcached
TTL: Minutes to hours, depending on feature freshness requirements
Invalidation: Write-through (update cache when feature source changes) or TTL-based

# Feature cache flow
def get_features(user_id):
    cache_key = f"features:{user_id}"
    features = redis.get(cache_key)
    if features:
        return deserialize(features)
    features = compute_features(user_id)
    redis.set(cache_key, serialize(features), ttl=300)  # 5 min
    return features

2. Inference (Prediction) Cache

Cache entire prediction results for identical requests. Highly effective when the same input is requested repeatedly.

Aspect	Design Choice
Cache key	hash(features + model_version)
Cache value	Serialized prediction + confidence score
Eviction	LRU + TTL-based expiration
Effectiveness	Best for: recommendations, search results, FAQs
Risk	Stale predictions if features change — use short TTL or invalidate

3. Model Cache

Keep model weights loaded in GPU/CPU memory to avoid reloading for each request.

GPU memory: Load model once, serve many requests. Multiple model versions can coexist.
LoRA adapter cache: Cache recently used adapters; evict LRU adapters when GPU memory fills.
Model warm-up: Run dummy inference on model load to trigger GPU kernel compilation and memory allocation.

Cache Strategy Decision Matrix

Scenario	Feature Cache?	Prediction Cache?	Model Cache?
User features (demographics)	Yes, TTL=1hr	—	—
Real-time features (session clicks)	No (always compute)	—	—
Identical product recommendations	—	Yes, TTL=5min	—
Fraud detection (unique transactions)	—	No (rarely repeats)	—
Multi-model serving	—	—	Yes, LRU eviction

Exercise: Design caching for a real-time product search ranking system. Three cache types are available. For each: define cache key format, TTL, eviction policy, and invalidation strategy. Describe how the cache hit ratio changes during a flash sale event with 10x traffic.