10. Caching

Caching is critical for AI system performance. Unlike traditional web caching, AI systems have multiple distinct caching opportunities.

1. Feature Cache

Feature computation can be expensive (e.g., computing a 7-day rolling average). Cache computed features so they are not recomputed for every request.

# Feature cache flow
Request: predict for user_id=42
Check feature cache (Redis): key = "features:user:42"
  Hit → return cached features (2ms)
  Miss → compute features from raw data (100ms)
         → store in cache with TTL (e.g., 5 min)
         → return features

Cache invalidation: Invalidate feature cache when underlying data changes (e.g., user completes a purchase → invalidate purchase-related features).

2. Model Cache

Keep model weights loaded in GPU/CPU memory. Avoid reloading for every request. This is standard in inference servers but worth explicit design.

• GPU memory: Load model once, serve many requests. Use dynamic batching to maximize utilization.
• Model versioning: Load multiple model versions simultaneously for A/B testing. Cache warm models.
• LoRA adapter cache: Cache recently used LoRA adapters in GPU memory; evict least recently used.

3. Prediction Cache

Cache identical prediction requests. Highly effective when the same input is requested multiple times (e.g., popular product recommendations, cached embedding computations).

# Prediction cache design
Hash(request features) → lookup in prediction cache (Redis/SearchCache)
  Hit → return cached prediction, skip model inference
  Miss → run model, cache prediction with TTL

Cache key: hash(features + model_version + model_id)
Cache value: serialized prediction + confidence score
Eviction: LRU + TTL-based expiration

4. Semantic Cache

For LLM-powered applications, semantic cache stores and retrieves results based on meaning, not exact match. Uses embeddings to find similar questions.

# Semantic cache (LLM response caching)
New query: "What's the weather in Tokyo?"
Embed query → vector (256-dim)
Search vector DB for similar queries (cosine similarity > 0.95)
  Hit → return cached response (no LLM call)
  Miss → call LLM, store (query, response, embedding) in cache

Benefits: Cache hits for rephrased questions
Cost: Embedding computation (cheap) vs LLM call (expensive)

Cache Strategy Decision Tree