5. Data Flow

Data flows through an AI system in a continuous loop: ingestion → preprocessing → training → evaluation → deployment → inference → monitoring → feedback.

1. Ingestion

Raw data arrives from multiple sources:

• Batch sources: Daily exports from databases, CSV files, S3 buckets.
• Streaming sources: Kafka topics, Kinesis streams, event hubs — user clicks, transactions, sensor readings.
• External APIs: Weather data, social media feeds, third-party enrichments.

2. Preprocessing

Raw data is cleaned and transformed into features:

# Preprocessing pipeline (batch)
Raw data → Validate schema → Handle missing values
→ Normalize/scale → Feature engineering → Feature validation
→ Write to Feature Store (online + offline copies)

Key decisions: Do you preprocess in the pipeline (painless) or at serving time (flexible)? The answer depends on latency requirements and feature complexity.

3. Training & Evaluation

Preprocessed features are fed into training jobs. This can happen on a schedule (nightly), on demand (triggered by drift), or continuously (online learning).

• Training pipeline: Split data (train/val/test) → train model → evaluate → log metrics → register model if metrics improve.
• Evaluation: Compare against champion model. If new model outperforms on key metrics, it moves to staging.

4. Deployment

The trained model is packaged and deployed to the serving infrastructure. Common formats: ONNX, TorchScript, TensorRT, or raw Hugging Face checkpoints.

5. Inference

Live requests flow through the inference pipeline:

# Online inference flow
Request → Fetch features (from online feature store + cache)
→ Feature transformation (consistent with training)
→ Model prediction → Post-processing → Return result
→ Log prediction + features to audit store

6. Feedback Loop

The most critical and often overlooked step. Model predictions are compared against actual outcomes (once known) and fed back into the system:

• Explicit feedback: User ratings, thumbs up/down, corrections.
• Implicit feedback: Click-through rates, dwell time, purchase conversion.
• Outcome labels: Fraud confirmed/not-confirmed after human review.
• Drift detection: Statistical tests comparing prediction distribution to training distribution.

# Feedback loop data flow
Inference → Store prediction + features → Wait for ground truth
→ Compute model quality metric (e.g., accuracy, precision)
→ Compare to threshold → If degraded: trigger retraining alert