Part 8
Evaluation does not stop at deployment. This part covers testing before release, measuring live behavior, comparing versions safely, and keeping an eye on drift, cost, and latency once the system is serving real users.
Offline evaluation runs the model against fixed datasets before anything reaches users. It is fast, cheap, and safe — you can test a new model or prompt against your golden set and compare to the previous version with confidence. Its limit is that offline data is always slightly different from live traffic, so offline scores are necessary but never sufficient.
Online evaluation measures the system under real conditions: actual queries, real users, and live feedback. Because traffic is uncontrolled, results can be noisy — a weekend surge or a new marketing campaign shifts numbers. Collect online signals (user feedback, completion rates, latency) alongside the offline metrics so you see both the controlled and the real-world picture.
A/B testing randomly divides live traffic between two versions and compares outcomes. To trust the result: randomize at the user level, run long enough to gather a meaningful sample, guard against peeking at results early, and pre-register the metric you will decide on. A/B tests are the final arbiter for user-facing changes because they capture real behavior instead of simulated scores.
Shadow deployment sends a copy of live traffic to a new model while users keep seeing the old one. The candidate's responses are logged and compared offline against the incumbent — without any user risk. It is the perfect middle step between offline evaluation and full rollout, because it gives you a preview of behavior on real traffic with zero exposure.
Drift is the slow change in data or behavior after launch: users start asking new things, the world changes, or the model's outputs shift. Monitor both input drift (distribution of queries, fields, topics) and prediction drift (distribution of outputs and metric values). Detect it with simple statistical checks — comparing recent distributions to a reference window — and alert when it crosses a threshold you can act on.
Quality is only part of the story. A slower or more expensive model needs to be enough better to justify the trade. Measure p50 and p95 latency (generation time, retrieval time, end-to-end) and cost per request. A common rule: optimize the metrics users notice (first-token latency, correctness) rather than chasing raw throughput at the expense of experience.
| Signal | Why it matters | Typical alert |
|---|---|---|
| Input drift | Users changed how they ask | Topic mix shifts > 20% |
| Output drift | Behavior changed without a change | Score distribution moves |
| Latency | Slowness chases users away | p95 > target |
| Cost per request | Bills balloon silently | Cost > budget per day |
Continuous monitoring turns metrics into action. Instrument every run with the context you need to debug: the prompt, the retrieved chunks, the model version, and the timestamp. Set alerts on the metrics that indicate real problems — not every metric, or the team will ignore the noise. When an alert fires, the run logs should be enough to reproduce and explain the failure without a guessing session.