Part 4
Retrieval-augmented generation (RAG) has two halves that must both work: the retriever finds useful context, and the generator turns that context into an answer. A weak retriever starves the model; a weak generator ignores good context. This part covers evaluating both halves and the pipelines built around them.
Retrieval quality is a ranking problem: given a query, are the relevant documents near the top? Measure it with the ranking metrics from Part 2 — Recall@K, Precision@K, MRR, MAP, and NDCG — where "relevance" means "this chunk contains the information needed to answer the question." Build a labeled set of query-to-relevant-chunk pairs and score the retriever on it independently of the generator.
How you split documents into chunks determines what the retriever can find. A chunk that splits a sentence in half, or mixes two unrelated topics, produces bad matches no matter how good the embedding model is. Evaluate chunking by checking how often the full answerable information lands in one retrievable chunk. Strategies to test: fixed-size windows with overlap, semantic splitting on paragraph or heading boundaries, and recursive splitting on structure.
The embedding model decides how "similar" two texts are in vector space. Good embeddings place paraphrases of the same meaning close together and unrelated topics far apart. Test them with a small set of pairs (query, relevant doc, hard negative) and check retrieval metrics or direct similarity separation. When you change embedding models, re-run the retrieval evaluation — small embedding changes can shift results noticeably.
A reranker re-scores the top K candidates to sharpen the ordering. To judge whether reranking actually helps, compare ranking metrics with and without the reranker on the same queries. Track both quality (NDCG) and the added latency, since a reranker that buys 2% NDCG at triple the latency may not be worth it.
Context precision asks whether the retrieved context is on-topic: are the chunks the system actually pulled relevant to the question, and are the relevant chunks ranked high enough? Low context precision means the generator is being given mostly irrelevant material and must fight through noise.
Context recall asks whether the retrieved set is complete: did the system find everything needed to answer well? A retriever can have high precision and still miss a key document, forcing the model to guess. The strongest signal is failure analysis — when answers are wrong, check whether the needed fact was even in the context.
| Metric | Question | Failure symptom |
|---|---|---|
| Context precision | Is what we retrieved relevant? | Answers ramble about unrelated topics |
| Context recall | Did we retrieve everything needed? | Answers confident but missing the key fact |
| Citation accuracy | Do citations match the answer? | Claim does not appear in the cited chunk |
| Hallucination rate | Is content grounded in context? | Fabricated names, numbers, or sources |
When answers cite sources, each claim should actually appear in the cited chunk. Manually or automatically verify every citation. A high citation rate with low citation accuracy is a false sense of security — users trust citations that point to content that never said what you claimed.
In RAG, hallucination is usually a grounding failure: the model produced a statement not present in the retrieved context. Separate the failure causes: was the fact missing from the context (retrieval problem), or present but contradicted by the model (generation problem)? That split tells you which half of the pipeline to fix.
RAGAS is an open-source framework that scores RAG pipelines with LLM-judged metrics: faithfulness (answer vs context), answer relevance, context precision, and context recall. Because it computes without hand-labeled data, it is a fast way to get a baseline across many queries.
from ragas import evaluate
from ragas.metrics import faithfulness, context_precision, context_recall
results = evaluate(dataset, metrics=[
faithfulness, context_precision, context_recall
])
print(results)DeepEval provides a metric library and a test harness for LLM and RAG applications. You write test cases declaring expected answers, and DeepEval computes metrics like answer relevance, faithfulness, and hallucination, then produces a pass/fail report you can wire into CI.
TruLens focuses on monitoring RAG (and other) applications in production. It instruments each pipeline run, records the feedback metrics per record, and shows trends over time so you can spot a retrieval regression or a drop in groundedness before users complain.