Measuring RAG quality: beyond retrieval hit rates

Most teams building RAG pipelines measure retrieval recall: what fraction of relevant documents did the retriever surface? This is a useful number, but it answers the wrong question for a production system. The question users care about is not whether retrieval found the right documents. It is whether the answers they received were correct, well-grounded in the retrieved content, and actually useful for their task.

A high retrieval hit rate with a poor generation step produces confidently wrong answers. A moderate retrieval hit rate with a strong generation step that appropriately hedges on incomplete evidence produces trustworthy responses. Retrieval recall is a proxy for quality. It is not quality.

This post describes a complete measurement system for RAG pipelines, from retrieval evaluation through generation quality and into user-level outcome signals.

The RAG quality measurement stack

A useful measurement framework has four layers:

1. Retrieval quality — are the right chunks being retrieved?
2. Grounding quality — does the generated answer reflect the retrieved content?
3. Answer quality — is the answer correct, complete, and appropriate?
4. User outcome signals — does the answer actually help the user accomplish their task?

Most teams have layer 1. Layers 2–4 are where the signal that matters lives.

Layer 1: retrieval quality

Retrieval quality metrics are the foundation. You need them, but you should not stop here.

Recall@k measures whether the relevant documents appeared in the top k retrieved results. For a query where you have a known-correct document, did it appear in the top 3 (or 5, or 10)?

Mean Reciprocal Rank (MRR) measures where in the ranked list the first relevant document appears. A document that appears first is better than one that appears fifth.

Context precision measures what fraction of the retrieved chunks are actually relevant to the query. High recall with low precision means you’re retrieving relevant documents alongside a lot of noise — which degrades generation quality because the model’s context is diluted.

To measure these, you need a labeled evaluation dataset: queries paired with the known-correct source documents. Building this dataset is the first investment you need to make in RAG quality measurement. ManyLayers Workspace’s eval pipeline can help: use the “question-from-document” generator to automatically produce evaluation queries from your knowledge base, then review and correct the ground truth assignments.

Layer 2: grounding quality

Grounding measures whether the generated answer is actually supported by the retrieved content, or whether the model is confabulating beyond what the documents say.

Citation accuracy — if your system includes citations (references to specific retrieved chunks), do those citations actually support the claim they’re attached to? A grading model can evaluate this at scale: for each claim in the response, identify the citation, retrieve the cited chunk, and ask whether the chunk supports the claim.

Faithfulness score — does every factual claim in the response appear (explicitly or by clear implication) in the retrieved context? Faithfulness scoring decomposes the response into atomic claims and checks each one against the retrieved chunks.

In ManyLayers Workspace’s eval pipeline, faithfulness grading is available as a built-in metric:

{
  "eval_pipeline": "rag-quality",
  "metrics": [
    "retrieval_recall_at_3",
    "context_precision",
    "faithfulness",
    "answer_relevance"
  ],
  "grading_model": "anthropic/claude-sonnet-4-5"
}

A faithfulness score below 0.80 is a serious problem — it means your model is frequently generating claims that aren’t grounded in the retrieved documents. This could indicate insufficient retrieval (the model doesn’t have the context it needs and is filling gaps with training data), overly aggressive summarization, or a model that is not well-calibrated for RAG tasks.

Layer 3: answer quality

Even a grounded answer can be unhelpful. Answer quality metrics evaluate whether the response actually serves the user’s information need.

Answer relevance — does the answer address what the question was actually asking? A response that is factually correct but answers a slightly different question than the one posed is a low-quality answer.

Completeness — for questions that have multiple parts, did the answer address all of them? An evaluation prompt that decomposes the question into sub-questions and scores the response against each sub-question captures completeness systematically.

Appropriate uncertainty expression — does the response accurately represent what the retrieved documents do and don’t say? A high-quality RAG response expresses uncertainty when the retrieved documents are ambiguous or incomplete (“the documentation doesn’t specify the exact behavior in this case, but…”). A response that presents uncertain information with false confidence is a quality failure even if technically grounded.

No-answer accuracy — when the knowledge base genuinely doesn’t contain the answer, does the system correctly decline to answer rather than confabulating? Measure this with a set of queries designed to have no correct answer in your knowledge base, and evaluate what fraction of responses correctly acknowledge the gap.

Evals for these dimensions require a grading model — they’re not computable with deterministic logic. ManyLayers Workspace’s ELO leaderboard allows you to compare model configurations on these dimensions across a consistent evaluation set, so you can track which model and retrieval configuration combination produces the best overall answer quality.

Layer 4: user outcome signals

Automated evals are lagging indicators. User signals are leading indicators that complement automated measurement.

Thumbs up / thumbs down — explicit feedback at the response level. Low overhead for users, and even sparse signal is valuable. A 5% feedback rate on high-traffic RAG deployments can generate thousands of labeled examples per month.

Follow-up question rate — if a user asks a follow-up question immediately after receiving a response, the first response probably didn’t fully answer their question. Track this rate per query type and per knowledge base section. A high follow-up rate on a specific topic is a signal that retrieval or generation quality is weak for that topic.

Task completion rate — for RAG deployments embedded in workflows (not just standalone chat), did the user complete the downstream task? If your customer support RAG system produces an answer and the user then opens a support ticket anyway, the answer didn’t resolve their problem.

Copy / paste behavior — users who copy a response are likely using it. Users who immediately retype or clarify their question probably got an unhelpful response. This signal is available via the Workspace analytics interface.

Building an evaluation dataset

A RAG evaluation dataset is not a one-time artifact. It needs to be maintained as your knowledge base evolves. When documents are updated, evaluation queries grounded in the old document version may become invalid. When new topics are added to the knowledge base, your evaluation dataset should include queries for those topics.

A practical approach:

• Seed the dataset with 100–200 manually crafted query/answer pairs covering your most important topics.
• Augment automatically using the ManyLayers question-generator, but review each generated pair before adding it to the dataset.
• Update evaluation pairs whenever a source document is substantially revised.
• Add failure cases from production: when a user feedback signal indicates a bad response, add that query to the eval dataset with the correct answer.

Run the full eval pipeline after every significant change to retrieval configuration, embedding model, chunking strategy, or generation model. The ELO leaderboard in ManyLayers Workspace makes it straightforward to compare configurations on a consistent test set.

Practical guidance

• Do not use retrieval recall as your only quality metric. It is a necessary but not sufficient indicator of RAG quality.
• Invest in building and maintaining an evaluation dataset before trying to optimize quality. Without ground truth, you’re flying blind.
• Measure faithfulness continuously in production using a sample of live traffic. A drift in faithfulness score is an early warning of a knowledge base staleness problem.
• Include no-answer accuracy in your evaluation suite. A RAG system that confidently answers questions it doesn’t know the answer to is more damaging than one that correctly declines.

Takeaway

RAG quality is a multi-dimensional property that no single metric captures. A complete measurement system covers retrieval, grounding, answer correctness, and user outcome signals in combination. Teams that invest in this measurement infrastructure can iterate on their RAG configuration with confidence — and catch quality regressions before users do.