How to evaluate a RAG system: metrics and regression tests

To evaluate a retrieval-augmented generation (RAG) system, measure retrieval and generation separately, then measure the whole pipeline. Retrieval is judged on whether the right passages were found. Generation is judged on whether the answer is supported by those passages and addresses the question. Both are scored against a fixed test set, and the same set is rerun on every change so that quality drops are caught before users see them.

Why RAG needs its own evaluation

A RAG system has two stages that can fail independently. The retriever may miss the relevant passage. The generator may ignore a good passage, misread it or add claims it does not contain. An answer can look fluent and still be wrong.

Judging only the final answer hides the cause. If retrieval found the right passage and the answer was still wrong, the fix is in the prompt or model. If retrieval missed, the fix is in chunking, embeddings, search or ranking.

Start with a test set

Every metric depends on a test set, sometimes called a golden set. Each item usually contains:

A question, written the way a real user would ask it.
The passages or documents that contain the answer.
A reference answer, or a list of facts a correct answer must include.
Optional labels such as topic, difficulty or question type.

Where the questions come from

The best source is real usage: search logs, support tickets and questions from subject experts. Synthetic questions generated by a model from your documents are useful for volume, but they tend to be easier than real ones because they echo the wording of the source. Review them by hand and mix them with real questions.

What to include

Cover the range of cases the system will meet:

Straightforward factual questions.
Questions that need information from more than one document.
Questions phrased differently from the source text.
Questions where documents conflict or one is out of date.
Questions the knowledge base cannot answer, where the correct response is to say so.
Questions that depend on user permissions, if access control applies.

As an indicative guide, a few dozen carefully chosen questions are enough to begin learning, and a few hundred give more stable numbers. Quality and coverage matter more than size. Keep the set in version control and treat changes to it as seriously as changes to code.

Retrieval metrics

Retrieval metrics compare the passages returned for each question with the passages labelled as relevant. They are computed over the top k results, where k is the number of passages you pass to the model.

Metric	What it tells you	Notes
Recall at k	The share of relevant passages that appear in the top k	The most important starting metric. If the passage is missing, the answer cannot be grounded.
Precision at k	The share of the top k that are relevant	Low precision means noise in the context.
Hit rate at k	Whether at least one relevant passage appears in the top k	Simple and easy to explain.
Mean reciprocal rank (MRR)	How high the first relevant passage ranks, on average	Useful when one passage holds the answer.
nDCG	Ranking quality with graded relevance	Useful when some passages are more relevant than others.

Low recall points to problems in chunking, embeddings or the search method. Good recall with poor ranking points to the re-ranker. Retrieval metrics need no language model, so they are cheap enough to run on every commit.

Generation metrics

Generation metrics judge the answer. The main ones are:

Faithfulness, also called groundedness. Every claim in the answer is supported by the retrieved context. This is the direct measure of hallucination in a RAG setting.
Answer relevance. The answer addresses the question that was asked.
Correctness. The answer agrees with the reference answer or contains the required facts.
Completeness. The answer includes all the key points, not only some.
Citation accuracy. Each cited source really supports the statement attached to it.
Abstention. When the context does not contain the answer, the system says so instead of guessing.

Faithfulness and correctness are different. An answer can be faithful to an out-of-date document and still be wrong. An answer can be correct from the model's general knowledge and still be unfaithful to the context. Track both.

Scoring answers

There are three ways to score generated answers, and most teams combine them.

Human review. The most trustworthy and the slowest. Use it to set the standard and to audit the other methods.
Model-based grading. A language model scores each answer against a rubric. It scales well but must be calibrated. Compare its scores with human scores on a sample, and adjust the rubric until they agree closely.
Programmatic checks. Code checks for required facts, forbidden content, valid citations or correct format. These are fast, deterministic and limited in scope.

A good grading rubric is specific. Ask a narrow question with a pass or fail answer, such as "Is every factual claim in the answer supported by the context?", and require the grader to give its reasoning before the verdict. Vague scales from one to ten produce noisy results.

Regression evals

A regression eval is the same test set run automatically whenever something changes. In a RAG system, many things change: prompts, models, chunk sizes, embedding models, ranking logic and the documents themselves.

A practical setup looks like this:

Store the test set, prompts and configuration in version control.
Run retrieval metrics on every change. They are fast and cheap.
Run generation metrics on every change that touches prompts, models or retrieval, or on a schedule.
Compare results with the last accepted baseline and flag any metric that falls beyond an agreed tolerance.
Review failures by reading the individual examples, not only the averages.
Update the baseline only when a change is accepted.

Model outputs vary between runs, so small movements are noise. Set thresholds wide enough to avoid false alarms.

Evaluating in production

Real users ask things the test set did not anticipate. In production, collect:

User feedback such as ratings and corrections.
The rate at which the system declines to answer.
Samples of real conversations, graded for faithfulness.
Queries that returned weak retrieval scores.
Latency and cost per answer.

Feed failures back into the test set, so it becomes a record of every way the system has gone wrong.

Common mistakes

Measuring only the final answer and never retrieval.
Building a test set with no unanswerable questions.
Trusting a model-based grader that has never been compared with human judgement.
Tuning prompts against the test set until it passes. Keep a held-out portion for honest checks.

Getting started

Begin small: a short test set, recall at k, and a faithfulness check. Then add coverage and automation.

CodeBaxh is a software development and AI company. We build evaluation into our RAG and knowledge assistant projects from the start, and our LLM evaluation and guardrails service can add a test harness to a system you already run. To discuss your system, use the project enquiry form.

Frequently asked questions

What is the most important RAG metric?

Start with retrieval recall and answer faithfulness. Recall tells you whether the right information reached the model. Faithfulness tells you whether the model stayed within it. Most quality problems show up in one of the two.

Can a language model reliably grade another model's answers?

It can be reliable for narrow, well-defined checks, provided you calibrate it against human review. Without calibration you do not know what the scores mean. Recheck agreement whenever you change the grader model or rubric.

How often should evals run?

Run fast retrieval checks on every change. Run full generation evals on any change to prompts, models or retrieval settings, and on a regular schedule to catch drift caused by new or updated documents.

Are open-source evaluation tools worth using?

Yes. Tools such as Ragas, DeepEval, TruLens and promptfoo provide ready-made metrics and test runners. They save time, but they do not replace a test set built from your own data and reviewed by people who know the subject.