How to Evaluate RAG Accuracy: Metrics That Actually Matter

A RAG system can score well on the metrics most teams reach for first and still confidently hallucinate in production. BLEU and ROUGE, both inherited from machine translation and summarization, measure how many words in a generated answer overlap with a reference answer. Neither one checks whether the answer’s claims actually trace back to what the system retrieved. A model can phrase something fluent, confident, and completely disconnected from its own retrieved context, and a word-overlap score will not flag it. That gap, between an answer that reads correctly and an answer that is grounded, is where most RAG accuracy measurement goes wrong before it even starts.

Most teams measuring RAG accuracy default to whatever metric their evaluation library ships with first, which is usually one built for a different problem. Knowing how to evaluate retrieval-augmented generation systems in a way that actually predicts production failure comes down to a smaller, more specific set of metrics, and they split cleanly into two categories most teams never separate: did the system find the right thing, and did the model actually use what it found.

Why BLEU and ROUGE Don’t Measure a RAG System’s Accuracy

Kapa.ai put it plainly in an article that’s otherwise about optimization, not evaluation: “surface metrics like BLEU do not capture faithfulness.” That sentence names the actual problem. A language model being graded on word overlap with a reference answer can restate a plausible-sounding fact that has nothing to do with the relevant documents the system actually pulled, and the score won’t move, because the score was never checking answer relevance to the retrieved context in the first place. It was checking overlap with a reference string.

This is not a small technical footnote. It’s the reason a RAG system can pass a benchmark built on reference-answer similarity and still fail the only question that matters in production: did the model’s answer come from somewhere real. Retrieval quality and generation quality are two different failure modes, and a single blended score can’t tell you which one broke. That split is the organizing idea behind every metric worth using below.

Retrieval Quality: Whether the Right Content Left the Vector Database

RAG exists specifically because a model’s own training data goes stale and can’t cover everything an organization knows, so it pulls from external knowledge sources at query time instead. Retrieval metrics answer a narrower question than most people think: not “was the answer correct,” but “did the retrieval step surface the content that could have made it correct.” Evidently AI’s evaluation guide names the standard set: precision@top k and recall@k from classic information retrieval, scored per single query against a labeled set, alongside hit rate and NDCG@k for ranking quality. Every one of these metrics is scored against content that came out of a vector database, not against the AI system’s final answer, which is exactly why a retrieval regression and a generation regression need to be told apart rather than graded with one number.

Precision and recall measure different things, and conflating them is the most common mistake in this section. Precision asks how much of what got retrieved was actually relevant. Recall asks whether the relevant content got retrieved at all, regardless of how much irrelevant content came along with it. For a retrieval augmented generation (RAG) pipeline specifically, recall is usually the more important failure mode to watch first. No amount of generation quality fixes a system that never retrieved the right chunk in the first place. A perfectly faithful model summarizing the wrong retrieved chunks is still wrong.

Context precision and context recall, the versions of these metrics scoped specifically to retrieved documents rather than a generic document set, measure whether the relevant information actually made it into the relevant chunks the model got to see, and are what AWS’s evaluation framework for Bedrock and Evidently both build around. These numbers move in response to a real change to your pipeline, chunking strategy, embedding choice, index configuration, which is exactly what makes them useful diagnostically, covered further in the sections below. Public retrieval-augmented generation benchmarks exist for comparing embedding models and retrievers in the abstract, but a benchmark built on someone else’s documents tells you little about how your own pipeline performs on your own content, which is the actual reason to build the evaluation datasets covered later in this piece instead of stopping at a published leaderboard number.

Generation Quality: Whether the Model Used What It Retrieved

Faithfulness, sometimes called groundedness, checks whether the claims in a generated answer are actually supported by the retrieved context, independent of whether the answer happens to read well. Answer relevance checks a separate thing: whether the answer actually addresses the user’s question, independent of whether it’s grounded. A model can be faithful to retrieved content and still dodge the question. It can also answer the question directly while fabricating details no retrieved document supports. These two metrics catch two different failures that a blended BLEU-style score collapses into one undifferentiated number.

Two architectures build a version of this checking into the pipeline itself rather than bolting it on afterward. Self-RAG trains a single model to emit reflection tokens as it generates: one token type decides whether retrieval is even needed, another critiques whether a retrieved passage is relevant, a third checks whether the generated text is actually supported by it. Corrective RAG is more mechanical: a lightweight evaluator grades each retrieved document’s relevance before generation, and a low grade triggers a re-query or discards the passage rather than letting it through unexamined. Neither replaces the metrics above. Both are evidence that evaluation belongs inside the pipeline, not only in a report generated after the fact.

The clearest demonstration of what a metric moving in response to a real fix actually looks like comes from AWS’s own writeup of a FAQ-style RAG system: context relevance started at 67%. The team changed the chunking approach to stop splitting mid-sentence and to keep each FAQ question paired with its answer in the same chunk. Context relevance moved to 93%. The more interesting detail is what happened next without any further change: the retrieval fix resolved existing hallucinations in the generated answers, without the team touching the prompt template at all. That’s faithfulness, a generation quality symptom, improving as a direct downstream consequence of a retrieval quality fix, which is the single clearest piece of evidence in this piece that separating retrieval metrics from generation metrics isn’t academic. It’s how you find out that a “generation problem” was a retrieval problem the whole time.

Where Obot Fits: The Trace Data Evaluation Depends On

Separating retrieval failures from generation failures only works if you have a record of each retrieval. Since Obot is our platform, here’s where it fits before the judging and suite-building sections. Once retrieval runs as a logged tool call rather than a direct, unobserved query against a vector store, every retrieval event already carries a record an evaluation suite can join against: which retrieval tool ran, with what query, for which agent and user, and when. Tool-level access control and a complete audit trail at the tool-call level, covered in more depth in Building Agentic RAG with MCP, produce exactly the per-call record that separates a retrieval failure from a generation failure after the fact, instead of forcing a team to reconstruct what happened from logs that were never designed for this. Obot approaches this as an Enterprise AI Control Plane built around three jobs: define what agents can reach, choose where they run, and prove what they did.

Gateway: Define What Agents Can Reach

Obot MCP Gateway is a centralized control plane built around that principle: every tool call, retrieval included, gets logged at the point it actually happens, which is also the point an evaluation pipeline needs the data from.

Device: Watch and Enforce With Obot Sentry

An evaluation suite only describes the pipeline it can see. A retrieval server a developer configures locally in Claude Code or Cursor never passes through a gateway, so its calls never show up in that record. Obot Sentry scans AI client configurations on developer machines and reports which MCP servers each one loads, so the retrieval paths you’re evaluating are the ones actually in use.

Hosted: Claude Environments

Obot v0.27 added Claude Environments, hosted Claude Code environments that run under the same control plane. Agents there start from the same policy-defined set of retrieval connections, which keeps evaluation runs comparable instead of varying with each developer’s local setup.

LLM as a Judge: Scoring Accuracy Without a Fixed Ground Truth

Benchmarking large language models in retrieval-augmented generation pipelines used to mean exactly one thing: reference-based scoring, grading an answer against a pre-written correct answer. That’s the older approach, and the one BLEU and ROUGE come from. It requires a test set of correct answers written in advance, doesn’t adapt well to edge cases the test set didn’t anticipate, and still can’t distinguish a grounded answer from a lucky one. LLM as a judge has become the dominant practical method for a reason: it can evaluate faithfulness and relevance against the actual retrieved documents for a given query, not against a fixed reference string, which is closer to what you actually want to know.

Hugging Face’s evaluation cookbook demonstrates a specific version of this worth copying directly: generate synthetic question and answer pairs from your own knowledge base using an LLM, then filter them through critique agents that check groundedness, relevance, and whether each question stands on its own without needing the source document for context. That filtering step matters, because an ungrounded or ambiguous synthetic question produces a meaningless score no matter how good the judge is. The notebook then scores answer correctness with a second LLM on a 1-5 rubric that requires a written rationale before a number, not a bare score with no explanation attached.

The honest trade off here: an LLM judge needs periodic validation against human-labeled ground truth, or you’re trusting a model to grade a model with no check on whether its judgment itself has drifted. Scores also don’t automatically transfer between different LLMs: a judge that scores reliably against other LLMs’ outputs in one domain can still miss failure modes specific to a model it wasn’t validated on, which is an argument for periodically re-checking the judge rather than treating validation as a one-time setup cost. RAGAS is a widely used open-source framework built specifically around this evaluation approach, named by multiple sources in this space as a practical starting point, though its exact metric implementations are worth checking against its own documentation directly rather than taking a secondary description on faith, the kind of claim that’s easy to get subtly wrong by paraphrasing someone else’s paraphrase.

Building a RAG Evaluation Suite That Tells You Where It Broke

The actual payoff of keeping retrieval metrics and generation metrics separate isn’t philosophical. It’s diagnostic. When accuracy drops in production monitoring, a regression in context precision or recall points you straight at chunking, indexing, or embedding choices, the ingestion side of the pipeline. A regression in faithfulness or answer relevance with retrieval metrics holding steady points at the prompt or the model itself. Collapse both into one score and you lose that signal entirely. You know something broke. You don’t know where.

This is also where teams discover their evaluation criteria were never written against anything a real user would actually ask. An evaluation prompt handed to an LLM judge is only as good as the rubric behind it, and a rubric built from a handful of known-answer examples tends to miss the edge cases real users hit. The fix: make sure the test set includes the kinds of questions users genuinely ask, pulled from production logs where possible.

This is where the trace data described earlier stops being optional. A context recall drop and a faithfulness drop look the same in an aggregate score, but a per-call retrieval record shows which one you’re looking at.

None of this is specific to a demo environment. In real-world applications, user inputs arrive messier than any test set anticipates, which is exactly why the evaluation suite needs to keep absorbing real production questions rather than staying frozen at whatever it looked like on launch day. The metrics that matter are the ones that move when you change something real and sit still when they should. A BLEU score moving tells you almost nothing actionable. Context recall dropping six points after an embedding model change tells you exactly where to look.

FAQ

What metrics should I use to evaluate RAG accuracy?


Split them by pipeline stage. For retrieval, use context precision and context recall (or precision@k and recall@k against a labeled test set). For generation, use faithfulness and answer relevance, scored against the actual retrieved context rather than a fixed reference answer.

Why don’t BLEU and ROUGE work for evaluating RAG?


Both measure word overlap with a reference answer, a method built for translation and summarization, where a correct answer has a fairly fixed phrasing. They don’t check whether a generated answer’s claims trace back to what was actually retrieved, so a fluent, ungrounded answer can score well.

What is the difference between retrieval accuracy and generation accuracy in RAG?


Retrieval accuracy measures whether the system found the right content. Generation accuracy measures whether the model used that content faithfully and answered the actual question. A system can fail at either stage independently, which is why a single blended accuracy number tells you less than the two separated.

What is a faithfulness metric in RAG evaluation?


Faithfulness, also called groundedness, checks whether every claim in a generated answer is actually supported by the retrieved context the model had access to. It catches the specific failure where an answer sounds right but invents details no retrieved document contains.

What is context precision versus context recall?


Context precision measures how much of what got retrieved was actually relevant to the query. Context recall measures whether the relevant content got retrieved at all, regardless of how much irrelevant content came with it. For most RAG pipelines, a recall failure is the more consequential one to catch first.

Can an LLM be used to judge RAG accuracy?


Yes, and it’s now the dominant practical method, since an LLM judge can score faithfulness and relevance against the actual retrieved documents for a specific query rather than a fixed written reference. It still needs periodic validation against human-labeled examples so the judge’s own accuracy doesn’t drift unnoticed.

How does Obot help with RAG evaluation?


When retrieval runs as an MCP tool, Obot MCP Gateway logs every retrieval call with the query, the agent, and the time, the record an evaluation suite needs to separate retrieval failures from generation failures. Obot Sentry finds retrieval servers configured locally that would otherwise sit outside that record.

What is RAGAS and is it worth using?


RAGAS is an open-source evaluation framework built around LLM-as-judge scoring for RAG pipelines, commonly referenced alongside the metrics covered here. Check its own documentation for the current implementation of any specific metric before relying on a secondary description of how it works.

How do I build a RAG evaluation dataset without hand-labeling hundreds of examples?


Generate synthetic question and answer pairs from your own source documents using an LLM, then filter them through a second pass that checks groundedness, relevance, and whether each question makes sense without the source document attached. This produces a usable test set without the cost of manual labeling at scale.The strongest published guidance organizes shadow AI management into five phases: discovering unsanctioned tools, governing their approval, protecting data in real time, educating employees at the point of risk, and measuring program effectiveness with board-readable KPIs. The gap in most versions of this framework is that all five phases are built around browser and SaaS-based AI tools and don’t extend to MCP servers or autonomous agents, which require the same five phases applied through device-level scanning, self-serve tool catalogs, and credential brokering instead of OAuth audits and browser monitoring.