Evaluating Agent RAG Systems: Score All Three Edges

Ship an agent-RAG system that demos well and an awkward question follows almost immediately. How will you know when it gets worse? For a lot of teams the honest answer is that a user will tell them. Someone pastes a question into the box, the answer comes back subtly wrong, and that complaint is the first time anyone measures quality. Until then the evaluation plan was the demo.

Most teams do build an evaluation of sorts. They collect questions, run the system, grade the answers, with a rubric or with an LLM doing the grading. That measures something real, and it is not enough. It scores the finished answer while staying blind to how the answer was produced. When the score drops, you know that something broke. You do not know what.

The fix is a mental model I think of as the evaluation triangle, and this post walks it edge by edge, starting with the pair everyone reaches for, then digging into the two nobody watches. The short version, for an agent-RAG system that rewrites queries, retrieves over its own corpus, maybe loops a few times, the triangle is the smallest evaluation that can still tell you which component to blame. And on every edge, the same rule applies. Prefer the metric that gives the same number every time, and reach for the LLM judge only when nothing deterministic can see what you need measured. What each edge should weigh is a separate question, and the answer follows from what the system is for.

The triangle, not the pipeline

Strip an agent-RAG system down to its data flow and three artifacts remain.

1. The query the user asked.
2. The context the system retrieved.
3. The answer the model gave.

Pair them up and you get three measurable relationships, and each one guards against a failure the other two cannot see.

TruLens calls this the RAG triad, context relevance, groundedness, and answer relevance, one evaluation per edge of the triangle. RAGAS slices the same space into faithfulness, answer relevance, and context relevance in the paper, with context precision and context recall added in the library. The names differ across frameworks. The geometry does not. Query against context, context against answer, query against answer.

Each edge catches a specific disease.

Retrieval relevance catches the index missing the passage that existed all along. Groundedness catches the answer that reads beautifully and is invented. Answer quality catches the answer that is perfectly faithful to useless context. When a regression lands, the three edge scores tell you which component to open before you look at a single trace.

Does the answer actually answer

Start where everyone starts, the query against the final answer. It is the pair every evaluation reaches for, and plenty of teams never measure anything else. With reference answers it has solid deterministic options. Exact match and token-level F1 are the SQuAD-era workhorses. ROUGE overlaps n-grams. BERTScore matches tokens by contextual embedding similarity instead of exact string equality, which punishes valid paraphrases less. All of them share one blind spot, a single gold answer can only reward answers close to it, so they suit factual lookup more than open synthesis. On a golden set of real user questions, that is usually fine.

Without references you are in judge territory, and this is where G-Eval set the pattern, chain-of-thought reasoning plus a form to fill in. Its headline result, 0.514 Spearman correlation with human summary judgments against 0.474 for the best prior scorer, came with GPT-4 as the backbone, and that number is from 2023. The pattern is the part that stuck. It is still an LLM with an LLM’s opinions, which gets its own section below.

For agent systems this edge also grows a trajectory. Did it call the right tools in a sane order, did it respect the policy, did the task actually complete. tau-bench grades agents by comparing the final database state against an annotated goal state, and its numbers are a useful reality check. In the paper’s runs, gpt-4o agents, state of the art for function calling at the time, succeeded on under half the tasks, and the consistency metric across repeated runs, pass^k, fell under 25% in the retail domain. Two lessons in one benchmark. Task completion is measurable deterministically when the task touches state, and an agent that passes today can still fail tomorrow, so reruns belong in the eval.

Did retrieval bring the right passages

Now dig one hop upstream, the query against what retrieval actually returned. This is the oldest problem in the stack, so it owns the most boring and most stable toolkit in the whole field. Recall at k asks whether the gold passages are in the top k. MRR asks how high the first correct one ranked. nDCG adds graded relevance and discounts things that rank low.

Every one of them is deterministic. Same retrieved list, same gold labels, same number, forever, on every machine. That property is worth more than it looks when a score moves and you need to know whether the system changed or the metric did.

The catch is labels. Recall at k needs to know which chunks are the right ones for each question, which means a golden set of maybe 50 to 100 real questions, each annotated with gold chunk ids. That is a day or two of human work, once, and it keeps paying every commit. If even that is too much, the reference-free fallbacks rank a query-chunk embedding similarity (fixed model, fixed score), RAGAS context relevance where the scorer extracts the sentences it needs and reports the ratio, or a judge grading each chunk against the question. Each step right costs stability, which is the trade this post keeps coming back to.

In the agentic loop this edge also runs at runtime, inside the loop itself. The agent’s “is this enough evidence” check before it decides to search again is a context relevance score, whether you call it that or not. Giving the loop a threshold turns a vibe into a number, and the agentic loop itself is a topic I covered earlier.

Is the answer built from the context

The last pair is the answer against the context it claims to rest on, and it is the one that guards against fabrication. The research formulation I like best is AIS, attributable to identified sources. An answer is attributable when a reasonable reader would accept “according to [the source], [the statement]” for each statement in it. Groundedness, faithfulness, attribution, and hallucination rate are all names for scoring that property.

The deterministic end of this edge is citation mechanics. If the agent cites chunk ids, every cited id must exist. If the agent quotes, every quoted span should be findable verbatim in the corpus. Both are unit tests, free to run, and they catch real failure classes, fabricated quotes and dead citations. What they cannot catch is paraphrased invention, an answer that cites chunk 7 and then says something chunk 7 never said.

The next rung is a fixed model. Decompose the answer into individual claims, run NLI entailment claim by claim against the retrieved context, and report the supported fraction. That is the SummaC approach at document scale, sentence-pair NLI scores aggregated into one consistency verdict, good enough to set a 74.4% balanced accuracy state of the art on their six-dataset benchmark. A pinned NLI model is deterministic for as long as you keep the version pinned. Swap the model and you have silently changed the metric, so pin it and say so in the config.

RAGAS faithfulness computes the same fraction with the verifier replaced by an LLM. Supported statements over total statements, per the paper, which makes it more flexible than NLI and less stable, again the same trade.

One score is a vibe, three are a diagnosis

Why is the answer-only evaluation not enough, when the answer is the thing users actually see? Because two systems with identical answer scores can be broken in completely different ways.

Consider the system that answers correctly from its parametric memory while retrieval returns garbage. The answer score looks green, the demo glows, and the retrieval index is rotting underneath. The retrieval edge catches it in one number. Then consider the reverse, retrieval surfaces exactly the right passages and the model ignores them and improvises. The answer edge might still pass if the improvisation is good, and you have learned nothing about the hallucination you care about.

The three-edge split also turns regressions into triage. Answer score dropped 8 points. If recall at k dropped too, retrieval or the index changed, and the fix lives in ingestion, embedding, or reranking. If recall held and groundedness dropped, generation changed, and the fix lives in the prompt, the model, or the context window packing. If both held and only the answer score fell, the query mix shifted or your judge is misbehaving, which happens more than anyone admits. One dashboard number cannot do this. Three can.

The judge is an LLM. Grade accordingly.

LLM-as-judge is how every framework scales these edges to open-ended output, and it is genuinely useful. It is also an LLM grading LLM output, so it inherits the failure modes of the thing being graded. The literature has the receipts.

Position bias first. With ChatGPT as the evaluator, Wang et al. got Vicuna-13B to beat ChatGPT on 66 of 80 questions purely by manipulating the order the two answers appeared in. Verbosity bias next, in the MT-Bench judge study, a repetitive padded answer fooled GPT-3.5 and Claude-v1 judges 91.3% of the time, while GPT-4 fell for it 8.7%. The same paper measured self-enhancement, GPT-4 judging its own outputs gave itself a 10% higher win rate, Claude-v1 gave itself 25%. And it goes deeper than favoritism. Panickssery et al. found GPT-4 could recognize its own writing 73.5% of the time, fine-tuning pushed that over 90%, and the better a model is at recognizing its own output, the more it prefers it.

For balance, the MT-Bench study also found GPT-4 as judge agreed with human preferences on 85% of votes, slightly above the 81% human-human agreement. So the judge is not useless. It is good on average and reliably wrong in specific, repeatable directions, which is manageable if you know the directions.

Two more properties to respect. Temperature 0 is not a determinism guarantee, Google’s own docs describe temperature 0 outputs as mostly deterministic with a small amount of variation still possible, and OpenAI describes its APIs as non-deterministic by default. And a judge model updated by its vendor under the same name is a silent metric change on your dashboard.

So when a judge is unavoidable, run it with hygiene.

• Pairwise, never side by side solo, and call it twice with the positions swapped. Only a win in both orders counts, disagreement is a tie. Per MT-Bench this makes verdicts conservative rather than consistent, which is the property you want.
• Few-shot anchor examples raise consistency materially, per the same study.
• Pin the judge model version in config and log it with every score.
• Validate the judge once against 50 or so of your own human labels, and re-check when anything changes.
• Run the eval suite twice before believing any regression you see.

Work left to right

Put every metric from every edge on one axis, ordered by how much the score jitters when you rerun it, and a pattern falls out.

Unit checks, schema conformance, citations resolving, quotes findable, refusals detected. Reference scores, exact match, F1, recall at k, MRR, nDCG, ROUGE against gold. Fixed-model scores, NLI entailment, BERTScore, embedding similarity, deterministic exactly as long as you pin the model. Then, at the far right, the LLM judge.

The rule I use is to work left to right and stop at the leftmost metric that can see the thing I need measured. Spend determinism only where the question genuinely needs judgment, open-ended answer quality against a rubric, and nowhere else. The deterministic suite is free and unchanging, so it can run on every commit, while the judge suite costs tokens per run, so it gates releases instead.

The ladder orders metrics by stability, not importance. What to weigh on each edge follows from the goal of the project. A policy-lookup assistant answering refund questions gets almost everything from deterministic metrics, because gold answers exist and correctness is checkable, so the answer edge against references carries the budget. A research assistant writing open-ended briefs has no single gold answer to score against, so it spends on groundedness and a rubric judge, and accepts some noise. A safety-sensitive deployment inverts the priorities, groundedness outranks everything, and a faithful “I don’t know” beats a fluent invention. The triangle is the same in every project. The goal picks the weights.

Putting the triangle to work

The whole idea fits in one compact example harness. A golden set of questions with gold answers and gold chunk ids, three deterministic edge metrics, and the judge only at the end, swapped.

import re
from collections import Counter

# edge 1, retrieval: deterministic given gold chunk ids
def recall_at_k(retrieved: list[str], gold: set[str], k: int) -> float:
    return len(set(retrieved[:k]) & gold) / len(gold)

# edge 2, grounding: deterministic if the agent cites what it used
def citation_resolution_rate(answer: str, corpus: set[str]) -> float:
    cited = re.findall(r"\[chunk:([\w-]+)\]", answer)
    return sum(c in corpus for c in cited) / max(len(cited), 1)

# edge 3, answer: deterministic against a gold answer
def token_f1(prediction: str, gold: str) -> float:
    p, g = prediction.lower().split(), gold.lower().split()
    overlap = sum((Counter(p) & Counter(g)).values())
    if not overlap:
        return 0.0
    precision, recall = overlap / len(p), overlap / len(g)
    return 2 * precision * recall / (precision + recall)

# judge hygiene: pairwise with position swap, disagreement is a tie
def pairwise_verdict(answer_a: str, answer_b: str, judge) -> str:
    first, second = judge(answer_a, answer_b), judge(answer_b, answer_a)
    if first == second:
        return first
    return "tie"

Around 50 to 100 questions drawn from real usage, annotated once, is enough for all three deterministic edges to be meaningful in CI. The judge, one pinned model, rubric with anchors, swap protocol, runs on release candidates and on prompt or model changes, and it gets spot-checked against human labels quarterly. Grounding gets an NLI pass over decomposed claims as the middle rung. In agentic flows the same relevance gate runs inside the loop, and trajectory assertions, right tools called, schema respected, run alongside the answer edge.

None of this is exotic. Retrieval metrics are older than every model in the pipeline, NLI is a solved deployment problem, and the golden set is a weekend of annotation. The triangle is the minimum complete evaluation for a RAG system. Anything less is a vibe with a dashboard, and the first bad week will tell you which one you built.