When Your RAG Knows It's Wrong

Go pull fifty queries your system got wrong last month. Actually pull them, into a spreadsheet, and read them one at a time.

If most of them are cases where the right chunk was sitting in the retrieved set and the model ignored it or talked around it, you have a generation problem and nothing in this article will help you.[1] If most of them are cases where the right chunk was never retrieved and the system answered anyway, in full confident prose, with no signal anywhere that something went missing: okay. Now we can talk.

That second failure is what this whole category exists for. Not "the model was wrong." Specifically: the model was wrong and had no idea, and neither did you, because a confident answer built on nothing looks exactly like a confident answer built on something. The claims bot from Part 1, the one that cited a policy exclusion that did not exist, is that failure in its purest form. Nothing in that pipeline was capable of noticing.

Roughly a third of the times someone tells me they need self-healing RAG, they have not read a single failure log. Not one. They read a paper, the paper had a nice diagram with a loop in it, and now the loop is on the roadmap for Q4.

I've mostly stopped arguing about architecture in those conversations. I just ask for the fifty queries. The conversation usually ends there, one way or the other.

01 / Definitions

Two papers, and the thing everyone actually builds

Before anything let's get the actual algorithm right, because the popular version of it is wrong in a way that changes what you'd build.

CRAG, as published, retrieves, then runs a retrieval evaluator over each query and document pair, then quantizes the resulting relevance scores into one of three confidence degrees and triggers exactly one action:[2]

Correct (at least one document above the upper threshold): the retrieved documents are not passed through untouched. They go through knowledge refinement, a decompose-then-recompose pass that splits documents into strips, scores each strip, drops the weak ones and reassembles the rest.
Incorrect (all documents below the lower threshold): the retrieved documents are discarded entirely, the query is rewritten into keywords, and web search supplies replacement knowledge.
Ambiguous (everything else): both of the above, combined. Refined internal knowledge plus external web knowledge, which is the deliberate soft option for when the evaluator isn't confident enough to commit either way.

Then it generates, once, and stops. Note what Correct does not mean: it does not mean "skip the work and go straight to the model." Refinement runs on the good path. That's the opposite of the mental model most people carry around where Correct is the fast lane and the machinery only kicks in when something looks broken. In CRAG the machinery always runs. Only the knowledge source changes.

The evaluator is also not prompt-shaped. The published version fine-tunes T5-large as a lightweight relevance scorer, about 0.77B parameters, deliberately much smaller than the generator. CRAG leaves your retriever and your generator alone, which is the sense in which it's plug-and-play, but the evaluator itself is a trained component. The paper's own limitation section says so directly: fine-tuning that external evaluator is unavoidable, and removing the need for it is future work.

Self-RAG is the one that's genuinely a training-time property end to end. The generator itself emits reflection tokens: whether to retrieve for this input at all, whether a retrieved passage is relevant, whether its own output segment is supported by that passage, and whether the response is useful. Those decisions live in the generator's weights, which is why swapping the underlying LLM breaks Self-RAG and doesn't break CRAG.[3]

And then there's the third thing, which is what almost everyone in production has actually built: a prompted relevance check inside control flow you own. One extra call, "are these documents relevant, YES or NO," keep what passes. I'm going to call that the corrective pattern rather than CRAG, and I'd encourage you to do the same in your own docs, because a YES/NO filter with no corrective action attached to the NO branch isn't corrective retrieval at all. It's a filter. Corrective means something different happens when the check fails. If nothing different happens, you've built a way to throw away context.

Fig.01
Single pass. The evaluator picks one of three knowledge sources, and generation happens once. The loop people draw from memory is not in the paper.

02 / The part nobody likes

A second opinion from the same doctor

Here's my actual objection, and it applies to the prompted corrective pattern, which is the version almost everyone ships.

If you grade retrieval with the same model family that generates the answer, you have not added a check. You've added a second opinion from the same doctor. The failure modes are correlated. The grader is confidently wrong in precisely the places the generator is confidently wrong, and those places are not randomly distributed: they're your domain jargon, your internal acronyms, your part numbers, your near-identical identifiers. That doesn't prove a relevance grader will repeat every generator mistake, but research on models judging their own generations has documented a related self-preference bias.[4]

I'd normally have to sell you on that as an intuition. I don't, because the CRAG authors ran the experiment. They compared their fine-tuned T5 evaluator against prompted ChatGPT on the same relevance-judgment task, in three configurations: plain prompting, chain of thought, and few-shot. The tuned 0.77B evaluator hit 84.3% accuracy. The best prompted configuration managed 64.7%. That is a very large gap in favor of the small purpose-trained thing over the large general thing, on a task the large general thing looks perfectly suited for.

Now apply it to the example I've been dragging through this entire series. SIL 10-9 120 and SIL 10-9 12k. In Part 2 the point was that a dense retriever cannot separate them, because the embeddings land almost on top of each other and cosine similarity has no notion of "one character different means a completely different article with markedly different electrical characteristics." Fine. Now put a prompted grader on top of that. Ask a language model whether a chunk about SIL 10-9 12k is relevant to a question about SIL 10-9 120. It says yes. Of course it says yes. It's running the same approximate, semantic, string-shaped reasoning the retriever ran. You've paid for an extra call and a few hundred milliseconds to be told the wrong answer is fine.

A check has to fail differently from the thing it's checking or it isn't a check. For identifier queries that means a deterministic gate: did the exact token from the query appear literally in this chunk, yes or no. That's a regex. It costs nothing, it runs in microseconds, and it outperforms your grader on that class of query for the boring reason that it has no opinions.

The second thing I'll pick a fight about is the web search fallback, and I want to be fair about its status: it isn't a bolt-on somebody added to the diagram later, it's core to the published method, and the authors' reasoning is sound. A system that knows its corpus can't answer something and reaches for help is better than one that clings to a bad corpus and fabricates. Agreed, in the abstract.

In an enterprise context it's still a laundering machine. Retrieval over your approved internal documents fails, so the system goes to the open internet, gets something plausible, and presents it in the same voice, in the same box, with the same confidence as the approved answers. The paper mitigates this by preferring authoritative sources like Wikipedia. Your corpus is not Wikipedia-shaped. Nobody publishes your return policy. So the fallback either returns a competitor's return policy or a generic one, and both are worse than silence. If you're going to do it, the provenance has to survive all the way to the user, visibly, on screen, and in my experience it rarely does, because nobody designs the fallback path with the care they spent on the happy path. NIST's Generative AI Profile makes the same operational point more politely: provenance tracking helps trace outputs back to their sources and improve information integrity.[5]

03 / Cost

The loop that isn't in the paper, and eats your budget anyway

CRAG is structurally bounded. One evaluator pass over a handful of query and document pairs with a sub-billion-parameter model, one action, one generation. The authors report roughly 0.363 seconds per instance for standard RAG against 0.512 for CRAG on PopQA, but Table 6 covers the generation phase only and explicitly excludes retrieval and data processing, so those are not end-to-end latency numbers. What they do establish is narrower and still useful: the published algorithm has no multi-round retry cost built into it.

What is expensive is the thing teams actually build, which is the iterative cousin: grade, reject, re-retrieve with a rewritten query, grade again, and keep going until something passes or somebody's patience runs out. That's an agentic extension of the corrective idea, not CRAG. FLARE is one published example of retrieval becoming an iterative runtime behavior, although its mechanism retrieves during generation rather than looping over a CRAG verdict.[6] But it's what ends up in production, because "and if it still fails, try again" is the single most natural line of code anyone has ever written.

In that shape, every round is at least one more generation-sized call plus another retrieval. Median latency barely moves, because most queries clear on the first pass and never enter the loop. p95 detonates. And p95 is what users experience as "this thing is slow," because the queries that trigger corrections are disproportionately the hard, specific, high-stakes ones somebody actually cared about.

Which brings me back to the cost pattern from Part 1. Agentic loop, no cap on retries, up to twelve rounds on ambiguous part-number lookups, roughly forty thousand dollars in a month on a returns tool. The mechanism is the trap: the queries that loop the most are the queries that were never going to succeed, so you spend the most money on the requests with the worst odds. That's structural, not a bug in anyone's implementation.

So if you're building the iterative version: cap it at one extra round, two if you've measured that the second earns its keep and you probably haven't. Put a wall-clock budget on the whole request, not just a retry count, because a count doesn't protect you from one round that hangs. And define, explicitly, what happens when you give up. Or, and I'd genuinely consider this first, don't build the iterative version. Single-pass with three well-chosen actions was good enough to beat standard RAG across four datasets, and it terminates.

The give-up path is the part nobody builds. Go look at any corrective architecture diagram people draw on whiteboards. Beautiful loop. No exit worth the ink.

In production the exit is the most important arrow on the page. "I couldn't find anything reliable on this, here's what I searched and here's who owns that documentation" is a genuinely good answer. Every team I know ships the alternative instead, which is a fluent paragraph assembled from whatever happened to survive.

Fig.02
Left: the paper's reported generation-phase timing, which excludes retrieval and data processing. Right: the shape of the iterative extension teams build instead, where the median never enters the loop and the tail pays for everything.

I shipped a grader that quietly deleted evidence

This is the part where I tell you what I got wrong, which I intend to keep doing in every one of these.

I put a relevance grader in front of generation on an internal documentation assistant. Standard shape, and note that it was the filter shape I complained about above rather than anything corrective: retrieve twenty, grade each chunk, keep what passes, generate from the survivors. Nothing happened on the reject branch except deletion. It tested well. Retrieval metrics were unchanged, because I was measuring retrieval and the grader isn't retrieval. Answer quality on my handful of test queries looked a bit better. I shipped it.

What I did not do was log the grader's decisions. I logged the query, the retrieved set, and the answer. The retrieved set I logged was post-grader. So the chunks the grader threw away simply did not exist anywhere in my observability, and here is what I therefore could not see: on short, underspecified queries, the ones where someone types four words and expects the system to know what they mean, the grader was rejecting correct chunks at a rate I'd now put somewhere north of twenty percent. Not because the chunks were bad. Because the query was too vague for anything to look clearly relevant against it, and a model asked "is this relevant" without enough context to decide will drift toward no.

So the system got quietly worse at exactly the queries where people needed the most help, and the trace looked cleaner than before. Four tight, well-scored chunks and a confident answer. Healthier on every dashboard I had. It ran that way for something like six weeks, until a complaint arrived specific enough that I went looking.

The cost wasn't dollars. It was six weeks of a team's patience with a tool they'd been asked to adopt, spent on a regression I introduced and then made invisible. You don't get that back by shipping a fix. Some of those people just quietly stopped using it and I never won them back. I'm still, honestly, a bit annoyed about that one, mostly at myself, and I'd rather say so here than write it up as a tidy lesson learned.

The rule that came out of it, which I'd now treat as non-negotiable: log the grader's verdict, its score, and the full text of every chunk it discarded. If you can't answer "what did my system decide not to show the model, and why" from your logs, you don't have a corrective system. You have a black box that occasionally deletes your evidence.

04 / The tuning

Making it actually work

Decide where the check sits

You can check before generation (are these documents relevant to the query), after generation (is this segment supported by the documents), or both. CRAG is the first kind. Self-RAG does both, since its reflection tokens cover retrieval relevance and support for its own output. They catch different failures. Pre-generation checks catch retrieval misses. Post-generation checks catch unsupported claims, including the case where retrieval was fine and the model embellished on top of it anyway. If you only build one, and the failure keeping you up at night is the insurer-bot failure, build the post-generation one. Grounding checks catch fabrications. Relevance checks catch misses. Fabrication is the one that puts you in a meeting with legal.

Small and purpose-built beats large and general

The instinct is that judging relevance is an important call so it deserves the good model. On the CRAG authors' PopQA relevance benchmark, the result points the other way, and hard: a 0.77B fine-tuned scorer beat prompted ChatGPT by roughly twenty points. Relevance scoring is a narrow discriminative job, it's the kind of thing small models are good at once they've seen your data, and you're running it once per document, which is ten or twenty times per query. If you can't fine-tune, a small fast model at temperature zero with a tight rubric and a couple of examples is the pragmatic substitute, and you should hold it to a measured accuracy number rather than assuming it's fine. Put the saved money in the retriever, which has been the thesis of this series since Part 1.

Thresholds, and the problem sitting underneath them

CRAG needs two: an upper one above which a document counts as relevant, a lower one below which the whole retrieval counts as failed. Everything between is Ambiguous, and the authors found that middle band matters, because dropping it made the whole system hostage to evaluator accuracy. You cannot derive those numbers from first principles, and copying them from a paper is worse than guessing because it feels justified. You get them by labeling a few hundred query and chunk pairs from your own traffic and finding where the distributions actually separate, which is to say you need a golden set, which is the thing nobody has.

Including me. I said back in Part 1 that I'd been meaning to build a golden dataset for my own side project for about eight months. It has now been about nine.

In my defense, I made the spreadsheet. It has headers.

Route on disagreement, not only on score

This is the piece I'd push hardest, and it falls straight out of the hybrid setup from Part 2. You already have two retrievers running: a keyword leg and a dense leg. When they agree on the top results, retrieval is almost certainly fine. When they disagree sharply, when the BM25 top five and the dense top five share nothing at all, that's your signal to spend the correction budget.

Disagreement between two mechanisms that fail differently is a cheaper and far better-calibrated alarm than a model grading itself, because it's the one signal in your pipeline that isn't correlated with the model's own blind spots. It's also strictly free: you computed both rankings already. Most queries won't trigger it, which is the point.

Which of the three things to actually build

Real Self-RAG earns its keep when you have the labeled reflection data and enough volume to amortize the training, and when you're not going to want to swap the generator, because you'd be retraining. That's a narrow window. Full CRAG is the middle option, and its cost is one fine-tuned evaluator plus a fallback knowledge source you trust, which in an enterprise setting probably isn't the open web. The prompted corrective pattern is what most of you will build, and it's defensible, as long as you write it down as what it is and give the reject branch something to do besides delete.

05 / Measurement

The number that tells you whether it worked

Spreadsheet again, and I'm not going to apologize for it. Query, retrieved chunks, grader verdict per chunk, final answer, and a column where a human writes yes or no. Fifty rows. It costs a morning, and it beat the eighty-thousand-dollar observability platform that was still onboarding at week six, a thing I keep repeating because it keeps being true.

The separation matters more than the tooling. RAGChecker reaches the same conclusion from the evaluation side: retrieval and generation need distinct diagnostic metrics, because an end-to-end score cannot tell you which component failed.[7]

The specific number to pull out of it is not overall accuracy. Accuracy will wobble a point or two and you won't be able to attribute the change to anything. The number is the error rate among rejected chunks: of everything your grader discarded, what fraction was actually relevant. Statisticians call that the false omission rate. Track both if you can. The omission rate tells you how much of what you deleted was good. The negative rate tells you how much of the good stuff you deleted. You want the first one for debugging a grader and the second one for deciding whether to keep it.[8]

Either way, neither number exists unless you did two things: logged the discards, and had a human label ground truth. That's the uncomfortable part of Fig.03. Without discard logs, the entire bottom row of that matrix is missing from your system, including the cases where the grader did its job perfectly. Without labels, you have discards but no way to sort them into right and wrong. A grader that's too permissive is merely annoying, because you can see the noise it let through. A grader that's too strict is invisible, because the evidence of its mistakes is the thing it deleted.

Fig.03
The whole bottom row is missing from an ordinary pipeline, and without human labels nothing in it can be sorted into right and wrong. Two different denominators, two different questions.

Something I keep meaning to check and haven't. When the CRAG evaluator was fine-tuned, the negative examples were sampled at random from the retrieval results: things that look similar to the query but aren't relevant. And the robustness experiment simulates a bad retriever by randomly removing accurate results.

Random negatives and random removals are clean. Real internal corpora don't fail cleanly. They fail by returning the chunk that's ninety percent right. The version of the policy from two years ago. The section describing the process for a different region. The onboarding doc that was correct until the reorg. Nothing about the published numbers tells me how a relevance scorer handles that class of near-miss, and I suspect the answer is badly, because ninety percent right is exactly what relevance looks like from the inside. Somebody has probably studied hard negatives here properly. I haven't gone and found it.

The question I can't get past

If your system is capable of recognizing that it retrieved the wrong thing, then the information needed to retrieve the right thing was sitting somewhere in the pipeline the whole time. So why didn't the retriever use it?

Sometimes the honest answer is that the correction layer is a patch over a chunking strategy nobody wanted to redo, or an embedding model chosen in week one and never revisited, or a corpus that has three contradictory versions of the same policy in it and no metadata to tell them apart. That's not automatically wrong. Patches ship, and shipping matters. But it's worth asking, out loud, before you add the layer: am I adding intelligence here, or am I adding an apology?

If you're running any of these in production and you've measured the error rate among your grader's rejects, I want the number. Genuinely, whatever it is. Comment or DM, I'll take it however it arrives, and if it's better than what I'd guess I'd like to hear what you did differently, because I'd rather be corrected here than in six months by a user.

Notes

References

Nelson F. Liu et al. Lost in the Middle: How Language Models Use Long Contexts, TACL 2024.
Shi-Qi Yan, Jia-Chen Gu, Yun Zhu and Zhen-Hua Ling. Corrective Retrieval Augmented Generation, 2024. See Algorithm 1 and Sections 4–5 for routing, evaluator training, experiments and timing.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil and Hannaneh Hajishirzi. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, ICLR 2024.
Arjun Panickssery, Samuel R. Bowman and Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations, 2024.
Chloe Autio et al. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, 2024.
Zhengbao Jiang et al. Active Retrieval Augmented Generation, EMNLP 2023.
Dongyu Ru et al. RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation, 2024.
Metric definitions: false omission rate and false-negative rate.