Key takeaways
A RAG (retrieval-augmented generation) demo is easy to build and easy to trust. That’s the trouble. The version you build in an afternoon answers your test questions, so you ship it. Then it slowly breaks in production in ways that are hard to see, and it’s almost never the model’s fault.
This piece covers the four places RAG breaks once real documents and real traffic arrive, and the fix for each. If chunk, embed, retrieve and generate are new to you, start with what RAG really is.
Why does the demo hide the failures?
A demo asks a few questions whose answers sit in obvious places. The corpus is too small for retrieval to miss. Production asks questions you never tried, against a corpus that grows and changes. The failures were there on day one. Nobody was measuring them.
Chunking sets the ceiling
How you split documents at index time limits everything after it. A fixed-size splitter is the usual culprit. Cut every 500 tokens and sooner or later you slice a clause from the condition that governs it, a figure from its unit or a table row from its header. No retriever or model can put them back. Research cataloguing RAG failure points lists chunking among the failures that happen before the model is involved.
Split on structure instead: headings, paragraphs, list items, table edges. Add a little overlap. Attach the source, section and date so retrieval can filter.
When the best unit to retrieve is small but the model needs more around it, fetch the small chunk and pass its parent section. Where answers span many passages, RAPTOR builds a tree of recursive summaries and improves significantly on flat chunks. A summary can answer a broad question no single chunk covers.
Retrieval has to work on the whole corpus
Dense embeddings capture meaning. That’s exactly why they miss exact terms. Ask about error code E-4021 and a passage about similar errors can outrank the one that names it.
Dense retrieval beats keyword search by 9 to 19 points on top-20 accuracy on average. But the average hides the queries where only the literal match will do. And as the corpus grows, near-duplicates push the right passage out of the top results.
Three fixes:
So embeddings, as the embeddings explainer describes, become the first of several stages.
When the model fights the context
Models discount retrieved evidence that contradicts what they learned in training, unless it’s clear and pointed enough to override them.
Packing makes it worse. Stuff in twenty chunks and the key one may land in the middle. The lost-in-the-middle effect shows that’s where models read least reliably. A bigger window won’t save you, because usable context is shorter than the advertised length.
So send fewer, stronger chunks. Put the best evidence at the start or end. Tell the model to answer only from the passages, cite the one behind each claim, and say when they don’t support an answer.
Then check that it did. Faithfulness, whether an answer is grounded in what was retrieved, can be scored without a reference answer by frameworks like RAGAS.
Retrieval can hand the model the right passage and the model can still answer from its own memory. Grounding is something you instruct and then measure. Never assume it.
The index ages
A policy is revised and the old chunk keeps showing up. A document is deleted and its chunks linger. New documents stay invisible until something indexes them. The standard survey of the field treats keeping knowledge current as a core operational job.
Run indexing as a pipeline:
Measure, or you won’t see any of this
All four failures stay hidden for one reason. A fluent, cited, wrong answer looks exactly like a fluent, cited, right one. The failure-point research concludes that a RAG system can only be validated in operation.
So build a fixed set of questions with known answers from your own corpus. Score the two halves separately. Did the right passage reach the top results? Is the answer faithful to what was retrieved? Run it on every index change, prompt edit and model swap, like tests.
What comes next
The next step is to let the model run the loop itself. It decides whether to retrieve, judges what came back and searches again before it answers. Self-RAG trains a model to retrieve on demand and critique its own output, with gains in factuality and citation accuracy. Retrieval becomes a tool the model chooses to call. That leads to agents and the Model Context Protocol.
You’ll change the embedding model, the chunker and the vector store. The evaluation set is what stays.
Frequently asked questions
Why does my RAG system get worse over time?
Two reasons, both operational. First, the index ages. Documents get revised, deleted or added, and a snapshot taken at ingest keeps serving the old version until you re-index. Second, the corpus grows. More near-duplicate passages means the right one is less likely to land in the top results. The model isn't getting worse. The parts around it are drifting, and only an evaluation set will show you when.
What is hybrid search in RAG?
Running two retrievers and combining their results. A dense (embedding) search matches meaning. A sparse keyword search like BM25 matches exact terms. Dense search alone blurs IDs, codes and rare jargon. Keyword search alone misses paraphrases. Combine them, then rerank the merged list, and you catch both. In most production systems it's the single biggest retrieval fix.
What does a reranker do, and do I need one?
A reranker is a second-stage model (usually a cross-encoder). It re-scores the top candidates from your first retrieval against the query and reorders them, so the most relevant passage lands first. First-stage retrieval optimises for speed over a whole corpus. The reranker optimises for precision over a short list. If your recall is decent but the right passage isn't at the top, a reranker is usually the fix.
How do I evaluate a RAG system?
Build a fixed set of questions with known-good answers from your real corpus. Then measure the two halves separately. Retrieval: did the correct passage make the top results (recall and rank)? Generation: is the answer faithful to what was retrieved, and does it answer the question? Run it on every index change and prompt edit, like tests, so you catch a regression before users do.
Why does the model ignore the documents I gave it?
Two common causes. The model leans towards its own training and may discount a retrieved passage that contradicts it, unless that passage is clear and pointed. And if you send many chunks, the relevant one can land in the middle of the prompt, where models read least reliably. The fix: send fewer, better-ordered chunks and tell the model to answer only from the passages and cite them.
Three-minute lesson on this When it makes things up Marketing course, basic. Free, no sign-up.