Key takeaways
A language model answers from memory. That memory is a blurred average of its training data. It can’t cite a source or notice that a fact has changed.
In 2020 a team of researchers proposed a simple fix. Fetch the relevant passages from your own documents when the question comes in, and put them in the prompt beside it. They called it retrieval-augmented generation, RAG for short. It’s now the standard way to put a model on top of private or fast-changing data.
The model still writes the answer. Retrieval decides what it reads first. This piece builds on the embeddings explainer and the context window explainer.
Six steps in two passes
The first pass happens once, before anyone asks anything.
You chunk your documents into passages, a paragraph or a section. It’s harder to get right than it looks. You embed each chunk into a list of numbers that puts similar meanings close together. When dense embeddings came out, they beat keyword search by 9 to 19 points on top-20 accuracy. Then you store each vector with its text in a vector database. It finds near neighbours fast with methods such as HNSW.
The second pass runs on every question.
You retrieve by embedding the question with the same model and fetching the closest chunks. You augment by putting those chunks in the context window beside the question. Then the model generates an answer based on them, ideally citing the chunk behind each claim.
The model is the last step. By the time it writes a word, the biggest decision is already made: which passages it gets to see.
In RAG the model writes the answer, but retrieval picks the evidence. That happens before the model sees anything.
Why does it work?
Reading real passages instead of recalling them cuts invented answers. A 2021 study said so in its title: retrieval augmentation reduces hallucination in conversation.
The original RAG paper showed it as a gain in ability. It set state-of-the-art results on open-domain question answering with a retriever over a Wikipedia index.
The knowledge lives in the index, not the weights. So you update it by re-indexing a document, not retraining a model. That’s why a survey of the field calls RAG the practical way to keep a model current, domain-specific and traceable to its sources.
Where does it break?
RAG fails quietly. A fluent wrong answer with a citation looks just like a right one.
The failures cluster in four places:
Three of the four happen before the model runs. Research on RAG failure points concludes you can only check a system in use, against questions with known answers. I take each failure and its fix apart in how RAG works and fails.
Three of the four ways RAG fails happen before the model runs. The model is usually working fine on bad inputs.
Where retrieval goes next
Right now retrieval is a fixed first step. Every question triggers the same fetch.
The next step lets the model decide when to search, what to search for and whether one search was enough. Retrieval then becomes a tool the model calls. That’s how agentic systems work. The Model Context Protocol standardises how a model reaches such tools, so “search this index” sits beside “query this database” and “call this API”.
RAG doesn’t make the model smarter. It gives the model access, like the context window does, but pointed at a library instead of one prompt.
Fetching is the easy part. The work is checking the right passage was pulled, that it still says what you think, and that the model trusted it over its own guess.
Frequently asked questions
What does RAG stand for?
Retrieval-augmented generation. 'Retrieval' fetches relevant text from your own data. 'Augmented' means that text is added to the prompt. 'Generation' is the model writing the answer from it. The name is the pipeline in order.
How is RAG different from fine-tuning a model?
Fine-tuning changes the model's weights so new knowledge is baked in. That's expensive and goes stale as soon as your data changes. RAG leaves the weights alone and supplies knowledge when the question comes in. Fine-tuning teaches a style or skill. RAG supplies current facts. Many production systems use a base model with RAG and skip fine-tuning.
Does RAG stop a model from hallucinating?
It cuts hallucination, because the model answers from retrieved source text instead of memory. It doesn't remove it. The model can still misread a passage, mix it with its own assumptions or contradict it. Grounding lowers the rate of invented answers. It doesn't guarantee a correct one.
Why does RAG give wrong answers even when the right document exists?
Usually the failure comes before the model. The passage may have been split badly during chunking. The retriever may have ranked the wrong chunk first. Or the right chunk may sit in the middle of a long context, where recall is weakest. The model can only work with what retrieval gave it.
Do I still need a large context window if I use RAG?
RAG means you need to put less in the window, because you send a few relevant chunks instead of a whole library. The two work together. Retrieval decides what's worth including, and the context window is the space it goes in. See the companion piece on the context window for that side.
Three-minute lesson on this When it makes things up Marketing course, basic. Free, no sign-up.