A model’s knowledge is limited to the data available when it was trained, so it cannot answer about private documents or anything newer. RAG brings the relevant information into the prompt at request time. The 2020 paper that named the technique paired a generator with a dense vector index of Wikipedia and found the combined model produced more specific, diverse and factual language than a model relying on its parameters alone.
encyclopedia
Retrieval-Augmented Generation (RAG)
Published Updated
Reviewed by Coy Robison, Founder, on
RAG combines search and generation: before the model answers, a retrieval step finds the passages most relevant to the question in an organization’s own documents and inserts them into the prompt. That lets a model answer from current, private information it was never trained on, and makes answers easier to check against their sources. Most RAG failures are retrieval failures, so quality depends as much on how documents are split, indexed and ranked as on the model.
Definition
A technique that augments model responses with retrieved context from trusted data sources.
Need this applied?
We help teams go from definitions to deployed workflows—safely and fast.
A RAG pipeline has a retrieval side and a generation side, joined by an augmentation step that decides what context the model sees. Surveys of the field describe a progression from “naive” retrieve-then-read pipelines to advanced and modular designs that add query rewriting, reranking and multiple retrieval steps.
- • Ingestion and chunking: split documents into passages small enough to retrieve precisely.
- • Indexing: embeddings for semantic search, often combined with keyword (BM25) search.
- • Retrieval and reranking: fetch candidates, then reorder them by relevance.
- • Generation: a prompt that presents the retrieved passages and instructs the model to use them.
Where retrieval goes wrong
AnthropicChunking can strip a passage of the context needed to find or interpret it; Anthropic’s example is a chunk reading “revenue grew by 3% over the previous quarter” that never names the company or the period. In Anthropic’s tests, prepending a short generated context to each chunk before indexing cut the top-20 retrieval failure rate by 49% when used for both embeddings and BM25, and by 67% when reranking was added.
When you may not need RAG
AnthropicIf the whole knowledge base is small, retrieval adds moving parts for little gain. Anthropic suggests that a knowledge base under about 200,000 tokens (roughly 500 pages) can simply be placed in the prompt, with prompt caching keeping that fast and affordable; retrieval becomes necessary as the collection grows past that.
Retrieval puts the right text in front of the model; it does not guarantee the model uses it faithfully. Instruct the model to answer only from the supplied documents and to cite them, evaluate retrieval quality and answer quality separately, and consider automated groundedness checks that compare each answer with its sources.
FAQ
Does RAG replace fine-tuning?
No. They solve different problems: RAG supplies knowledge at the moment of answering, while fine-tuning changes the model’s weights and therefore its behavior, style or format. Many systems use both.
Does RAG eliminate hallucinations?
It reduces them but does not eliminate them. The model can still misread or ignore retrieved text, or fill gaps when retrieval returns nothing useful, so it should be allowed to say it does not know and critical answers should be checked.
What is a chunk, and how big should it be?
A chunk is one of the passages a document is split into for indexing. Smaller chunks retrieve more precisely but lose surrounding context; adding a short description of where each chunk sits in its document is one way to keep both precision and context.
Can retrieved documents be used to attack a RAG system?
Yes. Retrieved text reaches the model alongside its instructions, so an attacker who can place text in a source the system retrieves can attempt indirect prompt injection. Treat retrieved content as untrusted data and limit what the model can do with it.
Email this summary + checklist
Get a copy of “Retrieval-Augmented Generation (RAG)” and an AI readiness checklist in your inbox.