RAG Paper Explained: Retrieval-Augmented Generation
Introduction
A language model stores what it learns during training in its weights - its parametric memory. That memory is hard to update, cannot point to its sources, and when it is missing a fact, the model may confidently make one up, which is called hallucination (see LLM tokens, context windows, embeddings and hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations). The 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, from Facebook AI Research, proposed a fix: give the model a second, non-parametric memory - a searchable collection of documents - and let it retrieve relevant passages before it generates an answer. The paper named the approach Retrieval-Augmented Generation (RAG), and RAG is now the most common way to build applications which answer questions from a company's own documents. This post explains the RAG paper with diagrams. We will go through its retriever and generator, dense retrieval with embeddings, the two RAG variants and the results, and then see how RAG is built today, with a small retrieval pipeline in Java.The paper: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Authors: Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal and 6 others (Facebook AI Research, University College London, New York University)
Published: May 2020 - NeurIPS 2020
Authors: Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal and 6 others (Facebook AI Research, University College London, New York University)
Published: May 2020 - NeurIPS 2020
The two memories
The paper combines two components which had each been studied on their own -- A generator - BART-large, a pre-trained sequence-to-sequence TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained with 400 million parameters. This is the parametric memory: it knows how to write fluent text, and a lot of general knowledge.
- A retriever - Dense Passage Retrieval (DPR), which searches an index of the whole of English Wikipedia, split into 21 million passages of 100 words each. This is the non-parametric memory: precise, inspectable, and easy to replace.
The RAG architecture: retrieve passages, then generate from the question and the passages together.
Dense retrieval with embeddings
Classic search engines, such as BM25, match the words of a query against the words of documents. Dense retrieval matches meanings instead. DPR uses two BERTClick to read tutorial: BERT Paper Explained: Bidirectional Transformers encoders -- a document encoder turns every passage into a vector of 768 numbers, an embedding, computed once and stored in an index, and
- a query encoder turns the question into an embedding in the same vector space.
Dense retrieval: the passage whose vector points most nearly the same way as the question's is retrieved, even though it shares no words with the question.
RAG-Sequence and RAG-Token
The retrieved passage is treated as a hidden variable: the model does not know which passage is the right one, so it combines the predictions made with each of the top-k passages, weighted by how relevant the retriever thinks each passage is. This is called marginalizing over the passages. The paper has two ways of doing it -- RAG-Sequence uses the same passage for the whole answer: it generates a complete answer with each passage, and combines the answers' probabilities.
- RAG-Token can use a different passage for each token it generates, so a single answer can combine facts from several passages.
The two RAG variants with two retrieved passages.
Training is end to end: the generator and the query encoder are fine-tuned together on question-answer pairs, with no labels saying which passage is correct. The document encoder and the index are kept fixed, so the 21 million passages never need to be re-encoded.
Results
- Open-domain question answering: RAG set new state-of-the-art results on Natural Questions, WebQuestions and CuratedTrec, and strong results on TriviaQA - beating both pure parametric models such as T5, and "retrieve and extract" systems which copy an answer span from a passage. RAG could even give correct answers which did not appear word for word in any retrieved passage.
- Generation: when writing Jeopardy-style questions, human judges found RAG's output more factual and more specific than BART's on its own.
- Fact verification: on FEVER, RAG came within a few points of systems which used much more complex, task-specific pipelines.
- Updatable knowledge: swapping the 2018 Wikipedia index for a 2016 one changed the model's answers about world leaders accordingly - its knowledge could be updated by replacing documents, with no retraining.
RAG today
Modern RAG systems keep the paper's idea, but usually not its joint training. Instead, an off-the-shelf embedding model and a large language model are connected through the prompt -A typical RAG pipeline today. The model's weights are not changed; knowledge comes in through the prompt.
A tiny retrieval pipeline in Java
The program below shows the retrieval step and how the prompt is assembled. To keep it self-contained, it represents each text as a vector of word counts rather than a neural embedding - real systems call an embedding model instead, but the similarity search works the same way, using cosine similarity (the dot product of two vectors divided by their lengths).Retrieving the most relevant chunks and building a prompt
//TinyRag.java
import java.util.*;
import java.util.stream.*;
public class TinyRag {
static final Set<String> STOP = Set.of("the", "a", "an", "is", "of", "to", "in", "and", "it", "for", "with", "what", "does", "do", "how", "i");
// A stand-in for an embedding model: word counts, ignoring common words
static Map<String, Integer> embed(String text) {
return Arrays.stream(text.toLowerCase().split("[^a-z0-9]+"))
.filter(w -> !w.isBlank() && !STOP.contains(w))
.collect(Collectors.toMap(w -> w, w -> 1, Integer::sum));
}
static double cosine(Map<String, Integer> a, Map<String, Integer> b) {
double dot = 0;
for (var e : a.entrySet()) dot += e.getValue() * b.getOrDefault(e.getKey(), 0);
double na = Math.sqrt(a.values().stream().mapToDouble(v -> v * v).sum());
double nb = Math.sqrt(b.values().stream().mapToDouble(v -> v * v).sum());
return (na == 0 || nb == 0) ? 0 : dot / (na * nb);
}
public static void main(String[] args) {
List<String> chunks = List.of(
"Annual leave: employees get 24 days of paid leave per year, and up to 5 unused days can be carried over.",
"Expenses: submit expense claims within 30 days, with receipts, through the finance portal.",
"Remote work: staff may work from home up to 3 days a week with their manager's approval.",
"Laptops: lost or stolen laptops must be reported to IT security within 24 hours.");
// Indexing: embed every chunk once
List<Map<String, Integer>> index = chunks.stream().map(TinyRag::embed).collect(Collectors.toList());
String question = "How many days of paid leave do I get, and can unused leave be carried over?";
Map<String, Integer> q = embed(question);
// Retrieval: rank chunks by similarity to the question and keep the top 2
List<Integer> top = IntStream.range(0, chunks.size()).boxed()
.sorted(Comparator.comparingDouble((Integer i) -> cosine(q, index.get(i))).reversed())
.limit(2).collect(Collectors.toList());
for (int i = 0; i < chunks.size(); i++)
System.out.printf("chunk %d similarity %.3f%n", i, cosine(q, index.get(i)));
// Augmentation: put the retrieved chunks into the prompt for the LLM
String prompt = "Answer the question using only the context below. Cite the chunk numbers you used.\n\n"
+ top.stream().map(i -> "[chunk " + i + "] " + chunks.get(i)).collect(Collectors.joining("\n"))
+ "\n\nQuestion: " + question;
System.out.println("\n--- Prompt sent to the LLM ---\n" + prompt);
}
}
chunk 0 similarity 0.769 chunk 1 similarity 0.084 chunk 2 similarity 0.067 chunk 3 similarity 0.074 --- Prompt sent to the LLM --- Answer the question using only the context below. Cite the chunk numbers you used. [chunk 0] Annual leave: employees get 24 days of paid leave per year, and up to 5 unused days can be carried over. [chunk 1] Expenses: submit expense claims within 30 days, with receipts, through the finance portal. Question: How many days of paid leave do I get, and can unused leave be carried over?
Explanation of the code
embed()plays the role of the document and query encoders. Every chunk is embedded once, at indexing time; only the question is embedded at query time.- The annual leave chunk scores highest because it shares the most meaningful words with the question. With word counts, a question worded differently ("vacation allowance") would miss it - which is exactly the problem neural embeddings, like DPR's, solve.
- The prompt tells the model to answer only from the context and to cite its sources. This is the "augmented" part of retrieval-augmented generation, and it is what reduces hallucination: the model is grounded in text it can see.
- A second chunk is retrieved even though it is not relevant. Choosing k, filtering by a minimum score, and re-ranking the results with a more accurate model are the main tuning knobs of real RAG systems. An agent can also decide for itself when to search, as in the ReAct paperClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs.
Summary
In this post we went through the Retrieval-Augmented Generation paper. We saw how it combines a parametric memory, the BART generator, with a non-parametric memory, a dense index of Wikipedia searched by DPR, and how dense retrieval compares the embeddings of questions and passages instead of their words. We looked at RAG-Sequence and RAG-Token, the paper's results in question answering, generation and fact verification, and how its knowledge could be updated by swapping the index. Lastly, we saw how RAG is built today with embedding models, vector databases and prompts, and built the retrieval and prompt-assembly steps in Java. In the next post, we will see how to adapt a large model to a task cheaply, in the LoRA paperClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs.Continue Reading: AI and LLM Papers Explained
LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL
LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL