RAG Paper Explained: Retrieval-Augmented Generation

Introduction

A language model stores what it learns during training in its weights - its parametric memory. That memory is hard to update, cannot point to its sources, and when it is missing a fact, the model may confidently make one up, which is called hallucination (see LLM tokens, context windows, embeddings and hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations). The 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, from Facebook AI Research, proposed a fix: give the model a second, non-parametric memory - a searchable collection of documents - and let it retrieve relevant passages before it generates an answer. The paper named the approach Retrieval-Augmented Generation (RAG), and RAG is now the most common way to build applications which answer questions from a company's own documents. This post explains the RAG paper with diagrams. We will go through its retriever and generator, dense retrieval with embeddings, the two RAG variants and the results, and then see how RAG is built today, with a small retrieval pipeline in Java.
The paper: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Authors: Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal and 6 others (Facebook AI Research, University College London, New York University)
Published: May 2020 - NeurIPS 2020

The two memories

The paper combines two components which had each been studied on their own -
  • A generator - BART-large, a pre-trained sequence-to-sequence TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained with 400 million parameters. This is the parametric memory: it knows how to write fluent text, and a lot of general knowledge.
  • A retriever - Dense Passage Retrieval (DPR), which searches an index of the whole of English Wikipedia, split into 21 million passages of 100 words each. This is the non-parametric memory: precise, inspectable, and easy to replace.
For each input, the retriever finds the top k passages (5 or 10 in the paper), and the generator produces the output while looking at the input together with those passages -
The query encoder embeds the question, maximum inner product search finds the top documents in the Wikipedia index, and the generator produces the answer from the question and documents Question x"Who wrote...?" Query encoderBERT → q(x) Search the indextop-k by q(x) · d(z) Document index21M Wikipedia passages d(z) GeneratorBART: x + passages Answer y the question also goes to the generator z₁..zₖ
The RAG architecture: retrieve passages, then generate from the question and the passages together.

Dense retrieval with embeddings

Classic search engines, such as BM25, match the words of a query against the words of documents. Dense retrieval matches meanings instead. DPR uses two BERTClick to read tutorial: BERT Paper Explained: Bidirectional Transformers encoders -
  • a document encoder turns every passage into a vector of 768 numbers, an embedding, computed once and stored in an index, and
  • a query encoder turns the question into an embedding in the same vector space.
The encoders are trained so that a question and the passages which answer it end up close together. The diagram shows the idea in two dimensions -
The question and three passages as vectors in one space; the passage pointing in the most similar direction gets the highest dot product Question"Who is the author of Hamlet?" Query encoder (BERT) Document encoder (BERT) Passagesencoded once, ahead of time one shared vector space (drawn in 2 dimensions; DPR uses 768) question q "Shakespeare wrote the play ..." "Hamlet is set in Denmark" "How to bake bread" highest dot product q · d → retrieved first
Dense retrieval: the passage whose vector points most nearly the same way as the question's is retrieved, even though it shares no words with the question.
The relevance score of a passage is simply the dot product of the two vectors, and finding the passages with the highest scores is called maximum inner product search (MIPS). (When the dot product and cosine similarity differ, and which to use, is explained in cosine similarity vs dot productClick to read tutorial: Cosine Similarity vs Dot Product for Embeddings.) With a library such as FAISS, it takes milliseconds even over 21 million passages. Because meaning is compared rather than words, the question about "the author of Hamlet" finds the passage which only says "Shakespeare wrote the play", ahead of a passage which shares the word "Hamlet" but does not answer the question.

RAG-Sequence and RAG-Token

The retrieved passage is treated as a hidden variable: the model does not know which passage is the right one, so it combines the predictions made with each of the top-k passages, weighted by how relevant the retriever thinks each passage is. This is called marginalizing over the passages. The paper has two ways of doing it -
  • RAG-Sequence uses the same passage for the whole answer: it generates a complete answer with each passage, and combines the answers' probabilities.
  • RAG-Token can use a different passage for each token it generates, so a single answer can combine facts from several passages.
RAG-Sequence writes a whole answer from each passage and combines the answers; RAG-Token combines the passages again at every token RAG-Sequence: one passage per whole answer passage 1 passage 2 token 1token 2token 3token 1token 2token 3 → complete answer A→ complete answer B combine A and B,weighted by retrievalscores RAG-Token: the passages are combined again for every token token 1token 2token 3 → one answer passage 1's share passage 2's share Token 1 here leans on passage 1 and token 2 on passage 2, so one answer can join facts from both. The shares are an illustration.
The two RAG variants with two retrieved passages.
RAG-Sequence suits answers that come from one source; RAG-Token suits answers which need facts spread over several passages. In the paper, the two were close on most tasks, with RAG-Token slightly better at generating Jeopardy questions and RAG-Sequence slightly better on most of the open-domain question answering datasets.

Training is end to end: the generator and the query encoder are fine-tuned together on question-answer pairs, with no labels saying which passage is correct. The document encoder and the index are kept fixed, so the 21 million passages never need to be re-encoded.

Results

  • Open-domain question answering: RAG set new state-of-the-art results on Natural Questions, WebQuestions and CuratedTrec, and strong results on TriviaQA - beating both pure parametric models such as T5, and "retrieve and extract" systems which copy an answer span from a passage. RAG could even give correct answers which did not appear word for word in any retrieved passage.
  • Generation: when writing Jeopardy-style questions, human judges found RAG's output more factual and more specific than BART's on its own.
  • Fact verification: on FEVER, RAG came within a few points of systems which used much more complex, task-specific pipelines.
  • Updatable knowledge: swapping the 2018 Wikipedia index for a 2016 one changed the model's answers about world leaders accordingly - its knowledge could be updated by replacing documents, with no retraining.

RAG today

Modern RAG systems keep the paper's idea, but usually not its joint training. Instead, an off-the-shelf embedding model and a large language model are connected through the prompt -
Indexing: documents are split into chunks, embedded and stored in a vector database. Querying: the question is embedded, similar chunks are retrieved, put into a prompt, and the LLM answers. Indexing (once, ahead of time) Documents Split into chunks Embed each Vectordatabase Querying (every question) Question Embed question Similaritysearch Prompt: question+ top chunks LLM→ answer top-k chunks are copied into the prompt
A typical RAG pipeline today. The model's weights are not changed; knowledge comes in through the prompt.
Each step of this pipeline - chunking, embedding, the vector index, re-ranking and the prompt - is explained in RAG architecture explainedClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval, and the common vector databases are compared in pgvector vs Chroma vs Qdrant vs Milvus vs RedisClick to read tutorial: pgvector vs Chroma vs Qdrant vs Milvus vs Redis Compared. This design is popular because the documents can be updated at any time, the answer can cite the chunks it used, access control can be applied to what is retrieved, and no model training is needed. Its quality depends heavily on retrieval: if the right chunk is not found, the model cannot use it.

A tiny retrieval pipeline in Java

The program below shows the retrieval step and how the prompt is assembled. To keep it self-contained, it represents each text as a vector of word counts rather than a neural embedding - real systems call an embedding model instead, but the similarity search works the same way, using cosine similarity (the dot product of two vectors divided by their lengths).
Retrieving the most relevant chunks and building a prompt
//TinyRag.java
import java.util.*;
import java.util.stream.*;

public class TinyRag {
  static final Set<String> STOP = Set.of("the", "a", "an", "is", "of", "to", "in", "and", "it", "for", "with", "what", "does", "do", "how", "i");

  // A stand-in for an embedding model: word counts, ignoring common words
  static Map<String, Integer> embed(String text) {
    return Arrays.stream(text.toLowerCase().split("[^a-z0-9]+"))
        .filter(w -> !w.isBlank() && !STOP.contains(w))
        .collect(Collectors.toMap(w -> w, w -> 1, Integer::sum));
  }

  static double cosine(Map<String, Integer> a, Map<String, Integer> b) {
    double dot = 0;
    for (var e : a.entrySet()) dot += e.getValue() * b.getOrDefault(e.getKey(), 0);
    double na = Math.sqrt(a.values().stream().mapToDouble(v -> v * v).sum());
    double nb = Math.sqrt(b.values().stream().mapToDouble(v -> v * v).sum());
    return (na == 0 || nb == 0) ? 0 : dot / (na * nb);
  }

  public static void main(String[] args) {
    List<String> chunks = List.of(
        "Annual leave: employees get 24 days of paid leave per year, and up to 5 unused days can be carried over.",
        "Expenses: submit expense claims within 30 days, with receipts, through the finance portal.",
        "Remote work: staff may work from home up to 3 days a week with their manager's approval.",
        "Laptops: lost or stolen laptops must be reported to IT security within 24 hours.");
    // Indexing: embed every chunk once
    List<Map<String, Integer>> index = chunks.stream().map(TinyRag::embed).collect(Collectors.toList());

    String question = "How many days of paid leave do I get, and can unused leave be carried over?";
    Map<String, Integer> q = embed(question);

    // Retrieval: rank chunks by similarity to the question and keep the top 2
    List<Integer> top = IntStream.range(0, chunks.size()).boxed()
        .sorted(Comparator.comparingDouble((Integer i) -> cosine(q, index.get(i))).reversed())
        .limit(2).collect(Collectors.toList());
    for (int i = 0; i < chunks.size(); i++)
      System.out.printf("chunk %d similarity %.3f%n", i, cosine(q, index.get(i)));

    // Augmentation: put the retrieved chunks into the prompt for the LLM
    String prompt = "Answer the question using only the context below. Cite the chunk numbers you used.\n\n"
        + top.stream().map(i -> "[chunk " + i + "] " + chunks.get(i)).collect(Collectors.joining("\n"))
        + "\n\nQuestion: " + question;
    System.out.println("\n--- Prompt sent to the LLM ---\n" + prompt);
  }
}
 OUTPUT of the above code
chunk 0 similarity 0.769
chunk 1 similarity 0.084
chunk 2 similarity 0.067
chunk 3 similarity 0.074

--- Prompt sent to the LLM ---
Answer the question using only the context below. Cite the chunk numbers you used.

[chunk 0] Annual leave: employees get 24 days of paid leave per year, and up to 5 unused days can be carried over.
[chunk 1] Expenses: submit expense claims within 30 days, with receipts, through the finance portal.

Question: How many days of paid leave do I get, and can unused leave be carried over?

Explanation of the code

  • embed() plays the role of the document and query encoders. Every chunk is embedded once, at indexing time; only the question is embedded at query time.
  • The annual leave chunk scores highest because it shares the most meaningful words with the question. With word counts, a question worded differently ("vacation allowance") would miss it - which is exactly the problem neural embeddings, like DPR's, solve.
  • The prompt tells the model to answer only from the context and to cite its sources. This is the "augmented" part of retrieval-augmented generation, and it is what reduces hallucination: the model is grounded in text it can see.
  • A second chunk is retrieved even though it is not relevant. Choosing k, filtering by a minimum score, and re-ranking the results with a more accurate model are the main tuning knobs of real RAG systems. An agent can also decide for itself when to search, as in the ReAct paperClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs.

Summary

In this post we went through the Retrieval-Augmented Generation paper. We saw how it combines a parametric memory, the BART generator, with a non-parametric memory, a dense index of Wikipedia searched by DPR, and how dense retrieval compares the embeddings of questions and passages instead of their words. We looked at RAG-Sequence and RAG-Token, the paper's results in question answering, generation and fact verification, and how its knowledge could be updated by swapping the index. Lastly, we saw how RAG is built today with embedding models, vector databases and prompts, and built the retrieval and prompt-assembly steps in Java. In the next post, we will see how to adapt a large model to a task cheaply, in the LoRA paperClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs.