RAG Architecture Explained: Chunking, Embeddings, Retrieval

Introduction

Retrieval-augmented generation (RAG) lets a large language model answer from your own documents - a company's policies, a product's manuals, a code base's wiki - instead of only from what it learned in training. The idea comes from the RAG paperClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation of 2020; this post is about how a RAG system is built today. It explains the RAG architecture step by step, with diagrams: the ingestion pipeline which prepares the documents, chunking and its effect on answers, which we will measure with a Java program, embeddings and the vector index, the retrieval step and how to improve it, and lastly how the retrieved text is put into the prompt and turned into an answer. It builds on the four concepts from tokens, context windows, embeddings and hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations.

(Note - This post is framework-neutral. Spring AI and LangChain4j provide ready-made components for every step shown here.)

Two pipelines

A RAG system has two parts which run at different times. The ingestion pipeline runs ahead of time, whenever documents are added or changed. The query pipeline runs for every question -
Ingestion: load documents, split into chunks, embed, store vectors with text and metadata. Query: embed the question, retrieve candidates, re-rank, build the prompt, generate the answer with sources. Ingestion pipeline - ahead of time 1. LoadPDF, HTML, wiki 2. Chunksplit into passages 3. Embedembedding model 4. Vector storevector+ chunk text+ metadata (source,date, access rights) Query pipeline - for every question Questionfrom the user 5. Embedsame model 6. Retrievetop candidates 7. Re-rankand filter 8. Promptquestion + chunks 9. LLManswer + sources similarity search The quality of the answer is decided mostly in steps 2, 6 and 7: the LLM can only use what retrieval puts in front of it.
The architecture of a RAG system. Steps 1-4 prepare the documents, steps 5-9 answer a question.
The rest of this post goes through these steps in order. The most important point is at the bottom of the diagram: a RAG system is a search system with an LLM at the end. If the right passage is not retrieved, no model can give the right answer, so most of the work of a good RAG system goes into the steps before the LLM.

Loading documents

Ingestion starts by turning each source into clean text: extracting the text from PDFs and HTML, dropping navigation menus, headers, footers and boilerplate, and keeping the structure that carries meaning - headings, lists, tables. Each document also gets metadata: where it came from, when it was last changed, which product or department it belongs to, and who may see it. Metadata is copied onto every chunk, so that answers can cite their sources and searches can be filtered later. Ingestion runs again whenever a document changes; the chunks of the old version must be deleted, or the system will retrieve outdated text.

Chunking

Documents are split into chunks - passages of typically a few hundred tokens - before they are embedded, for three reasons - Where the chunk boundaries fall matters as much as the chunk size. The common strategies are -
StrategyHow it splitsTrade-off
Fixed sizeEvery N tokens or wordsSimple and predictable, but cuts sentences and rules in half
Fixed size with overlapEvery N tokens, each chunk repeating the last M tokens of the previous oneText near a boundary appears in two chunks; costs more storage
By structureBy sentences, paragraphs or sections, merged up to a size limitKeeps units of meaning together; chunk sizes vary
SemanticWhere the topic changes, detected by comparing the embeddings of neighbouring sentencesFollows the meaning; slower to ingest
The program below chunks a short leave policy in three ways and retrieves the chunks for one question, keeping only those with a similarity score of at least 0.30. As in the RAG paper postClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation, word counts stand in for an embedding model, so that it runs without any API.
How the chunking strategy changes what is retrieved
//ChunkingDemo.java
import java.util.*;
import java.util.stream.*;

public class ChunkingDemo {

  static final String DOCUMENT = "Leave policy. Every employee gets 24 days of paid leave per calendar year. "
      + "Leave must be requested in the HR portal at least two weeks in advance. Unused leave does not expire "
      + "at once: up to 5 unused days can be carried over into the next year, and they must be used by "
      + "the end of March. Sick leave is separate and is not counted against paid leave.";

  // Fixed-size chunking by words: each chunk has `size` words, and starts `size - overlap` words after the previous one.
  static List<String> chunk(String text, int size, int overlap) {
    String[] words = text.split("\\s+");
    List<String> chunks = new ArrayList<>();
    for (int start = 0; start < words.length; start += size - overlap) {
      chunks.add(String.join(" ", Arrays.copyOfRange(words, start, Math.min(start + size, words.length))));
      if (start + size >= words.length) break;
    }
    return chunks;
  }

  // Sentence-based chunking: one chunk per sentence, so a rule is never cut in half.
  static List<String> chunkBySentence(String text) {
    return List.of(text.split("(?<=\\.)\\s+"));
  }

  static final Set<String> STOP = Set.of("the", "a", "an", "is", "of", "to", "in", "and", "it", "at", "be", "by",
      "can", "does", "not", "how", "many", "i", "my", "into", "they", "are", "per", "must", "for", "on", "as");

  // A stand-in for an embedding model: word counts without common words (a real system calls an embedding model).
  static Map<String, Integer> embed(String text) {
    return Arrays.stream(text.toLowerCase().split("[^a-z0-9]+"))
        .filter(w -> !w.isBlank() && !STOP.contains(w))
        .collect(Collectors.toMap(w -> w, w -> 1, Integer::sum));
  }

  static double cosine(Map<String, Integer> a, Map<String, Integer> b) {
    double dot = 0;
    for (var e : a.entrySet()) dot += e.getValue() * b.getOrDefault(e.getKey(), 0);
    double na = Math.sqrt(a.values().stream().mapToDouble(v -> v * v).sum());
    double nb = Math.sqrt(b.values().stream().mapToDouble(v -> v * v).sum());
    return (na == 0 || nb == 0) ? 0 : dot / (na * nb);
  }

  static void retrieve(String label, List<String> chunks, String question, double minScore) {
    Map<String, Integer> q = embed(question);
    System.out.println(label + " - " + chunks.size() + " chunks");
    IntStream.range(0, chunks.size()).boxed()
        .sorted(Comparator.comparingDouble((Integer i) -> cosine(q, embed(chunks.get(i)))).reversed())
        .forEach(i -> {
          double score = cosine(q, embed(chunks.get(i)));
          System.out.printf("  %s %.3f  [%d] %s%n", score >= minScore ? "KEEP" : "drop", score, i, chunks.get(i));
        });
    System.out.println();
  }

  public static void main(String[] args) {
    String question = "How many unused leave days can be carried over into next year?";
    retrieve("12 words per chunk, no overlap", chunk(DOCUMENT, 12, 0), question, 0.30);
    retrieve("12 words per chunk, 4 words overlap", chunk(DOCUMENT, 12, 4), question, 0.30);
    retrieve("One sentence per chunk", chunkBySentence(DOCUMENT), question, 0.30);
  }
}
 OUTPUT of the above code
12 words per chunk, no overlap - 6 chunks
  KEEP 0.857  [3] 5 unused days can be carried over into the next year, and
  KEEP 0.327  [0] Leave policy. Every employee gets 24 days of paid leave per calendar
  drop 0.286  [1] year. Leave must be requested in the HR portal at least two
  drop 0.286  [2] weeks in advance. Unused leave does not expire at once: up to
  drop 0.169  [4] they must be used by the end of March. Sick leave is
  drop 0.169  [5] separate and is not counted against paid leave.

12 words per chunk, 4 words overlap - 8 chunks
  KEEP 0.676  [5] be carried over into the next year, and they must be used
  KEEP 0.571  [4] at once: up to 5 unused days can be carried over into
  KEEP 0.378  [1] paid leave per calendar year. Leave must be requested in the HR
  KEEP 0.327  [0] Leave policy. Every employee gets 24 days of paid leave per calendar
  drop 0.286  [3] weeks in advance. Unused leave does not expire at once: up to
  drop 0.239  [7] March. Sick leave is separate and is not counted against paid leave.
  drop 0.169  [6] they must be used by the end of March. Sick leave is
  drop 0.134  [2] requested in the HR portal at least two weeks in advance. Unused

One sentence per chunk - 5 chunks
  KEEP 0.733  [3] Unused leave does not expire at once: up to 5 unused days can be carried over into the next year, and they must be used by the end of March.
  KEEP 0.378  [1] Every employee gets 24 days of paid leave per calendar year.
  drop 0.267  [0] Leave policy.
  drop 0.252  [4] Sick leave is separate and is not counted against paid leave.
  drop 0.134  [2] Leave must be requested in the HR portal at least two weeks in advance.

Explanation of the code

The answer to the question is one rule: up to 5 days, carried over, used by the end of March. The diagram shows where each strategy put its boundaries in that rule -
Without overlap the rule is cut before "5 unused days" and before "they must be used"; with overlap "up to 5" stays together but the deadline is still cut off; one sentence per chunk keeps the whole rule The rule: "... up to 5 unused days can be carried over into the next year, and they must be used by the end of March." No overlap ... up to 5 unused days can be carried over into the next year, and they must ... best chunk has lost "up to" and the deadline 4 words overlap at once: up to 5 unused days can be carried over into be carried over into the next year, and they must be used "up to 5" kept together; "end of March" still in a third chunk One sentence Unused leave does not expire at once: up to 5 ... carried over ... used by the end of March.
The same rule in three chunking strategies, as produced by the program.
  • No overlap: the best chunk scored 0.857, but it starts with "5 unused days" - the words "up to" ended up in the previous chunk, which was dropped. An LLM given this chunk could answer "5 days" as a fixed number, and would not know about the March deadline.
  • With overlap: "up to 5" is now in one chunk, and four chunks pass the threshold. The deadline is still cut off, because the rule is longer than one chunk.
  • One sentence per chunk: the whole rule is one chunk, with the highest score of its run, and only one other chunk passes the threshold. The model gets the complete rule and little else.
  • The minimum score kept unrelated chunks, such as the one about sick leave, out of the prompt. Without it, a fixed top-k always fills the prompt, even when nothing relevant exists.
In real systems, chunks of a few hundred tokens split by structure - sections, then paragraphs, then sentences - with a small overlap, are a good default. The best size depends on the documents and the questions, and is worth measuring, as above, on real questions.

Embeddings and the vector index

Each chunk is turned into an embeddingClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations by an embedding model, and stored in a vector store with its text and metadata. Three rules keep this step correct -
  • One model for both sides: the question must be embedded with the same model as the chunks. Vectors from different models live in different spaces and cannot be compared. Use the similarity measure the model was trained with - see cosine similarity vs dot productClick to read tutorial: Cosine Similarity vs Dot Product for Embeddings.
  • Changing the model means re-embedding everything: a new embedding model, or a new version of one, requires re-running ingestion for all documents.
  • Keep the text next to the vector: the vector is only used to find the chunk; the LLM is given the chunk's text.
Comparing a question with every stored vector is exact but slow for millions of chunks. Vector stores therefore use an approximate nearest neighbour (ANN) index, most often HNSW (Hierarchical Navigable Small World), a graph through which a search jumps quickly to the region of similar vectors. It occasionally misses a true nearest neighbour, in exchange for answers in milliseconds. How the common vector stores differ - in indexes, filtering, scale and licences - is compared in pgvector vs Chroma vs Qdrant vs Milvus vs RedisClick to read tutorial: pgvector vs Chroma vs Qdrant vs Milvus vs Redis Compared.

Retrieval and re-ranking

A plain similarity search returns the top k chunks by embedding similarity. Production systems usually add some of these steps, from cheap to expensive -
Metadata filters narrow the search, hybrid search finds about fifty candidates by meaning and by keywords, a re-ranker scores them precisely, and the best five go into the prompt Metadata filteronly HR documents, current versions, ones this user may see Hybrid search → about 50 candidatesvector similarity + keyword search (BM25), merged Re-ranker → best 5reads question and chunk together Into the prompt fast,broad slow,precise
Retrieval as a funnel: each stage looks at fewer chunks, more carefully.
  • Metadata filters restrict the search before it starts - by document type, date, product, or the user's access rights. Filtering by access rights is essential: a RAG system must never put a document into a user's prompt which that user may not read.
  • Hybrid search runs a keyword search such as BM25 next to the vector search and merges the two lists. Embeddings are good at meaning but weak at exact terms - error codes, product names, class names such as ConcurrentHashMap - which keyword search finds reliably. How the lists are merged is shown below.
  • Re-ranking scores the candidates again with a more precise but slower model, a cross-encoder, also shown below.
  • Query rewriting lets an LLM turn a vague or conversational question ("and what about the second one?") into a complete search query before retrieval. Going one step further, an agent can decide when and what to search, as in the ReAct paperClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs.
  • A minimum score, as in the program above, keeps irrelevant chunks out of the prompt.
The scores of a keyword search and a vector search are on different scales and cannot simply be added. Reciprocal rank fusion (RRF) ignores the scores and uses only the positions: each list gives a chunk 1 / (60 + rank), and the chunk's points from both lists are added up -
Keyword search ranks C, A, D; vector search ranks A, B, C; reciprocal rank fusion gives A 0.0325, C 0.0323, B 0.0161 and D 0.0159, so A and C, found by both, come first Keyword search (BM25) 1. chunk C1/61 2. chunk A1/62 3. chunk D1/63 + 1. chunk A1/61 2. chunk B1/62 3. chunk C1/63 Vector search Fused ranking 1. chunk A1/62 + 1/61 = 0.0325 2. chunk C1/61 + 1/63 = 0.0323 3. chunk B1/62 = 0.0161 4. chunk D1/63 = 0.0159 Chunks found by both searches rise to the top; a chunk only one search found still gets a place.
Reciprocal rank fusion with k = 60, the constant most systems use.
Re-ranking then looks at the fused candidates more closely. The vector search used a bi-encoder: the question and each chunk were turned into vectors separately, and only the vectors were compared. A cross-encoder, usually a BERTClick to read tutorial: BERT Paper Explained: Bidirectional Transformers-style model, reads the question and a chunk together, so every word of the question can be compared with every word of the chunk -
A bi-encoder embeds the question and the chunk separately and compares the two vectors, so chunk vectors can be computed ahead of time; a cross-encoder reads both texts together and outputs a relevance score, which is more precise but must run for every pair Bi-encoder (vector search) question chunk encoder encoder cosine → score chunk vectors computed once, ahead of time: fast over millions Cross-encoder (re-ranking) question + chunk, read together one Transformer relevance score once per question-chunk pair: precise, but slow
Why vector search and re-ranking use different kinds of model.
This is why the two are combined: the bi-encoder is fast enough to search the whole index, and the cross-encoder is precise enough to pick the best few out of fifty, but far too slow to run over every chunk.

Augmentation: building the prompt

The retrieved chunks are put into the prompt together with the question and instructions which tell the model how to use them. A typical template -
A prompt template for RAG
You answer questions about the company's HR policies.
Use only the sources below. If they do not contain the answer, say that you do not know.
Cite the sources you used as [1], [2], ...

[1] (hr/leave-policy.pdf, updated 2026-01-15)
Unused leave does not expire at once: up to 5 unused days can be carried over into the next year,
and they must be used by the end of March.

[2] (hr/leave-policy.pdf, updated 2026-01-15)
Every employee gets 24 days of paid leave per calendar year.

Question: How many unused leave days can be carried over into next year?
It uses several prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples at once. Each part has a job: "use only the sources" grounds the answer, "say that you do not know" gives the model a way out instead of inventing an answer - the main defence against hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations - and the numbered sources with their metadata let the answer cite them, so that a reader can check it. The retrieved chunks must also fit the token budget: the system prompt, the chunks, any chat history and the room for the answer all share one context window.

Generation and checking the result

The LLM then writes the answer from the prompt, usually at a low temperatureClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained, since there is one right answer to find. For the template above, a good answer would be "Up to 5 unused days can be carried over, and they must be used by the end of March [1]." Before showing an answer, an application can check that the cited source numbers exist, and log the question, the retrieved chunks and the answer together - when an answer is wrong, these logs show whether retrieval or generation failed.

To improve a RAG system, measure the two halves separately on a set of real questions with known answers -
Recall at k is measured on the retrieved chunks against the known source chunk; correctness and faithfulness are measured on the answer against the known answer and the retrieved chunks Test questionwith known answer Retrieval→ top k chunks Generation→ answer recall@kis the source chunk amongthe k retrieved? no LLM needed correctness and faithfulnessmatches the known answer? every statementbacked by the chunks? judged by people or an LLM A wrong answer with low recall@k is a retrieval problem; with high recall@k, a generation problem.
Measuring retrieval and generation separately tells you which half to fix.
  • Retrieval quality: is the chunk that contains the answer among the retrieved chunks? This measure, often called recall@k, needs no LLM and is cheap to run after every change to chunking, embeddings or re-ranking.
  • Answer quality: is the answer correct, and is every statement in it supported by the retrieved chunks (faithfulness)? This is usually judged by people, or by a second LLM with a checklist.

Summary

In this post we went through the architecture of a RAG system. We saw its two pipelines - ingestion, which loads, chunks, embeds and stores documents ahead of time, and the query pipeline, which embeds the question, retrieves, re-ranks, builds the prompt and generates the answer. With a Java program we saw how chunk boundaries decide what the model gets to read: fixed-size chunks lost part of a rule, overlap helped, and sentence-based chunks kept the rule whole. We went through embeddings and approximate nearest neighbour indexes, the retrieval funnel of metadata filters, hybrid search and re-ranking, a prompt template which grounds the answer and asks for sources, and how to measure retrieval and answers separately. The research behind the idea is explained in the RAG paper postClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation.