RAG Architecture Explained: Chunking, Embeddings, Retrieval
Introduction
Retrieval-augmented generation (RAG) lets a large language model answer from your own documents - a company's policies, a product's manuals, a code base's wiki - instead of only from what it learned in training. The idea comes from the RAG paperClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation of 2020; this post is about how a RAG system is built today. It explains the RAG architecture step by step, with diagrams: the ingestion pipeline which prepares the documents, chunking and its effect on answers, which we will measure with a Java program, embeddings and the vector index, the retrieval step and how to improve it, and lastly how the retrieved text is put into the prompt and turned into an answer. It builds on the four concepts from tokens, context windows, embeddings and hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations.(Note - This post is framework-neutral. Spring AI and LangChain4j provide ready-made components for every step shown here.)
Two pipelines
A RAG system has two parts which run at different times. The ingestion pipeline runs ahead of time, whenever documents are added or changed. The query pipeline runs for every question -The architecture of a RAG system. Steps 1-4 prepare the documents, steps 5-9 answer a question.
Loading documents
Ingestion starts by turning each source into clean text: extracting the text from PDFs and HTML, dropping navigation menus, headers, footers and boilerplate, and keeping the structure that carries meaning - headings, lists, tables. Each document also gets metadata: where it came from, when it was last changed, which product or department it belongs to, and who may see it. Metadata is copied onto every chunk, so that answers can cite their sources and searches can be filtered later. Ingestion runs again whenever a document changes; the chunks of the old version must be deleted, or the system will retrieve outdated text.Chunking
Documents are split into chunks - passages of typically a few hundred tokens - before they are embedded, for three reasons -- An embedding of a whole 50-page manual blurs all its topics into one vector; an embedding of one section stays specific to that section, so searches can find it.
- Only the relevant passages should go into the prompt, to save tokens and to keep the model focused - see the context windowClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations.
- Embedding models accept a limited number of tokensClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java per text.
| Strategy | How it splits | Trade-off |
|---|---|---|
| Fixed size | Every N tokens or words | Simple and predictable, but cuts sentences and rules in half |
| Fixed size with overlap | Every N tokens, each chunk repeating the last M tokens of the previous one | Text near a boundary appears in two chunks; costs more storage |
| By structure | By sentences, paragraphs or sections, merged up to a size limit | Keeps units of meaning together; chunk sizes vary |
| Semantic | Where the topic changes, detected by comparing the embeddings of neighbouring sentences | Follows the meaning; slower to ingest |
How the chunking strategy changes what is retrieved
//ChunkingDemo.java
import java.util.*;
import java.util.stream.*;
public class ChunkingDemo {
static final String DOCUMENT = "Leave policy. Every employee gets 24 days of paid leave per calendar year. "
+ "Leave must be requested in the HR portal at least two weeks in advance. Unused leave does not expire "
+ "at once: up to 5 unused days can be carried over into the next year, and they must be used by "
+ "the end of March. Sick leave is separate and is not counted against paid leave.";
// Fixed-size chunking by words: each chunk has `size` words, and starts `size - overlap` words after the previous one.
static List<String> chunk(String text, int size, int overlap) {
String[] words = text.split("\\s+");
List<String> chunks = new ArrayList<>();
for (int start = 0; start < words.length; start += size - overlap) {
chunks.add(String.join(" ", Arrays.copyOfRange(words, start, Math.min(start + size, words.length))));
if (start + size >= words.length) break;
}
return chunks;
}
// Sentence-based chunking: one chunk per sentence, so a rule is never cut in half.
static List<String> chunkBySentence(String text) {
return List.of(text.split("(?<=\\.)\\s+"));
}
static final Set<String> STOP = Set.of("the", "a", "an", "is", "of", "to", "in", "and", "it", "at", "be", "by",
"can", "does", "not", "how", "many", "i", "my", "into", "they", "are", "per", "must", "for", "on", "as");
// A stand-in for an embedding model: word counts without common words (a real system calls an embedding model).
static Map<String, Integer> embed(String text) {
return Arrays.stream(text.toLowerCase().split("[^a-z0-9]+"))
.filter(w -> !w.isBlank() && !STOP.contains(w))
.collect(Collectors.toMap(w -> w, w -> 1, Integer::sum));
}
static double cosine(Map<String, Integer> a, Map<String, Integer> b) {
double dot = 0;
for (var e : a.entrySet()) dot += e.getValue() * b.getOrDefault(e.getKey(), 0);
double na = Math.sqrt(a.values().stream().mapToDouble(v -> v * v).sum());
double nb = Math.sqrt(b.values().stream().mapToDouble(v -> v * v).sum());
return (na == 0 || nb == 0) ? 0 : dot / (na * nb);
}
static void retrieve(String label, List<String> chunks, String question, double minScore) {
Map<String, Integer> q = embed(question);
System.out.println(label + " - " + chunks.size() + " chunks");
IntStream.range(0, chunks.size()).boxed()
.sorted(Comparator.comparingDouble((Integer i) -> cosine(q, embed(chunks.get(i)))).reversed())
.forEach(i -> {
double score = cosine(q, embed(chunks.get(i)));
System.out.printf(" %s %.3f [%d] %s%n", score >= minScore ? "KEEP" : "drop", score, i, chunks.get(i));
});
System.out.println();
}
public static void main(String[] args) {
String question = "How many unused leave days can be carried over into next year?";
retrieve("12 words per chunk, no overlap", chunk(DOCUMENT, 12, 0), question, 0.30);
retrieve("12 words per chunk, 4 words overlap", chunk(DOCUMENT, 12, 4), question, 0.30);
retrieve("One sentence per chunk", chunkBySentence(DOCUMENT), question, 0.30);
}
}
12 words per chunk, no overlap - 6 chunks KEEP 0.857 [3] 5 unused days can be carried over into the next year, and KEEP 0.327 [0] Leave policy. Every employee gets 24 days of paid leave per calendar drop 0.286 [1] year. Leave must be requested in the HR portal at least two drop 0.286 [2] weeks in advance. Unused leave does not expire at once: up to drop 0.169 [4] they must be used by the end of March. Sick leave is drop 0.169 [5] separate and is not counted against paid leave. 12 words per chunk, 4 words overlap - 8 chunks KEEP 0.676 [5] be carried over into the next year, and they must be used KEEP 0.571 [4] at once: up to 5 unused days can be carried over into KEEP 0.378 [1] paid leave per calendar year. Leave must be requested in the HR KEEP 0.327 [0] Leave policy. Every employee gets 24 days of paid leave per calendar drop 0.286 [3] weeks in advance. Unused leave does not expire at once: up to drop 0.239 [7] March. Sick leave is separate and is not counted against paid leave. drop 0.169 [6] they must be used by the end of March. Sick leave is drop 0.134 [2] requested in the HR portal at least two weeks in advance. Unused One sentence per chunk - 5 chunks KEEP 0.733 [3] Unused leave does not expire at once: up to 5 unused days can be carried over into the next year, and they must be used by the end of March. KEEP 0.378 [1] Every employee gets 24 days of paid leave per calendar year. drop 0.267 [0] Leave policy. drop 0.252 [4] Sick leave is separate and is not counted against paid leave. drop 0.134 [2] Leave must be requested in the HR portal at least two weeks in advance.
Explanation of the code
The answer to the question is one rule: up to 5 days, carried over, used by the end of March. The diagram shows where each strategy put its boundaries in that rule -The same rule in three chunking strategies, as produced by the program.
- No overlap: the best chunk scored 0.857, but it starts with "5 unused days" - the words "up to" ended up in the previous chunk, which was dropped. An LLM given this chunk could answer "5 days" as a fixed number, and would not know about the March deadline.
- With overlap: "up to 5" is now in one chunk, and four chunks pass the threshold. The deadline is still cut off, because the rule is longer than one chunk.
- One sentence per chunk: the whole rule is one chunk, with the highest score of its run, and only one other chunk passes the threshold. The model gets the complete rule and little else.
- The minimum score kept unrelated chunks, such as the one about sick leave, out of the prompt. Without it, a fixed top-k always fills the prompt, even when nothing relevant exists.
Embeddings and the vector index
Each chunk is turned into an embeddingClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations by an embedding model, and stored in a vector store with its text and metadata. Three rules keep this step correct -- One model for both sides: the question must be embedded with the same model as the chunks. Vectors from different models live in different spaces and cannot be compared. Use the similarity measure the model was trained with - see cosine similarity vs dot productClick to read tutorial: Cosine Similarity vs Dot Product for Embeddings.
- Changing the model means re-embedding everything: a new embedding model, or a new version of one, requires re-running ingestion for all documents.
- Keep the text next to the vector: the vector is only used to find the chunk; the LLM is given the chunk's text.
Retrieval and re-ranking
A plain similarity search returns the top k chunks by embedding similarity. Production systems usually add some of these steps, from cheap to expensive -Retrieval as a funnel: each stage looks at fewer chunks, more carefully.
- Metadata filters restrict the search before it starts - by document type, date, product, or the user's access rights. Filtering by access rights is essential: a RAG system must never put a document into a user's prompt which that user may not read.
- Hybrid search runs a keyword search such as BM25 next to the vector search and merges the two lists. Embeddings are good at meaning but weak at exact terms - error codes, product names, class names such as
ConcurrentHashMap- which keyword search finds reliably. How the lists are merged is shown below. - Re-ranking scores the candidates again with a more precise but slower model, a cross-encoder, also shown below.
- Query rewriting lets an LLM turn a vague or conversational question ("and what about the second one?") into a complete search query before retrieval. Going one step further, an agent can decide when and what to search, as in the ReAct paperClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs.
- A minimum score, as in the program above, keeps irrelevant chunks out of the prompt.
Reciprocal rank fusion with k = 60, the constant most systems use.
Why vector search and re-ranking use different kinds of model.
Augmentation: building the prompt
The retrieved chunks are put into the prompt together with the question and instructions which tell the model how to use them. A typical template -A prompt template for RAG
You answer questions about the company's HR policies.
Use only the sources below. If they do not contain the answer, say that you do not know.
Cite the sources you used as [1], [2], ...
[1] (hr/leave-policy.pdf, updated 2026-01-15)
Unused leave does not expire at once: up to 5 unused days can be carried over into the next year,
and they must be used by the end of March.
[2] (hr/leave-policy.pdf, updated 2026-01-15)
Every employee gets 24 days of paid leave per calendar year.
Question: How many unused leave days can be carried over into next year?
Generation and checking the result
The LLM then writes the answer from the prompt, usually at a low temperatureClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained, since there is one right answer to find. For the template above, a good answer would be "Up to 5 unused days can be carried over, and they must be used by the end of March [1]." Before showing an answer, an application can check that the cited source numbers exist, and log the question, the retrieved chunks and the answer together - when an answer is wrong, these logs show whether retrieval or generation failed.To improve a RAG system, measure the two halves separately on a set of real questions with known answers -
Measuring retrieval and generation separately tells you which half to fix.
- Retrieval quality: is the chunk that contains the answer among the retrieved chunks? This measure, often called recall@k, needs no LLM and is cheap to run after every change to chunking, embeddings or re-ranking.
- Answer quality: is the answer correct, and is every statement in it supported by the retrieved chunks (faithfulness)? This is usually judged by people, or by a second LLM with a checklist.
Summary
In this post we went through the architecture of a RAG system. We saw its two pipelines - ingestion, which loads, chunks, embeds and stores documents ahead of time, and the query pipeline, which embeds the question, retrieves, re-ranks, builds the prompt and generates the answer. With a Java program we saw how chunk boundaries decide what the model gets to read: fixed-size chunks lost part of a rule, overlap helped, and sentence-based chunks kept the rule whole. We went through embeddings and approximate nearest neighbour indexes, the retrieval funnel of metadata filters, hybrid search and re-ranking, a prompt template which grounds the answer and asks for sources, and how to measure retrieval and answers separately. The research behind the idea is explained in the RAG paper postClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation.Continue Reading: AI and LLM Papers Explained
pgvector vs Chroma vs Qdrant vs Milvus vs Redis ComparedClick to read tutorial: pgvector vs Chroma vs Qdrant vs Milvus vs Redis Compared LLM Tokens, Context Windows, Embeddings and HallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained
pgvector vs Chroma vs Qdrant vs Milvus vs Redis ComparedClick to read tutorial: pgvector vs Chroma vs Qdrant vs Milvus vs Redis Compared LLM Tokens, Context Windows, Embeddings and HallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained