Fine-Tuning vs RAG vs Prompt Engineering: Which to Use
Introduction
A large language model gives an answer which is too vague, in the wrong format, or simply wrong about your company. There are three standard ways to fix it, and online advice often makes them sound like rivals: improve the prompt, add retrieval-augmented generation (RAG), or fine-tune the model. They are not rivals. Each changes a different thing - what the model is told, what it can read, and what it is - and so each solves a different kind of problem. Picking the wrong one is the usual reason an LLM project spends weeks and does not improve. This post explains when to use prompt engineering, RAG and fine-tuning with diagrams: what each one changes, the order in which to try them, a decision flow and table, a Java calculator which shows when fine-tuning pays for itself, how to combine them, and the mistakes to avoid.(Note - The three techniques are explained one by one in prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples, RAG architecture explainedClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval and the LoRA paper postClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs. This post is about choosing between them.)
Three levers, three different jobs
A model's answer depends on two things: the model itself, with the weights it learned in training, and the text it is given in its context window on every request. The three techniques act on those two places -The three techniques act on different places: two on the context window, one on the weights.
- Prompt engineering changes the instructions, examples and format in the context. It is free to try, instantly reversible, and costs tokens on every request.
- RAG puts documents found for this question into the context. The model's weights do not change; what it can use does, per request.
- Fine-tuning continues training the model on your own examples, so the weights change. What it learns well is behaviour: a style, a format, a vocabulary, a skill.
Start with the prompt and a test set
Prompting comes first for three reasons. It needs no infrastructure and no training data. Each change takes minutes to try, not days. And it shows what is really wrong: if a clear instruction and three good examples fix the problem, nothing else was needed. The GPT-3 paperClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners showed how far examples in the prompt can go, and chain-of-thought promptingClick to read tutorial: Chain-of-Thought Prompting Paper Explained how far a changed instruction can.Before changing anything, build a test set: 30 to 100 real inputs with the answers you would accept, and a way to score them. Run it on the first prompt, and again after every change. Without it, "the answers feel better" is all you have, and the three techniques cannot be compared - or even told apart from luck. The test set is also what tells you when prompting has run out. The signs are -
- the prompt keeps growing, and fixing one case breaks another,
- the format is right most of the time, but not often enough for code that parses it,
- the examples and instructions make every request expensive and slow, or
- the answers are well written but contain facts which are missing from the prompt.
When the missing piece is knowledge: RAG
A model knows only what was in its training data, up to a cutoff date. It has never seen your internal documents, your product's latest release, or this morning's prices - and when asked, it tends to produce a plausible answer anyway, which is a hallucinationClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations. RAG fixes this where the problem is: it finds the relevant text and puts it in front of the model, as the RAG paperClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation proposed. Its strengths are exactly what knowledge needs -- Fresh: change a document and the next answer changes. Nothing is retrained.
- Traceable: each answer can cite the chunks it used, so a reader can check it.
- Controlled: the search can filter by who is allowed to read what. The weights of a model cannot enforce permissions; a retrieval filter can.
- Cheap to start: an embedding model, a vector storeClick to read tutorial: pgvector vs Chroma vs Qdrant vs Milvus vs Redis Compared and a prompt, as in RAG architecture explainedClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval.
When the missing piece is behaviour: fine-tuning
Fine-tuning trains a model further on examples of the input and the output you want, so that the behaviour is built in instead of requested in every prompt. It is the same supervised step which began the training of InstructGPT, described in the InstructGPT paper postClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF, and with LoRAClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs it needs only a small number of extra weights. It is the right tool when -- Style and tone must be consistent: a brand voice, the way your support team writes, a fixed report layout.
- The format must be reliable: a particular JSON shape, a house markup, a domain's notation.
- A narrow task runs at high volume: classifying, extracting or routing millions of items, where a small fine-tuned model can replace a large prompted one - cheaper and faster, as in knowledge distillationClick to read tutorial: Knowledge Distillation Explained: Teacher and Student Models.
- The prompt has grown too long: examples and rules which are sent with every request can be moved into the weights.
- The domain has its own language or skill: a specialist vocabulary, or a way of reasoning the base model handles poorly.
Choosing: a decision flow
Put together, the order is almost always the same: prompt first, then find out what is wrong, then pick the lever which matches it -Prompt first, then match the lever to the problem.
| The problem | First move | Why |
|---|---|---|
| Answers need private, recent or very detailed facts | RAG | The facts live in documents which can be updated and cited |
| Users must see where an answer came from | RAG | Retrieved chunks carry their source with them |
| Different users may read different documents | RAG with access filters | Permissions are enforced at retrieval; weights cannot enforce them |
| The knowledge changes daily | RAG | Re-index the changed documents; no retraining |
| The model ignores the format or tone you ask for | A clearer prompt with examples, then fine-tuning | Examples show a format; fine-tuning makes it stick |
| A narrow task such as classifying or extracting, at high volume | Fine-tune a small model | Cheaper and faster than a large model with a long prompt |
| Thousands of requests a day share a long fixed prompt | Fine-tuning, or prompt caching | Fine-tuning moves the fixed text into the weights; prompt caching makes the provider charge less for a prefix it has seen before |
| You have no labelled examples yet | Prompting and RAG | Fine-tuning needs hundreds to thousands of good examples |
What it costs: a calculator in Java
Cost is rarely the first question, but it often decides the second. Every approach changes how many tokens each request sends, and fine-tuning adds a one-time cost and sometimes a higher price per token. The program below compares four ways of building a support-ticket classifier. The prices and sizes are illustrative assumptions - replace them with your own provider's prices and your own prompt sizes. A hosted fine-tuned model is assumed to cost twice as much per token as the base model, as some providers charge.What each approach costs per request, per year, and when fine-tuning pays off
//ApproachCosts.java
public class ApproachCosts {
// One way of building the feature. Prices are dollars per million tokens.
record Approach(String name, int inputTokens, int outputTokens, double inputPrice, double outputPrice,
double oneTimeCost, double monthlyCost) {
double perRequest() {
return (inputTokens * inputPrice + outputTokens * outputPrice) / 1_000_000;
}
// Total cost of the first 12 months at a given number of requests per month
double firstYear(long requestsPerMonth) {
return oneTimeCost + 12 * monthlyCost + 12 * requestsPerMonth * perRequest();
}
}
public static void main(String[] args) {
// Illustrative numbers for a support-ticket classifier. Replace them with your own prices and prompt sizes.
// Every request carries 100 tokens of ticket text and gets a 30-token answer.
Approach[] approaches = {
new Approach("Zero-shot prompt (short instructions)", 300 + 100, 30, 0.50, 1.50, 0, 0),
new Approach("Few-shot prompt (20 examples, 2,000 tokens)", 300 + 2_000 + 100, 30, 0.50, 1.50, 0, 0),
new Approach("RAG (3 retrieved chunks, 1,200 tokens)", 300 + 1_200 + 100, 30, 0.50, 1.50, 300, 100),
// A fine-tuned model has learned the examples, so its prompt is short, but its tokens cost more here
new Approach("Fine-tuned model (short prompt)", 100 + 100, 30, 1.00, 3.00, 1_000, 0),
};
System.out.printf("%-46s %s%n", "Approach", "cost per 1,000 requests");
for (Approach a : approaches)
System.out.printf("%-46s $%.3f%n", a.name(), a.perRequest() * 1000);
System.out.printf("%n%-46s %12s %12s %12s%n", "First-year cost at requests per month:", "10,000", "100,000", "1,000,000");
for (Approach a : approaches)
System.out.printf("%-46s %12s %12s %12s%n", a.name(),
String.format("$%,.0f", a.firstYear(10_000)), String.format("$%,.0f", a.firstYear(100_000)),
String.format("$%,.0f", a.firstYear(1_000_000)));
// When does fine-tuning pay for itself against the few-shot prompt?
Approach fewShot = approaches[1], tuned = approaches[3];
double savedPerRequest = fewShot.perRequest() - tuned.perRequest();
double breakEven = (tuned.oneTimeCost() - fewShot.oneTimeCost()) / savedPerRequest;
System.out.printf("%nThe fine-tuned model saves $%.6f per request against the few-shot prompt.%n", savedPerRequest);
System.out.printf("Its $%,.0f setup cost is repaid after %,.0f requests (about %,.0f a month for a year).%n",
tuned.oneTimeCost(), breakEven, breakEven / 12);
}
}
Approach cost per 1,000 requests Zero-shot prompt (short instructions) $0.245 Few-shot prompt (20 examples, 2,000 tokens) $1.245 RAG (3 retrieved chunks, 1,200 tokens) $0.845 Fine-tuned model (short prompt) $0.290 First-year cost at requests per month: 10,000 100,000 1,000,000 Zero-shot prompt (short instructions) $29 $294 $2,940 Few-shot prompt (20 examples, 2,000 tokens) $149 $1,494 $14,940 RAG (3 retrieved chunks, 1,200 tokens) $1,601 $2,514 $11,640 Fine-tuned model (short prompt) $1,035 $1,348 $4,480 The fine-tuned model saves $0.000955 per request against the few-shot prompt. Its $1,000 setup cost is repaid after 1,047,120 requests (about 87,260 a month for a year).
Explanation of the code
- The few-shot prompt costs five times as much per request as the zero-shot prompt ($1.245 against $0.245 per 1,000 requests), because its 2,000 tokens of examples are sent again with every request. The fine-tuned model has learned those examples, so its prompt is short: even at twice the price per token, it costs $0.290 per 1,000 requests.
- At 10,000 requests a month, fine-tuning is the wrong choice on cost: $1,035 in the first year against $149 for the few-shot prompt. At 100,000 a month the two are close ($1,348 and $1,494), and at a million a month the fine-tuned model costs less than a third as much ($4,480 against $14,940).
- The break-even is simple division: the $1,000 setup cost divided by the $0.000955 saved on every request gives about a million requests, or 87,000 a month for a year. Cheaper base-model tokens or a shorter few-shot prompt push it further out; a longer few-shot prompt or a cheaper fine-tuned model brings it closer.
- The RAG row is there to show its price tag, not to compete: it costs more than a short prompt because of the retrieved text and the monthly cost of the vector store, and it solves a different problem - missing knowledge. The zero-shot prompt is the cheapest of all, so the real question is whether its answers are good enough, and only the test set can say.
- The numbers leave out the biggest cost of fine-tuning: the people's time to collect, clean and check the examples. Several providers also charge less for repeated prompt prefixes (prompt caching), which shrinks the gap for long few-shot prompts.
Cumulative cost, from the numbers in the program above. Below the crossing point the prompt is cheaper; above it, the fine-tuned model is.
Combining the three
The three levers do not exclude each other, and serious systems usually use more than one -- Prompt + RAG is the most common setup: instructions and output format in the prompt, facts from retrieval. It is where almost every RAG systemClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval starts.
- Fine-tuned model + RAG uses each for its own job: fine-tuning for the answer style, the citation format and the way to say "I don't know", RAG for the facts. The fine-tuning examples teach how to use retrieved text, not what it says.
- Fine-tuning the retrieval models is a third option: training the embedding model or the re-ranker on your own questions and documents, when retrieval - not the generating model - is the weak step.
- A small fine-tuned model for the narrow, high-volume step, and a large prompted model for the hard cases, is a common way to keep cost down.
Mistakes to avoid
- Fine-tuning to add facts. It is slow, unreliable and cannot be updated. Use RAG.
- Expecting a bigger or reasoning model to know your facts. A stronger model writes better, and a reasoning modelClick to read tutorial: Reasoning Models and Test-Time Compute Explained thinks longer, but neither has read your documents. Missing knowledge still needs RAG.
- Using RAG to fix style. Retrieved documents do not change how a model writes. Use examples in the prompt, or fine-tuning.
- Skipping the baseline. Without a test set and a first prompt, you cannot tell whether a technique helped, or whether it cost more than it saved.
- Blaming the model for bad retrieval. When a RAG system is wrong, first check whether the right chunk was retrieved; if it was not, no model could have answered correctly.
- Fine-tuning on messy examples. Inconsistent or wrong examples are learned too. A few hundred clean ones beat thousands of sloppy ones.
- Choosing by fashion. The cheapest technique which passes your test set is the right one, however unglamorous.
Summary
In this post we compared prompt engineering, RAG and fine-tuning. We saw that they act on different places - prompting and RAG on the context window, fine-tuning on the weights - so that RAG gives a model knowledge, fine-tuning changes its behaviour, and prompting steers both cheaply. We went through why to start with a prompt and a test set, when the problem is missing knowledge and RAG is the answer, and when it is behaviour and fine-tuning is, with a decision flow and a table of problems and first moves. In Java, we calculated what four approaches cost, and found that fine-tuning repaid its setup cost only after about a million requests. Lastly, we looked at combining the techniques and the mistakes to avoid. In the next post, we will look at models which are trained to think before they answer, in reasoning models and test-time computeClick to read tutorial: Reasoning Models and Test-Time Compute Explained.Continue Reading: AI and LLM Papers Explained
Reasoning Models and Test-Time Compute ExplainedClick to read tutorial: Reasoning Models and Test-Time Compute Explained Knowledge Distillation Explained: Teacher and Student ModelsClick to read tutorial: Knowledge Distillation Explained: Teacher and Student Models Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained
Reasoning Models and Test-Time Compute ExplainedClick to read tutorial: Reasoning Models and Test-Time Compute Explained Knowledge Distillation Explained: Teacher and Student ModelsClick to read tutorial: Knowledge Distillation Explained: Teacher and Student Models Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained