Fine-Tuning vs RAG vs Prompt Engineering: Which to Use

Introduction

A large language model gives an answer which is too vague, in the wrong format, or simply wrong about your company. There are three standard ways to fix it, and online advice often makes them sound like rivals: improve the prompt, add retrieval-augmented generation (RAG), or fine-tune the model. They are not rivals. Each changes a different thing - what the model is told, what it can read, and what it is - and so each solves a different kind of problem. Picking the wrong one is the usual reason an LLM project spends weeks and does not improve. This post explains when to use prompt engineering, RAG and fine-tuning with diagrams: what each one changes, the order in which to try them, a decision flow and table, a Java calculator which shows when fine-tuning pays for itself, how to combine them, and the mistakes to avoid.

(Note - The three techniques are explained one by one in prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples, RAG architecture explainedClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval and the LoRA paper postClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs. This post is about choosing between them.)

Three levers, three different jobs

A model's answer depends on two things: the model itself, with the weights it learned in training, and the text it is given in its context window on every request. The three techniques act on those two places -
Prompt engineering writes the instructions in the context window, RAG adds retrieved documents to the context window, and fine-tuning changes the model's weights Prompt engineering changes what the model is told RAG changes what the model can read Fine-tuning changes what the model is Context window - read again on every request Instructions, role, examples, output format Documents retrieved for this question The user's question Weights - learned in training the same for every request until they are trained again
The three techniques act on different places: two on the context window, one on the weights.
  • Prompt engineering changes the instructions, examples and format in the context. It is free to try, instantly reversible, and costs tokens on every request.
  • RAG puts documents found for this question into the context. The model's weights do not change; what it can use does, per request.
  • Fine-tuning continues training the model on your own examples, so the weights change. What it learns well is behaviour: a style, a format, a vocabulary, a skill.
The rule of thumb which follows is worth remembering: RAG gives a model knowledge, fine-tuning changes its behaviour, and prompt engineering steers both cheaply. Most disappointments come from using one for the other's job.

Start with the prompt and a test set

Prompting comes first for three reasons. It needs no infrastructure and no training data. Each change takes minutes to try, not days. And it shows what is really wrong: if a clear instruction and three good examples fix the problem, nothing else was needed. The GPT-3 paperClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners showed how far examples in the prompt can go, and chain-of-thought promptingClick to read tutorial: Chain-of-Thought Prompting Paper Explained how far a changed instruction can.

Before changing anything, build a test set: 30 to 100 real inputs with the answers you would accept, and a way to score them. Run it on the first prompt, and again after every change. Without it, "the answers feel better" is all you have, and the three techniques cannot be compared - or even told apart from luck. The test set is also what tells you when prompting has run out. The signs are -
  • the prompt keeps growing, and fixing one case breaks another,
  • the format is right most of the time, but not often enough for code that parses it,
  • the examples and instructions make every request expensive and slow, or
  • the answers are well written but contain facts which are missing from the prompt.
The first three are behaviour problems. The last is a knowledge problem, and it points somewhere else.

When the missing piece is knowledge: RAG

A model knows only what was in its training data, up to a cutoff date. It has never seen your internal documents, your product's latest release, or this morning's prices - and when asked, it tends to produce a plausible answer anyway, which is a hallucinationClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations. RAG fixes this where the problem is: it finds the relevant text and puts it in front of the model, as the RAG paperClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation proposed. Its strengths are exactly what knowledge needs - Why not teach the facts by fine-tuning instead? Because fine-tuning is a poor way to add knowledge. The model absorbs facts imperfectly and cannot say where they came from, and they cannot be updated or removed without training again. A 2024 study by Gekhman and colleagues went further: models learned new facts slowly during fine-tuning, and as they did, they became more likely to make things up. Retrieval keeps the facts outside the model, where they can be read, cited and replaced. RAG has its own price: the quality of the answers is capped by the quality of the retrieval, and every request carries the retrieved text as extra tokens.

When the missing piece is behaviour: fine-tuning

Fine-tuning trains a model further on examples of the input and the output you want, so that the behaviour is built in instead of requested in every prompt. It is the same supervised step which began the training of InstructGPT, described in the InstructGPT paper postClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF, and with LoRAClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs it needs only a small number of extra weights. It is the right tool when -
  • Style and tone must be consistent: a brand voice, the way your support team writes, a fixed report layout.
  • The format must be reliable: a particular JSON shape, a house markup, a domain's notation.
  • A narrow task runs at high volume: classifying, extracting or routing millions of items, where a small fine-tuned model can replace a large prompted one - cheaper and faster, as in knowledge distillationClick to read tutorial: Knowledge Distillation Explained: Teacher and Student Models.
  • The prompt has grown too long: examples and rules which are sent with every request can be moved into the weights.
  • The domain has its own language or skill: a specialist vocabulary, or a way of reasoning the base model handles poorly.
It also has real requirements. You need examples - from a few hundred to a few thousand good ones; the LIMA study (Zhou et al., 2023) tuned a 65-billion-parameter model on only 1,000 carefully chosen examples and got strong results, but quality matters far more than quantity. You need a test set to show it helped and did not break something else. You need a model you are allowed to train: an open-weight model which you fine-tune yourself, or a closed model through its provider's fine-tuning service - the difference is explained in open-weight vs closed-weight LLMsClick to read tutorial: Open-Weight vs Closed-Weight vs Open-Source LLMs Explained. You must pay for training and hosting, and repeat the work when you move to a newer base model. And fine-tuning on messy or contradictory examples teaches the mess.

Choosing: a decision flow

Put together, the order is almost always the same: prompt first, then find out what is wrong, then pick the lever which matches it -
Write a prompt and a test set; if the result is good enough, ship it; otherwise find what is wrong: missing knowledge means RAG, wrong behaviour means fine-tuning, both mean a fine-tuned model with RAG 1. Write a clear prompt and a test set of real inputs Good enough on the test set? Ship it 2. What is going wrong? Facts are missing, outdated, private or must be cited Style, format or skill is wrong, or the prompt is long and costly at your volume Both: the right format and the right facts are needed Add RAG Fine-tune (for example LoRA) Fine-tuned model + RAG yes no After every change, run the test set again and compare with the last run.
Prompt first, then match the lever to the problem.
The same choice as a table, by the problem you actually have -
The problemFirst moveWhy
Answers need private, recent or very detailed factsRAGThe facts live in documents which can be updated and cited
Users must see where an answer came fromRAGRetrieved chunks carry their source with them
Different users may read different documentsRAG with access filtersPermissions are enforced at retrieval; weights cannot enforce them
The knowledge changes dailyRAGRe-index the changed documents; no retraining
The model ignores the format or tone you ask forA clearer prompt with examples, then fine-tuningExamples show a format; fine-tuning makes it stick
A narrow task such as classifying or extracting, at high volumeFine-tune a small modelCheaper and faster than a large model with a long prompt
Thousands of requests a day share a long fixed promptFine-tuning, or prompt cachingFine-tuning moves the fixed text into the weights; prompt caching makes the provider charge less for a prefix it has seen before
You have no labelled examples yetPrompting and RAGFine-tuning needs hundreds to thousands of good examples

What it costs: a calculator in Java

Cost is rarely the first question, but it often decides the second. Every approach changes how many tokens each request sends, and fine-tuning adds a one-time cost and sometimes a higher price per token. The program below compares four ways of building a support-ticket classifier. The prices and sizes are illustrative assumptions - replace them with your own provider's prices and your own prompt sizes. A hosted fine-tuned model is assumed to cost twice as much per token as the base model, as some providers charge.
What each approach costs per request, per year, and when fine-tuning pays off
//ApproachCosts.java
public class ApproachCosts {

  // One way of building the feature. Prices are dollars per million tokens.
  record Approach(String name, int inputTokens, int outputTokens, double inputPrice, double outputPrice,
                  double oneTimeCost, double monthlyCost) {

    double perRequest() {
      return (inputTokens * inputPrice + outputTokens * outputPrice) / 1_000_000;
    }

    // Total cost of the first 12 months at a given number of requests per month
    double firstYear(long requestsPerMonth) {
      return oneTimeCost + 12 * monthlyCost + 12 * requestsPerMonth * perRequest();
    }
  }

  public static void main(String[] args) {
    // Illustrative numbers for a support-ticket classifier. Replace them with your own prices and prompt sizes.
    // Every request carries 100 tokens of ticket text and gets a 30-token answer.
    Approach[] approaches = {
        new Approach("Zero-shot prompt (short instructions)", 300 + 100, 30, 0.50, 1.50, 0, 0),
        new Approach("Few-shot prompt (20 examples, 2,000 tokens)", 300 + 2_000 + 100, 30, 0.50, 1.50, 0, 0),
        new Approach("RAG (3 retrieved chunks, 1,200 tokens)", 300 + 1_200 + 100, 30, 0.50, 1.50, 300, 100),
        // A fine-tuned model has learned the examples, so its prompt is short, but its tokens cost more here
        new Approach("Fine-tuned model (short prompt)", 100 + 100, 30, 1.00, 3.00, 1_000, 0),
    };

    System.out.printf("%-46s %s%n", "Approach", "cost per 1,000 requests");
    for (Approach a : approaches)
      System.out.printf("%-46s $%.3f%n", a.name(), a.perRequest() * 1000);

    System.out.printf("%n%-46s %12s %12s %12s%n", "First-year cost at requests per month:", "10,000", "100,000", "1,000,000");
    for (Approach a : approaches)
      System.out.printf("%-46s %12s %12s %12s%n", a.name(),
          String.format("$%,.0f", a.firstYear(10_000)), String.format("$%,.0f", a.firstYear(100_000)),
          String.format("$%,.0f", a.firstYear(1_000_000)));

    // When does fine-tuning pay for itself against the few-shot prompt?
    Approach fewShot = approaches[1], tuned = approaches[3];
    double savedPerRequest = fewShot.perRequest() - tuned.perRequest();
    double breakEven = (tuned.oneTimeCost() - fewShot.oneTimeCost()) / savedPerRequest;
    System.out.printf("%nThe fine-tuned model saves $%.6f per request against the few-shot prompt.%n", savedPerRequest);
    System.out.printf("Its $%,.0f setup cost is repaid after %,.0f requests (about %,.0f a month for a year).%n",
        tuned.oneTimeCost(), breakEven, breakEven / 12);
  }
}
 OUTPUT of the above code
Approach                                       cost per 1,000 requests
Zero-shot prompt (short instructions)          $0.245
Few-shot prompt (20 examples, 2,000 tokens)    $1.245
RAG (3 retrieved chunks, 1,200 tokens)         $0.845
Fine-tuned model (short prompt)                $0.290

First-year cost at requests per month:               10,000      100,000    1,000,000
Zero-shot prompt (short instructions)                   $29         $294       $2,940
Few-shot prompt (20 examples, 2,000 tokens)            $149       $1,494      $14,940
RAG (3 retrieved chunks, 1,200 tokens)               $1,601       $2,514      $11,640
Fine-tuned model (short prompt)                      $1,035       $1,348       $4,480

The fine-tuned model saves $0.000955 per request against the few-shot prompt.
Its $1,000 setup cost is repaid after 1,047,120 requests (about 87,260 a month for a year).

Explanation of the code

  • The few-shot prompt costs five times as much per request as the zero-shot prompt ($1.245 against $0.245 per 1,000 requests), because its 2,000 tokens of examples are sent again with every request. The fine-tuned model has learned those examples, so its prompt is short: even at twice the price per token, it costs $0.290 per 1,000 requests.
  • At 10,000 requests a month, fine-tuning is the wrong choice on cost: $1,035 in the first year against $149 for the few-shot prompt. At 100,000 a month the two are close ($1,348 and $1,494), and at a million a month the fine-tuned model costs less than a third as much ($4,480 against $14,940).
  • The break-even is simple division: the $1,000 setup cost divided by the $0.000955 saved on every request gives about a million requests, or 87,000 a month for a year. Cheaper base-model tokens or a shorter few-shot prompt push it further out; a longer few-shot prompt or a cheaper fine-tuned model brings it closer.
  • The RAG row is there to show its price tag, not to compete: it costs more than a short prompt because of the retrieved text and the monthly cost of the vector store, and it solves a different problem - missing knowledge. The zero-shot prompt is the cheapest of all, so the real question is whether its answers are good enough, and only the test set can say.
  • The numbers leave out the biggest cost of fine-tuning: the people's time to collect, clean and check the examples. Several providers also charge less for repeated prompt prefixes (prompt caching), which shrinks the gap for long few-shot prompts.
The same numbers as a chart: the fine-tuned model starts higher, because of the setup cost, and then climbs more slowly -
The few-shot prompt costs more per request, so the fine-tuned model, which starts at 1,000 dollars, becomes cheaper after about 1.05 million requests $0$1,000$2,000$3,000 00.5 million1 million1.5 million2 million requests cumulative cost break-even: about 1.05 million requests few-shot prompt fine-tuned model
Cumulative cost, from the numbers in the program above. Below the crossing point the prompt is cheaper; above it, the fine-tuned model is.

Combining the three

The three levers do not exclude each other, and serious systems usually use more than one -
  • Prompt + RAG is the most common setup: instructions and output format in the prompt, facts from retrieval. It is where almost every RAG systemClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval starts.
  • Fine-tuned model + RAG uses each for its own job: fine-tuning for the answer style, the citation format and the way to say "I don't know", RAG for the facts. The fine-tuning examples teach how to use retrieved text, not what it says.
  • Fine-tuning the retrieval models is a third option: training the embedding model or the re-ranker on your own questions and documents, when retrieval - not the generating model - is the weak step.
  • A small fine-tuned model for the narrow, high-volume step, and a large prompted model for the hard cases, is a common way to keep cost down.

Mistakes to avoid

  • Fine-tuning to add facts. It is slow, unreliable and cannot be updated. Use RAG.
  • Expecting a bigger or reasoning model to know your facts. A stronger model writes better, and a reasoning modelClick to read tutorial: Reasoning Models and Test-Time Compute Explained thinks longer, but neither has read your documents. Missing knowledge still needs RAG.
  • Using RAG to fix style. Retrieved documents do not change how a model writes. Use examples in the prompt, or fine-tuning.
  • Skipping the baseline. Without a test set and a first prompt, you cannot tell whether a technique helped, or whether it cost more than it saved.
  • Blaming the model for bad retrieval. When a RAG system is wrong, first check whether the right chunk was retrieved; if it was not, no model could have answered correctly.
  • Fine-tuning on messy examples. Inconsistent or wrong examples are learned too. A few hundred clean ones beat thousands of sloppy ones.
  • Choosing by fashion. The cheapest technique which passes your test set is the right one, however unglamorous.

Summary

In this post we compared prompt engineering, RAG and fine-tuning. We saw that they act on different places - prompting and RAG on the context window, fine-tuning on the weights - so that RAG gives a model knowledge, fine-tuning changes its behaviour, and prompting steers both cheaply. We went through why to start with a prompt and a test set, when the problem is missing knowledge and RAG is the answer, and when it is behaviour and fine-tuning is, with a decision flow and a table of problems and first moves. In Java, we calculated what four approaches cost, and found that fine-tuning repaid its setup cost only after about a million requests. Lastly, we looked at combining the techniques and the mistakes to avoid. In the next post, we will look at models which are trained to think before they answer, in reasoning models and test-time computeClick to read tutorial: Reasoning Models and Test-Time Compute Explained.