Reasoning Models and Test-Time Compute Explained

Introduction

For years, making a language model better meant making it bigger. Then, in September 2024, OpenAI's o1 showed another way: let the model think before it answers, by writing a long chain of thought which it checks and corrects along the way, and it becomes far better at maths, coding and science problems. DeepSeek-R1 repeated the result in the open four months later. The idea behind these reasoning models is that computation spent when the model answers - test-time compute - can buy as much accuracy as computation spent in training. This post explains reasoning models and test-time compute with diagrams: the two places where compute can be spent, what a reasoning model does differently, the three ways of spending more compute on one question, a Java simulation of voting over many answers which shows when it helps and when it backfires, how to use reasoning models well, and their limits.

(Note - This post builds on chain-of-thought promptingClick to read tutorial: Chain-of-Thought Prompting Paper Explained, the scaling lawsClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained and the DeepSeek-R1 paper postClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL, which explains how such a model is trained.)

Two places to spend compute

The scaling lawsClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained describe how a model improves when more compute goes into training: a bigger model, more data, a longer run. That compute is spent once, by the lab, and every user shares the result. Test-time compute is the other place -
Compute can be spent in training, once, by the lab, or at test time, on every request, by whoever asks; both lead to better answers Train-time compute a bigger model, more data, a longer training run paid once, by the lab predicted by the scaling laws fixed when the model ships Better answers Test-time compute more thinking tokens, more samples, search over answers paid on every request, by the asker can be turned up for hard questions and down for easy ones Two dials for the same goal: accuracy
Compute can be spent when the model is made, or when it answers.
Why would spending at answer time pay off? Because a model which answers immediately does a fixed amount of work per token, however hard the question: every token passes once through the same stack of layers of the TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained - the point made in the chain-of-thought paper postClick to read tutorial: Chain-of-Thought Prompting Paper Explained. A hard problem needs more steps than a fixed amount of work allows. A model which is allowed to write 5,000 tokens of reasoning can spend them where the problem is hard, check its own work, and try again. Snell and colleagues showed in 2024 that this is a real trade: in Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, spending test-time compute well, with more for hard questions and less for easy ones, let a smaller model outperform one 14 times larger on problems where the small model had a fair chance.

What a reasoning model does differently

With chain-of-thought prompting, a person asks an ordinary model to reason step by step. A reasoning model has been trained to do it, always, and for much longer: it was rewarded by reinforcement learning for reaching correct answers on maths, code and logic problems, as described in the DeepSeek-R1 paper postClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL, and it learned that planning, checking and backtracking earn reward. Its response has two parts -
A standard model answers directly; a reasoning model first writes many thinking tokens, which are billed as output tokens and often hidden or summarised, and then a short answer Standard model Prompt Answer Reasoning model Prompt Thinking tokens: plan, try, check, "wait...", backtrack Answer hundreds to tens of thousands of tokens; billed as output tokens; often hidden, or shown only as a summary what you read
A reasoning model writes its thinking first, then a short answer.
Behaviour which nobody wrote down emerges from this training: the model re-reads the problem, tries a second approach when the first fails, and notices its own mistakes. It also differs from chain-of-thought prompting in how far it goes: reasoning of 10,000 tokens or more is routine, where a prompted chain of thought runs to a few hundred.

Three ways to spend test-time compute

More compute at answer time can be spent in three different shapes -
Think longer in one long chain, sample many answers and vote or check, or search over several reasoning paths and keep the promising ones 1. Think longer 2. Sample, then choose 3. Search one long chain of thought: plan, check, backtrack what reasoning models do vote orcheck n separate answers; the most common, or the one a checker accepts, wins explore several reasoning paths; a scorer keeps the promising ones
Orange circles are reasoning steps or rejected samples, green ones what is kept, and grey ones the branches a search drops.
  • Think longer (sequential). One chain of thought, allowed to run further. This is what a reasoning model does by itself. OpenAI reported for o1 that accuracy on competition maths rose steadily with the logarithm of the compute spent on thinking: every doubling of thinking bought a similar gain, so the returns diminish but do not stop quickly.
  • Sample many, then choose (parallel). Generate n separate answers and pick one: the most common answer, as in the self-consistency of the chain-of-thought paper postClick to read tutorial: Chain-of-Thought Prompting Paper Explained, or the one a checker accepts - a unit test for code, a proof checker for maths, or a trained scorer. The Java program below simulates this. Brown and colleagues (Large Language Monkeys, 2024) found that with a reliable checker, the share of coding problems solved kept rising as the number of samples grew by orders of magnitude.
  • Search. Instead of independent samples, explore a tree of partial reasoning, score each step with a process reward model (like the reward model of RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF, but trained on whether individual steps are correct, as in Lightman and colleagues' Let's Verify Step by Step, 2023), and spend more effort on the promising branches. It is the most complex, and the DeepSeek-R1 team reported that tree search did not scale well for them, so the simple long chain of thought won in practice.

Voting over many answers, simulated in Java

Sampling many answers and voting is easy to simulate, and it teaches something which a plain description hides: voting does not make a model correct, it amplifies whatever answer the model gives most often. The program below answers one question many times in four scenarios - the model is right 60% of the time with one wrong answer (A), right 40% of the time with one wrong answer (B), right 40% of the time with the other 60% spread over four different wrong answers (C), and, for comparison, a perfect checker which picks the right answer whenever one appears (D). It then counts what the samples cost. The samples are assumed to be independent.
Accuracy of a vote over n samples, and what the samples cost
//ThinkingVotes.java
import java.util.Random;

public class ThinkingVotes {

  // Simulates answering a question n times and taking the most common answer (a plurality vote).
  // wrongAnswers: how many different wrong answers the model gives; pCorrect: chance of the right one per sample.
  // The samples are independent, which is the assumption that makes voting work - real samples are less independent.
  static double voteAccuracy(double pCorrect, int wrongAnswers, int n, int trials, Random random) {
    int correctVotes = 0;
    for (int t = 0; t < trials; t++) {
      int[] counts = new int[1 + wrongAnswers];                 // index 0 is the correct answer
      for (int i = 0; i < n; i++) {
        double r = random.nextDouble();
        int answer = r < pCorrect ? 0 : 1 + (int) ((r - pCorrect) / (1 - pCorrect) * wrongAnswers);
        counts[Math.min(answer, wrongAnswers)]++;
      }
      int best = 0, tied = 0;                                   // pick the most common answer, ties broken at random
      for (int a = 0; a < counts.length; a++) {
        if (counts[a] > counts[best]) { best = a; tied = 1; }
        else if (counts[a] == counts[best] && a != best) { tied++; if (random.nextInt(tied) == 0) best = a; }
      }
      if (best == 0) correctVotes++;
    }
    return (double) correctVotes / trials;
  }

  public static void main(String[] args) {
    Random random = new Random(2026);          // fixed seed, so the output is repeatable
    int trials = 20_000;
    int[] samples = {1, 5, 15, 45};

    System.out.println("Accuracy of a vote over n independent samples (" + trials + " simulated questions each):");
    System.out.printf("%-58s", "");
    for (int n : samples) System.out.printf("%8s", "n=" + n);
    System.out.println();
    Object[][] scenarios = {
        {"A: right answer 60%, one wrong answer 40%", 0.60, 1},
        {"B: right answer 40%, one wrong answer 60%", 0.40, 1},
        {"C: right answer 40%, 60% spread over 4 wrong answers", 0.40, 4},
    };
    for (Object[] s : scenarios) {
      System.out.printf("%-58s", s[0]);
      for (int n : samples)
        System.out.printf("%7.1f%%", 100 * voteAccuracy((double) s[1], (int) s[2], n, trials, random));
      System.out.println();
    }

    // If a perfect checker could pick the correct answer out of the samples (a unit test, say), one correct sample is enough
    System.out.printf("%-58s", "D: as C, but a perfect checker picks the right one");
    for (int n : samples) System.out.printf("%7.1f%%", 100 * (1 - Math.pow(0.60, n)));
    System.out.println();

    // The cost side: every sample is a full answer
    System.out.println();
    System.out.println("Cost: an answer of 2,000 thinking tokens and 200 answer tokens, at $10 per million output tokens");
    for (int n : samples)
      System.out.printf("  n=%-3d %,7d output tokens  $%.4f per question%n", n, n * 2_200, n * 2_200 * 10 / 1e6);
  }
}
 OUTPUT of the above code
Accuracy of a vote over n independent samples (20000 simulated questions each):
                                                               n=1     n=5    n=15    n=45
A: right answer 60%, one wrong answer 40%                    60.1%   68.2%   78.5%   91.3%
B: right answer 40%, one wrong answer 60%                    39.8%   31.5%   21.4%    8.7%
C: right answer 40%, 60% spread over 4 wrong answers         39.6%   44.8%   72.5%   96.4%
D: as C, but a perfect checker picks the right one           40.0%   92.2%  100.0%  100.0%

Cost: an answer of 2,000 thinking tokens and 200 answer tokens, at $10 per million output tokens
  n=1     2,200 output tokens  $0.0220 per question
  n=5    11,000 output tokens  $0.1100 per question
  n=15   33,000 output tokens  $0.3300 per question
  n=45   99,000 output tokens  $0.9900 per question

Explanation of the code

  • A shows voting at its best. A model which is right 60% of the time becomes right 91.3% of the time with 45 samples, because the wrong answer is outvoted. It gets there slowly: 68.2% with 5 samples, 78.5% with 15.
  • B shows the danger. When the model is wrong more often than right, with the same wrong answer each time, voting makes it worse: 39.8% with one sample, 8.7% with 45. Voting does not add correctness; it amplifies the answer the model prefers.
  • C is the realistic case, and the reason self-consistency works: the model is right only 40% of the time, but its mistakes scatter over several wrong answers, so the right answer, at 40%, still beats each wrong one, at 15%. With one sample it is right 39.6% of the time; with 45 samples, 96.4%. With 5 samples the gain is small (44.8%): the benefit arrives late.
  • D shows what a checker is worth. If something reliable can test an answer - unit tests, a proof checker - then a single correct sample is enough: 92.2% with 5 samples and 100% from 15. This is why reasoning models improve most in maths and code, where answers can be checked, and why the DeepSeek-R1 training used such checks as its reward.
  • The cost lines are the other side of the table. Every sample is a full answer: 45 samples of 2,200 tokens is 99,000 output tokens, or $0.99 per question against $0.022 for one answer - 45 times as much, in money and in tokens.
  • The simulation assumes the samples are independent. Real samples from the same model share its habits, and the same mistake tends to repeat, so the real gains are smaller than the simulated ones.
The same results as a chart -
Accuracy against the number of samples in four scenarios: voting helps when the right answer is the most common, hurts when a wrong answer is, and a perfect checker needs few samples 0%25%50%75%100% n = 1n = 5n = 15n = 45 number of samples (the cost grows in the same proportion) DCAB A: right 60%, one wrong answer B: right 40%, one wrong answer C: right 40%, 60% spread over 4 wrong answers D: as C, with a perfect checker
Voting amplifies the most common answer: it lifts A and C and sinks B. A perfect checker (D) needs far fewer samples.

Using reasoning models well

  • Use them for problems with several steps - maths, debugging, planning, data analysis, hard questions about code - where a wrong turn early ruins the answer and checking pays off.
  • Do not use them for easy work. Looking up a fact, classifying a ticket, rewriting a sentence or a quick chat reply gain nothing from long thinking, and cost more and take longer. A small standard model is the better tool, and a common design is to route easy requests to it and only hard ones to a reasoning model.
  • Prompt differently. State the problem, the constraints and what a good answer looks like. Do not tell the model to "think step by step" or lay out the steps for it: it already thinks, and prescribing the method can make it worse. See prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples.
  • Set the budget. Many APIs let you choose how much the model may think - a reasoning effort level, or a maximum number of thinking tokens. Lower it for latency and cost, raise it for hard problems, and make sure the maximum output length leaves room for both thinking and answer.
  • Watch the bill. Take a question answered in 300 tokens by a standard model, and by a reasoning model with 6,000 tokens of thinking first. At the same price per token, the second costs 21 times as much in output tokens - and the thinking length varies from question to question, so the cost does too.
  • Expect fixed sampling settings. Some reasoning models fix their own temperature, and some APIs do not accept the settings described in temperature, top-k and top-p samplingClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained.
  • Combine them with tools. A reasoning model which can call tools can think, look something up, and think again about the result - the loop of the ReAct paperClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs, which is how most modern coding and research agents work.
Smaller models can inherit the skill too: DeepSeek fine-tuned small models on the reasoning written by R1, which is sequence-level distillationClick to read tutorial: Knowledge Distillation Explained: Teacher and Student Models, and the distilled 7-billion-parameter model beat much larger ones on maths benchmarks.

Limits

  • Overthinking. Reasoning models can spend thousands of tokens on a trivial question, and long chains can talk themselves out of a correct first answer.
  • Thinking does not supply facts. If the model does not know your company's policy or yesterday's news, more thinking only produces a more elaborate guess. Missing knowledge needs retrieval - see fine-tuning vs RAG vs prompt engineeringClick to read tutorial: Fine-Tuning vs RAG vs Prompt Engineering: Which to Use.
  • The written reasoning is not a guaranteed account of the model's process. Research, starting with Turpin and colleagues' Language Models Don't Always Say What They Think (2023), has found that a model's stated reasoning does not always reflect what produced its answer, so it is a useful clue, not a proof.
  • Hallucinations remain. Reasoning models are more accurate on checkable problems, but they can still state wrong things with confidence, and their long answers make mistakes harder to spot.
  • Gains are strongest where answers can be verified. In open-ended writing or advice there is no reliable checker, so extra compute buys much less.
  • Cost and latency are variable. The slowest and costliest requests are the hardest ones - the ones where you may most need an answer quickly.

Summary

In this post we went through reasoning models and test-time compute. We saw that compute can be spent in training, once, or at answer time, on every request, and that a model which is allowed to think can spend its effort where a problem is hard. We saw how a reasoning model - trained with reinforcement learning to write long, self-checking chains of thought before a short answer - differs from chain-of-thought prompting, and the three shapes of test-time compute: think longer, sample many answers and choose, and search. In Java, we simulated voting over many answers, and saw it lift a model which is usually right, sink one which is usually wrong in the same way, work well when mistakes scatter, and need far fewer samples with a perfect checker - at a cost which grows with every sample. Lastly, we looked at using reasoning models well, and at their limits. In the next post, we will look at who can run and change these models, in open-weight, closed-weight and open-source LLMsClick to read tutorial: Open-Weight vs Closed-Weight vs Open-Source LLMs Explained.