Reasoning Models and Test-Time Compute Explained
Introduction
For years, making a language model better meant making it bigger. Then, in September 2024, OpenAI's o1 showed another way: let the model think before it answers, by writing a long chain of thought which it checks and corrects along the way, and it becomes far better at maths, coding and science problems. DeepSeek-R1 repeated the result in the open four months later. The idea behind these reasoning models is that computation spent when the model answers - test-time compute - can buy as much accuracy as computation spent in training. This post explains reasoning models and test-time compute with diagrams: the two places where compute can be spent, what a reasoning model does differently, the three ways of spending more compute on one question, a Java simulation of voting over many answers which shows when it helps and when it backfires, how to use reasoning models well, and their limits.(Note - This post builds on chain-of-thought promptingClick to read tutorial: Chain-of-Thought Prompting Paper Explained, the scaling lawsClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained and the DeepSeek-R1 paper postClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL, which explains how such a model is trained.)
Two places to spend compute
The scaling lawsClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained describe how a model improves when more compute goes into training: a bigger model, more data, a longer run. That compute is spent once, by the lab, and every user shares the result. Test-time compute is the other place -Compute can be spent when the model is made, or when it answers.
What a reasoning model does differently
With chain-of-thought prompting, a person asks an ordinary model to reason step by step. A reasoning model has been trained to do it, always, and for much longer: it was rewarded by reinforcement learning for reaching correct answers on maths, code and logic problems, as described in the DeepSeek-R1 paper postClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL, and it learned that planning, checking and backtracking earn reward. Its response has two parts -A reasoning model writes its thinking first, then a short answer.
- Thinking tokens are ordinary tokens, generated one at a time like any others (see tokens and context windowsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations), but they are working notes, not the answer. Some models show them - DeepSeek-R1 writes them between
<think>tags - while several commercial APIs keep them hidden or return only a summary. - The answer is usually short, and written with the thinking in view, so it can use the result of the checks.
- You pay for the thinking on most APIs: the thinking tokens are billed as output tokens even when you cannot see them, and they take time to generate. They also share the context windowClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations and the maximum output length with the answer.
Three ways to spend test-time compute
More compute at answer time can be spent in three different shapes -Orange circles are reasoning steps or rejected samples, green ones what is kept, and grey ones the branches a search drops.
- Think longer (sequential). One chain of thought, allowed to run further. This is what a reasoning model does by itself. OpenAI reported for o1 that accuracy on competition maths rose steadily with the logarithm of the compute spent on thinking: every doubling of thinking bought a similar gain, so the returns diminish but do not stop quickly.
- Sample many, then choose (parallel). Generate n separate answers and pick one: the most common answer, as in the self-consistency of the chain-of-thought paper postClick to read tutorial: Chain-of-Thought Prompting Paper Explained, or the one a checker accepts - a unit test for code, a proof checker for maths, or a trained scorer. The Java program below simulates this. Brown and colleagues (Large Language Monkeys, 2024) found that with a reliable checker, the share of coding problems solved kept rising as the number of samples grew by orders of magnitude.
- Search. Instead of independent samples, explore a tree of partial reasoning, score each step with a process reward model (like the reward model of RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF, but trained on whether individual steps are correct, as in Lightman and colleagues' Let's Verify Step by Step, 2023), and spend more effort on the promising branches. It is the most complex, and the DeepSeek-R1 team reported that tree search did not scale well for them, so the simple long chain of thought won in practice.
Voting over many answers, simulated in Java
Sampling many answers and voting is easy to simulate, and it teaches something which a plain description hides: voting does not make a model correct, it amplifies whatever answer the model gives most often. The program below answers one question many times in four scenarios - the model is right 60% of the time with one wrong answer (A), right 40% of the time with one wrong answer (B), right 40% of the time with the other 60% spread over four different wrong answers (C), and, for comparison, a perfect checker which picks the right answer whenever one appears (D). It then counts what the samples cost. The samples are assumed to be independent.Accuracy of a vote over n samples, and what the samples cost
//ThinkingVotes.java
import java.util.Random;
public class ThinkingVotes {
// Simulates answering a question n times and taking the most common answer (a plurality vote).
// wrongAnswers: how many different wrong answers the model gives; pCorrect: chance of the right one per sample.
// The samples are independent, which is the assumption that makes voting work - real samples are less independent.
static double voteAccuracy(double pCorrect, int wrongAnswers, int n, int trials, Random random) {
int correctVotes = 0;
for (int t = 0; t < trials; t++) {
int[] counts = new int[1 + wrongAnswers]; // index 0 is the correct answer
for (int i = 0; i < n; i++) {
double r = random.nextDouble();
int answer = r < pCorrect ? 0 : 1 + (int) ((r - pCorrect) / (1 - pCorrect) * wrongAnswers);
counts[Math.min(answer, wrongAnswers)]++;
}
int best = 0, tied = 0; // pick the most common answer, ties broken at random
for (int a = 0; a < counts.length; a++) {
if (counts[a] > counts[best]) { best = a; tied = 1; }
else if (counts[a] == counts[best] && a != best) { tied++; if (random.nextInt(tied) == 0) best = a; }
}
if (best == 0) correctVotes++;
}
return (double) correctVotes / trials;
}
public static void main(String[] args) {
Random random = new Random(2026); // fixed seed, so the output is repeatable
int trials = 20_000;
int[] samples = {1, 5, 15, 45};
System.out.println("Accuracy of a vote over n independent samples (" + trials + " simulated questions each):");
System.out.printf("%-58s", "");
for (int n : samples) System.out.printf("%8s", "n=" + n);
System.out.println();
Object[][] scenarios = {
{"A: right answer 60%, one wrong answer 40%", 0.60, 1},
{"B: right answer 40%, one wrong answer 60%", 0.40, 1},
{"C: right answer 40%, 60% spread over 4 wrong answers", 0.40, 4},
};
for (Object[] s : scenarios) {
System.out.printf("%-58s", s[0]);
for (int n : samples)
System.out.printf("%7.1f%%", 100 * voteAccuracy((double) s[1], (int) s[2], n, trials, random));
System.out.println();
}
// If a perfect checker could pick the correct answer out of the samples (a unit test, say), one correct sample is enough
System.out.printf("%-58s", "D: as C, but a perfect checker picks the right one");
for (int n : samples) System.out.printf("%7.1f%%", 100 * (1 - Math.pow(0.60, n)));
System.out.println();
// The cost side: every sample is a full answer
System.out.println();
System.out.println("Cost: an answer of 2,000 thinking tokens and 200 answer tokens, at $10 per million output tokens");
for (int n : samples)
System.out.printf(" n=%-3d %,7d output tokens $%.4f per question%n", n, n * 2_200, n * 2_200 * 10 / 1e6);
}
}
Accuracy of a vote over n independent samples (20000 simulated questions each):
n=1 n=5 n=15 n=45
A: right answer 60%, one wrong answer 40% 60.1% 68.2% 78.5% 91.3%
B: right answer 40%, one wrong answer 60% 39.8% 31.5% 21.4% 8.7%
C: right answer 40%, 60% spread over 4 wrong answers 39.6% 44.8% 72.5% 96.4%
D: as C, but a perfect checker picks the right one 40.0% 92.2% 100.0% 100.0%
Cost: an answer of 2,000 thinking tokens and 200 answer tokens, at $10 per million output tokens
n=1 2,200 output tokens $0.0220 per question
n=5 11,000 output tokens $0.1100 per question
n=15 33,000 output tokens $0.3300 per question
n=45 99,000 output tokens $0.9900 per question
Explanation of the code
- A shows voting at its best. A model which is right 60% of the time becomes right 91.3% of the time with 45 samples, because the wrong answer is outvoted. It gets there slowly: 68.2% with 5 samples, 78.5% with 15.
- B shows the danger. When the model is wrong more often than right, with the same wrong answer each time, voting makes it worse: 39.8% with one sample, 8.7% with 45. Voting does not add correctness; it amplifies the answer the model prefers.
- C is the realistic case, and the reason self-consistency works: the model is right only 40% of the time, but its mistakes scatter over several wrong answers, so the right answer, at 40%, still beats each wrong one, at 15%. With one sample it is right 39.6% of the time; with 45 samples, 96.4%. With 5 samples the gain is small (44.8%): the benefit arrives late.
- D shows what a checker is worth. If something reliable can test an answer - unit tests, a proof checker - then a single correct sample is enough: 92.2% with 5 samples and 100% from 15. This is why reasoning models improve most in maths and code, where answers can be checked, and why the DeepSeek-R1 training used such checks as its reward.
- The cost lines are the other side of the table. Every sample is a full answer: 45 samples of 2,200 tokens is 99,000 output tokens, or $0.99 per question against $0.022 for one answer - 45 times as much, in money and in tokens.
- The simulation assumes the samples are independent. Real samples from the same model share its habits, and the same mistake tends to repeat, so the real gains are smaller than the simulated ones.
Voting amplifies the most common answer: it lifts A and C and sinks B. A perfect checker (D) needs far fewer samples.
Using reasoning models well
- Use them for problems with several steps - maths, debugging, planning, data analysis, hard questions about code - where a wrong turn early ruins the answer and checking pays off.
- Do not use them for easy work. Looking up a fact, classifying a ticket, rewriting a sentence or a quick chat reply gain nothing from long thinking, and cost more and take longer. A small standard model is the better tool, and a common design is to route easy requests to it and only hard ones to a reasoning model.
- Prompt differently. State the problem, the constraints and what a good answer looks like. Do not tell the model to "think step by step" or lay out the steps for it: it already thinks, and prescribing the method can make it worse. See prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples.
- Set the budget. Many APIs let you choose how much the model may think - a reasoning effort level, or a maximum number of thinking tokens. Lower it for latency and cost, raise it for hard problems, and make sure the maximum output length leaves room for both thinking and answer.
- Watch the bill. Take a question answered in 300 tokens by a standard model, and by a reasoning model with 6,000 tokens of thinking first. At the same price per token, the second costs 21 times as much in output tokens - and the thinking length varies from question to question, so the cost does too.
- Expect fixed sampling settings. Some reasoning models fix their own temperature, and some APIs do not accept the settings described in temperature, top-k and top-p samplingClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained.
- Combine them with tools. A reasoning model which can call tools can think, look something up, and think again about the result - the loop of the ReAct paperClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs, which is how most modern coding and research agents work.
Limits
- Overthinking. Reasoning models can spend thousands of tokens on a trivial question, and long chains can talk themselves out of a correct first answer.
- Thinking does not supply facts. If the model does not know your company's policy or yesterday's news, more thinking only produces a more elaborate guess. Missing knowledge needs retrieval - see fine-tuning vs RAG vs prompt engineeringClick to read tutorial: Fine-Tuning vs RAG vs Prompt Engineering: Which to Use.
- The written reasoning is not a guaranteed account of the model's process. Research, starting with Turpin and colleagues' Language Models Don't Always Say What They Think (2023), has found that a model's stated reasoning does not always reflect what produced its answer, so it is a useful clue, not a proof.
- Hallucinations remain. Reasoning models are more accurate on checkable problems, but they can still state wrong things with confidence, and their long answers make mistakes harder to spot.
- Gains are strongest where answers can be verified. In open-ended writing or advice there is no reliable checker, so extra compute buys much less.
- Cost and latency are variable. The slowest and costliest requests are the hardest ones - the ones where you may most need an answer quickly.
Summary
In this post we went through reasoning models and test-time compute. We saw that compute can be spent in training, once, or at answer time, on every request, and that a model which is allowed to think can spend its effort where a problem is hard. We saw how a reasoning model - trained with reinforcement learning to write long, self-checking chains of thought before a short answer - differs from chain-of-thought prompting, and the three shapes of test-time compute: think longer, sample many answers and choose, and search. In Java, we simulated voting over many answers, and saw it lift a model which is usually right, sink one which is usually wrong in the same way, work well when mistakes scatter, and need far fewer samples with a perfect checker - at a cost which grows with every sample. Lastly, we looked at using reasoning models well, and at their limits. In the next post, we will look at who can run and change these models, in open-weight, closed-weight and open-source LLMsClick to read tutorial: Open-Weight vs Closed-Weight vs Open-Source LLMs Explained.Continue Reading: AI and LLM Papers Explained
Open-Weight vs Closed-Weight vs Open-Source LLMs ExplainedClick to read tutorial: Open-Weight vs Closed-Weight vs Open-Source LLMs Explained Fine-Tuning vs RAG vs Prompt Engineering: Which to UseClick to read tutorial: Fine-Tuning vs RAG vs Prompt Engineering: Which to Use Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained
Open-Weight vs Closed-Weight vs Open-Source LLMs ExplainedClick to read tutorial: Open-Weight vs Closed-Weight vs Open-Source LLMs Explained Fine-Tuning vs RAG vs Prompt Engineering: Which to UseClick to read tutorial: Fine-Tuning vs RAG vs Prompt Engineering: Which to Use Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained