DeepSeek-R1 Paper Explained: Reasoning via RL
Introduction
In September 2024, OpenAI's o1 showed that a language model which "thinks" at length before answering - writing a long chain of thought, checking its work and trying other approaches - is dramatically better at maths, coding and science problems. How such a model was trained was not published. Four months later, the Chinese lab DeepSeek released DeepSeek-R1, a model with comparable reasoning results, along with a paper explaining how it was made and the model's weights under an open licence. Its most striking finding: reasoning can emerge from reinforcement learning alone, with simple rule-based rewards and no examples of how to reason. This post explains the DeepSeek-R1 paper with diagrams. We will go through DeepSeek-R1-Zero and its "aha moment", the GRPO algorithm - whose group-relative advantages we will compute in Java - the four-stage pipeline behind DeepSeek-R1, distillation into smaller models, and the results.The paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Authors: DeepSeek-AI
Published: January 2025 - a version was published in Nature in September 2025
Authors: DeepSeek-AI
Published: January 2025 - a version was published in Nature in September 2025
Background: reasoning and test-time compute
Chain-of-thought promptingClick to read tutorial: Chain-of-Thought Prompting Paper Explained showed that models answer better when they reason step by step. Reasoning models take this much further: they are trained to produce long chains of thought - often thousands of tokens - before giving an answer. This spends more computation when the model is used, at test time, rather than only during training, and accuracy keeps improving as the model is allowed to think longer. The open question was how to train this behaviour. Supervised fine-tuning needs large numbers of high-quality reasoning examples, which are expensive to write. DeepSeek asked whether a model could learn to reason purely from being rewarded for correct answers.DeepSeek-R1-Zero: reinforcement learning from a base model
DeepSeek-R1-Zero starts from DeepSeek-V3-Base, a pre-trained mixture-of-expertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts model with 671 billion parameters, of which 37 billion are used for each token. Unlike the InstructGPTClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF recipe, there is no supervised fine-tuning step - reinforcement learning is applied directly to the base model. The model is given a simple template which asks it to put its reasoning between<think> tags and its final answer between <answer> tags, and is trained on maths, coding and logic problems whose answers can be checked automatically. Two rule-based rewards are used -
- an accuracy reward - is the final answer correct? For maths, the answer is compared with the known result; for code, the program is compiled and run against test cases, and
- a format reward - did the model put its thinking inside the
<think>tags?
One step of R1-Zero's training: sample a group, score it with rules, compare each answer with its group.
GRPO: group relative policy optimization
The reinforcement learning algorithm is GRPO, introduced by DeepSeek in an earlier paper on maths reasoning. It is a variant of PPO, the algorithm used in RLHF, with one important change. PPO needs a second neural network, a critic or value model, as large as the policy itself, to estimate how good each answer is expected to be - so that the update can tell whether an answer was better or worse than expected. GRPO drops the critic. Instead, for each question it samples a group of G answers and uses the group's own average reward as the baseline -Where the baseline comes from: a second network in PPO, the group itself in GRPO.
Rule-based rewards and group-relative advantages
//GrpoAdvantages.java
import java.util.List;
import java.util.regex.*;
public class GrpoAdvantages {
static final Pattern FORMAT = Pattern.compile("^<think>.+</think>\\s*<answer>(.+)</answer>$", Pattern.DOTALL);
// Rule-based reward: 1 for a correct answer, plus 0.5 for using the think/answer format
static double reward(String output, String expected) {
Matcher m = FORMAT.matcher(output.trim());
boolean formatted = m.matches();
String answer = formatted ? m.group(1).trim() : output.replaceAll(".*?(\\d+)\\s*$", "$1").trim();
return (answer.equals(expected) ? 1.0 : 0.0) + (formatted ? 0.5 : 0.0);
}
public static void main(String[] args) {
String expected = "408"; // question: what is 17 x 24 ?
List<String> group = List.of(
"<think>17 x 24 = 17 x 20 + 17 x 4 = 340 + 68 = 408</think><answer>408</answer>",
"<think>17 x 24 = 17 x 25 - 17 = 425 - 17 = 408</think><answer>408</answer>",
"<think>17 x 24 = 340 + 58 = 398</think><answer>398</answer>",
"<think>Wait, let me recheck: 24 x 17 = 240 + 168 = 408</think><answer>408</answer>",
"The answer is 408",
"<think>17 x 24 is about 400</think><answer>400</answer>");
double[] r = group.stream().mapToDouble(o -> reward(o, expected)).toArray();
double mean = java.util.Arrays.stream(r).average().orElse(0);
double std = Math.sqrt(java.util.Arrays.stream(r).map(x -> (x - mean) * (x - mean)).average().orElse(0));
System.out.printf("Group mean reward %.3f, std %.3f%n", mean, std);
for (int i = 0; i < r.length; i++) {
double advantage = (r[i] - mean) / std;
System.out.printf("answer %d: reward %.1f advantage %+.2f %s%n", i + 1, r[i], advantage,
advantage > 0 ? "-> reinforce" : "-> discourage");
}
}
}
Group mean reward 1.083, std 0.449 answer 1: reward 1.5 advantage +0.93 -> reinforce answer 2: reward 1.5 advantage +0.93 -> reinforce answer 3: reward 0.5 advantage -1.30 -> discourage answer 4: reward 1.5 advantage +0.93 -> reinforce answer 5: reward 1.0 advantage -0.19 -> discourage answer 6: reward 0.5 advantage -1.30 -> discourage
Explanation of the code
- The reward is computed by rules - a regular expression for the format and a string comparison for the answer - so no reward model is involved. (The weights 1 and 0.5 are this example's choice; the paper does not give its exact values.)
- The four correct, well-formatted answers get a positive advantage and will be reinforced, including the one which checked its work with "Wait, let me recheck". The two wrong answers get negative advantages.
- Answer 5 is correct but ignores the format, so it scores below the group average and is slightly discouraged - which is how the format reward teaches the model to reason inside the think tags.
- Nothing rewards the reasoning itself. The model is only told whether its final answer was right; any reasoning strategy which leads to more right answers gets reinforced indirectly.
What emerged: longer thinking and the "aha moment"
Over thousands of RL steps, R1-Zero's accuracy on the AIME 2024 maths competition rose from 15.6% to 71.0% (pass@1 - the share of problems solved on the first attempt), and to 86.7% with majority voting over 64 samples, the self-consistencyClick to read tutorial: Chain-of-Thought Prompting Paper Explained technique - matching OpenAI's o1-0912. Even more interesting was how it got there -- Thinking got longer on its own: the average length of its responses grew steadily during training, from hundreds to thousands of tokens. Nobody told the model to think longer; longer thinking simply earned more reward on hard problems.
- Reflection emerged: the model started to re-examine its earlier steps, try alternative approaches, and verify its answers - behaviours which were never shown to it.
- The "aha moment": the paper shows a response from an intermediate version of the model where it stops mid-solution, flags the moment in so many words, and goes back to re-evaluate its approach. The authors describe it as an "aha moment" for the researchers as much as for the model.
DeepSeek-R1: a four-stage pipeline
To keep R1-Zero's reasoning while making the model readable and useful as a general assistant, DeepSeek-R1 was trained in four stages, again starting from DeepSeek-V3-Base -How DeepSeek-R1 was trained, and how its data was reused to distil smaller models.
- Cold start: the base model is first fine-tuned on a few thousand long chain-of-thought examples in a clean, readable format, collected partly from R1-Zero's outputs and cleaned up by people. This gives RL a better starting point than the raw base model.
- Reasoning-oriented RL: the same GRPO training as R1-Zero, plus a reward for keeping the chain of thought in a single language. This slightly lowered benchmark scores, but made the reasoning far easier to read.
- Rejection sampling and SFT: the RL model generates many answers to many prompts, and only the correct and readable ones are kept - about 600,000 reasoning samples. About 200,000 non-reasoning samples (writing, factual questions, translation) are added, and DeepSeek-V3-Base is fine-tuned on all 800,000.
- RL for all scenarios: a final RL stage combines rule-based rewards for reasoning tasks with reward models for helpfulness and harmlessness on general prompts, as in RLHF.
Distillation: reasoning in small models
DeepSeek also used the 800,000 samples to fine-tune much smaller open models - Qwen2.5 and Llama models from 1.5 to 70 billion parameters. This is distillation: a small "student" model learns from the outputs of a large "teacher". The results were strong: the distilled 7B Qwen model scored 55.5% on AIME 2024, beating the much larger QwQ-32B-Preview, and the distilled 32B model scored 72.6%. Running the same large-scale RL directly on a 32B model gave much weaker results than distilling from R1. The paper's conclusion: for small models, distillation from a strong reasoning model is far more effective than RL on their own - although pushing the frontier still needs bigger base models and large-scale RL.Results
- Maths: 79.8% pass@1 on AIME 2024 and 97.3% on MATH-500 - on par with OpenAI's o1-1217.
- Coding: a Codeforces rating of 2,029, better than 96.3% of human participants.
- Knowledge and general tasks: clear gains over DeepSeek-V3 on knowledge benchmarks such as MMLU and GPQA Diamond, and strong results on writing and open-ended question answering.
- Openness: the weights of R1, R1-Zero and the six distilled models were released under the MIT licence, so anyone could run, study and build on a frontier-level reasoning model.
Impact
DeepSeek-R1 showed that a clear recipe - large-scale RL with verifiable rewards, on a strong base model - produces reasoning ability, and that reasoning behaviours such as reflection can emerge without being demonstrated. Its open weights and the low reported cost of its training challenged assumptions about how much it takes to build a frontier model, and "RL with verifiable rewards" became a standard stage in training language models, alongside the pre-training and RLHF stages covered in the earlier posts. It also shows how the papers in this category build on each other: the TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained provided the architecture, the scaling lawsClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained the case for scale, and papers from GPT-3Click to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners to ReActClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs the ways to prompt, align, adapt and connect models - while R1 showed how far training a model to think could go.Summary
In this post we went through the DeepSeek-R1 paper. We saw how DeepSeek-R1-Zero learned to reason through reinforcement learning alone, starting from a base model with only rule-based accuracy and format rewards, and how its thinking grew longer and more reflective on its own, including its "aha moment". We went through GRPO, which replaces PPO's critic with a group of sampled answers, and computed its group-relative advantages in Java. We then went through DeepSeek-R1's four-stage pipeline, which fixed R1-Zero's readability and language mixing, and distillation of its reasoning into smaller models. Lastly, we looked at the results, which matched OpenAI's o1 with open weights, and the paper's impact. The posts after the papers explain the core concepts of working with LLMs, starting with tokens, context windows, embeddings and hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations.Continue Reading: AI and LLM Papers Explained
LLM Tokens, Context Windows, Embeddings and HallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained
LLM Tokens, Context Windows, Embeddings and HallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained