Scaling Laws for Language Models (and Chinchilla) Explained
Introduction
Why did language models grow from BERT's 340 million parameters in 2018 to GPT-3's 175 billion just two years later - and why did AI labs believe that spending ever more money on training would pay off? A large part of the answer is a 2020 paper from OpenAI, Scaling Laws for Neural Language Models. It found that a language model's performance improves in a smooth, predictable way as you increase the model size, the amount of training data and the compute used for training. This post explains scaling laws for language models with charts. We will first see what was measured and the power laws which were found. We will then look at the formula for training compute, and how to split a compute budget between model size and data - including the 2022 Chinchilla paper, which overturned the original answer. Along the way we will write a Java calculator for compute-optimal model sizes.The paper: Scaling Laws for Neural Language Models
Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei (OpenAI, Johns Hopkins University)
Published: January 2020
Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei (OpenAI, Johns Hopkins University)
Published: January 2020
What was measured
The authors trained hundreds of decoder-only TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained language models, from 768 to 1.5 billion parameters, on up to 23 billion tokens of web text. For each model they measured the test loss - the cross-entropy loss, a measure of how surprised the model is by text it has not seen; lower is better (see softmax, cross-entropy and KL divergenceClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained). They then studied how the loss depends on three quantities -- N - the number of model parameters (excluding the embedding parameters),
- D - the size of the training dataset, in tokensClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations, and
- C - the amount of compute used for training, in floating point operations (FLOPs).
The finding: loss follows power laws
When the other two quantities are not the bottleneck, the loss falls as a power law in each of N, D and C, over more than seven orders of magnitude. For model size, the paper's fit is -L(N) = (Nc / N)0.076, where Nc ≈ 8.8 × 1013The paper's fitted law for model size, plotted from 100 thousand to 100 billion parameters.
L(D) = (Dc / D)0.095, and for compute, L(C) ∝ C-0.050. The paper also found -
- Shape matters little: within a wide range, the number of layers, the width and the number of attention heads hardly affect the loss. What matters is the total parameter count.
- Larger models are more sample-efficient: a bigger model reaches the same loss after seeing fewer tokens.
- Overfitting is predictable: if N grows without D, the gains slow down in a way that can also be predicted from N and D.
Training compute: C ≈ 6ND
Training a Transformer costs about 6 floating point operations per parameter per token: 2 for the forward pass (a multiply and an add for each parameter) and about 4 for the backward pass, which computes the gradients. So the total training compute is approximately -C ≈ 6 × N × DThe 2020 answer: make the model bigger
Kaplan and colleagues concluded that most of an increase in compute should go into model size. In their fit, the optimal model size grows roughly as C0.73, while the amount of data grows only as about C0.27 - so a 10× bigger budget should buy a model about 5.4× bigger, trained on only about 1.9× more data. Models should be trained to well short of convergence. GPT-3, trained on "only" 300 billion tokens, followed this advice, as did other models of the time such as Gopher (280 billion parameters, 300 billion tokens) and Megatron-Turing NLG (530 billion parameters).The 2022 correction: Chinchilla
In 2022, a team at DeepMind published Training Compute-Optimal Large Language Models (Hoffmann et al.). They trained over 400 models, from 70 million to over 16 billion parameters on 5 to 500 billion tokens, and fitted the trade-off again with three different methods. All three gave the same answer: model size and training data should grow in equal proportion - both roughly as C0.5. In practice this works out to about 20 training tokens per parameter. The earlier result had been skewed by, among other things, using the same learning-rate schedule for every training length. The diagram shows how differently the two papers would spend a 10 times bigger training budget -The same 10× budget, spent the 2020 way and the Chinchilla way.
By this measure, the large models of the day were badly undertrained - far too big for the data they had seen. To prove it, DeepMind trained Chinchilla with the same compute budget as their own Gopher model, but split differently -
Gopher vs Chinchilla: a quarter of the parameters, 4.7 times the data.
A compute-optimal model calculator in Java
With Chinchilla's rule of thumb, D = 20N, the compute formula becomes C = 6 × N × 20N = 120N², so the compute-optimal model size for a budget is N = √(C / 120). The program below applies this to a few compute budgets, and also shows the loss predicted by the 2020 paper's law for some model sizes -Compute-optimal model size and data for a budget
//ComputeOptimal.java
public class ComputeOptimal {
static final double TOKENS_PER_PARAM = 20; // Chinchilla rule of thumb
static String human(double x) {
if (x >= 1e12) return String.format("%.1fT", x / 1e12);
if (x >= 1e9) return String.format("%.0fB", x / 1e9);
return String.format("%.0fM", x / 1e6);
}
public static void main(String[] args) {
// C = 6 * N * D and D = 20 * N => N = sqrt(C / 120)
double gpt3 = 6 * 175e9 * 300e9;
double[] budgets = {1e21, gpt3, 5.76e23, 1e25};
String[] names = {"1e21 FLOPs", "GPT-3's budget", "Gopher's budget", "1e25 FLOPs"};
System.out.printf("GPT-3 training compute: %.2e FLOPs%n%n", gpt3);
System.out.printf("%-16s %-12s %-14s %s%n", "Budget", "FLOPs", "Optimal size", "Optimal tokens");
for (int i = 0; i < budgets.length; i++) {
double n = Math.sqrt(budgets[i] / (6 * TOKENS_PER_PARAM));
System.out.printf("%-16s %-12.2e %-14s %s%n", names[i], budgets[i], human(n), human(TOKENS_PER_PARAM * n));
}
// Kaplan et al. (2020): L(N) = (Nc / N)^0.076
System.out.println();
for (double n : new double[] {1e8, 1e9, 1e10, 1e11}) {
System.out.printf("Predicted loss for %-5s parameters: %.3f%n", human(n), Math.pow(8.8e13 / n, 0.076));
}
}
}
GPT-3 training compute: 3.15e+23 FLOPs Budget FLOPs Optimal size Optimal tokens 1e21 FLOPs 1.00e+21 3B 58B GPT-3's budget 3.15e+23 51B 1.0T Gopher's budget 5.76e+23 69B 1.4T 1e25 FLOPs 1.00e+25 289B 5.8T Predicted loss for 100M parameters: 2.830 Predicted loss for 1B parameters: 2.376 Predicted loss for 10B parameters: 1.994 Predicted loss for 100B parameters: 1.674
Explanation of the code
- For Gopher's budget, the formula gives about 69 billion parameters and 1.4 trillion tokens - almost exactly the size DeepMind chose for Chinchilla.
- For GPT-3's budget, the compute-optimal model would have had about 51 billion parameters trained on about 1 trillion tokens, instead of 175 billion parameters on 300 billion tokens.
- Because N grows with the square root of C, a 100× bigger budget buys only a 10× bigger model - the rest goes into data. This is why training datasets grew from hundreds of billions to tens of trillions of tokens.
- The predicted losses fall by the same factor, about 0.84, for each 10× increase in parameters - the straight line in the chart above.
After Chinchilla: training past the optimum
"Compute-optimal" means the best model for a fixed training budget. But a popular model is run billions of times after it is trained, and a smaller model is cheaper to run. So many later models deliberately train a smaller model on far more than 20 tokens per parameter - Meta's Llama 3 8B, for example, was trained on over 15 trillion tokens, nearly 2,000 tokens per parameter. The loss keeps falling with more data, just more slowly, and the extra training cost is repaid by cheaper inference.Mixture-of-experts models bend the formula in another way: only part of their parameters is used for each token, so the N in C ≈ 6ND is the active parameter count, and the model can hold far more parameters than its compute budget suggests - see mixture of expertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts.
Scaling laws have since been found for other axes too - for example, for the compute spent at inference time on reasoning, which is the idea behind models like DeepSeek-R1Click to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL.
Limitations
- Scaling laws predict the loss, not specific abilities. Some abilities, such as multi-step arithmetic or chain-of-thought reasoningClick to read tutorial: Chain-of-Thought Prompting Paper Explained, appear fairly suddenly at certain scales, which a smooth loss curve does not show.
- The fitted constants depend on the dataset, the tokenizer and the architecture, and do not carry over exactly to other setups.
- The laws assume an unlimited supply of good data. High-quality text on the web is finite, which has made data quality, deduplication and synthetic data major research areas.
Summary
In this post we went through the scaling laws paper and the Chinchilla paper. We saw that a language model's test loss falls as a smooth power law in its size, its training data and its training compute, which made the results of very large training runs predictable. We used the formula C ≈ 6ND for training compute, and saw how the answer to "bigger model or more data?" changed from mostly bigger models in 2020 to equal growth - about 20 tokens per parameter - with Chinchilla in 2022. We wrote a Java calculator which reproduces Chinchilla's size from Gopher's budget, and lastly looked at why many models now train past the compute-optimal point, and at the limits of scaling laws. In the next post, we will see what scale made possible, in the GPT-3 paperClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners.Continue Reading: AI and LLM Papers Explained
GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL
GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL