Scaling Laws for Language Models (and Chinchilla) Explained

Introduction

Why did language models grow from BERT's 340 million parameters in 2018 to GPT-3's 175 billion just two years later - and why did AI labs believe that spending ever more money on training would pay off? A large part of the answer is a 2020 paper from OpenAI, Scaling Laws for Neural Language Models. It found that a language model's performance improves in a smooth, predictable way as you increase the model size, the amount of training data and the compute used for training. This post explains scaling laws for language models with charts. We will first see what was measured and the power laws which were found. We will then look at the formula for training compute, and how to split a compute budget between model size and data - including the 2022 Chinchilla paper, which overturned the original answer. Along the way we will write a Java calculator for compute-optimal model sizes.
The paper: Scaling Laws for Neural Language Models
Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei (OpenAI, Johns Hopkins University)
Published: January 2020

What was measured

The authors trained hundreds of decoder-only TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained language models, from 768 to 1.5 billion parameters, on up to 23 billion tokens of web text. For each model they measured the test loss - the cross-entropy loss, a measure of how surprised the model is by text it has not seen; lower is better (see softmax, cross-entropy and KL divergenceClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained). They then studied how the loss depends on three quantities -

The finding: loss follows power laws

When the other two quantities are not the bottleneck, the loss falls as a power law in each of N, D and C, over more than seven orders of magnitude. For model size, the paper's fit is -
L(N) = (Nc / N)0.076, where Nc ≈ 8.8 × 1013
A power law is a straight line on a chart where both axes are logarithmic. Each time the model is made 10 times bigger, the loss is multiplied by the same factor, 10-0.076 ≈ 0.84 - a 16% reduction.
Test loss against model parameters on log-log axes is a straight line 2345 10⁵10⁶10⁷10⁸10⁹10¹⁰10¹¹ Non-embedding parameters N (log scale) Test loss (log scale) L(N) = (8.8×10¹³ / N)^0.076 each 10× more parameters → about 16% lower loss
The paper's fitted law for model size, plotted from 100 thousand to 100 billion parameters.
Similar laws hold for data, L(D) = (Dc / D)0.095, and for compute, L(C) ∝ C-0.050. The paper also found -
  • Shape matters little: within a wide range, the number of layers, the width and the number of attention heads hardly affect the loss. What matters is the total parameter count.
  • Larger models are more sample-efficient: a bigger model reaches the same loss after seeing fewer tokens.
  • Overfitting is predictable: if N grows without D, the gains slow down in a way that can also be predicted from N and D.
This predictability is what changed the field. A lab could train a series of small models, fit the curve, and forecast the loss of a model a hundred times bigger before spending millions of dollars to train it.

Training compute: C ≈ 6ND

Training a Transformer costs about 6 floating point operations per parameter per token: 2 for the forward pass (a multiply and an add for each parameter) and about 4 for the backward pass, which computes the gradients. So the total training compute is approximately -
C ≈ 6 × N × D
For GPT-3, with 175 billion parameters trained on 300 billion tokens, that is 6 × 175×109 × 300×109 ≈ 3.15 × 1023 FLOPs. The formula raises the central question of the scaling-law papers: for a fixed compute budget, should you train a bigger model on less data, or a smaller model on more data?

The 2020 answer: make the model bigger

Kaplan and colleagues concluded that most of an increase in compute should go into model size. In their fit, the optimal model size grows roughly as C0.73, while the amount of data grows only as about C0.27 - so a 10× bigger budget should buy a model about 5.4× bigger, trained on only about 1.9× more data. Models should be trained to well short of convergence. GPT-3, trained on "only" 300 billion tokens, followed this advice, as did other models of the time such as Gopher (280 billion parameters, 300 billion tokens) and Megatron-Turing NLG (530 billion parameters).

The 2022 correction: Chinchilla

In 2022, a team at DeepMind published Training Compute-Optimal Large Language Models (Hoffmann et al.). They trained over 400 models, from 70 million to over 16 billion parameters on 5 to 500 billion tokens, and fitted the trade-off again with three different methods. All three gave the same answer: model size and training data should grow in equal proportion - both roughly as C0.5. In practice this works out to about 20 training tokens per parameter. The earlier result had been skewed by, among other things, using the same learning-rate schedule for every training length. The diagram shows how differently the two papers would spend a 10 times bigger training budget -
With 10 times more compute, the 2020 paper grows the model 5.4 times and the data 1.9 times; Chinchilla grows both 3.2 times 10× more compute, C ≈ 6 × N × D, split between model size N and data D 2020 (Kaplan) N5.4× bigger model D1.9× more data 2022 (Chinchilla) N3.2× bigger model D3.2× more data In both rows N × D grows 10 times (5.4 × 1.9 ≈ 3.2 × 3.2 ≈ 10); only the split differs.
The same 10× budget, spent the 2020 way and the Chinchilla way.
Over many such steps the difference adds up: following the 2020 advice, models grew much faster than their datasets.

By this measure, the large models of the day were badly undertrained - far too big for the data they had seen. To prove it, DeepMind trained Chinchilla with the same compute budget as their own Gopher model, but split differently -
Gopher has 4 times the parameters of Chinchilla but Chinchilla saw 4.7 times more training tokens, for the same compute Parameters Training tokens Gopher280B Chinchilla70B Gopher300B Chinchilla1.4T Same training compute (about 5.8 × 10²³ FLOPs) Chinchilla outperformed Gopher, GPT-3 and Megatron-Turing NLG on most benchmarks
Gopher vs Chinchilla: a quarter of the parameters, 4.7 times the data.
Chinchilla (70 billion parameters, 1.4 trillion tokens) beat Gopher (280 billion parameters) on nearly every evaluation, including a 7-point improvement on the MMLU knowledge benchmark, where it scored 67.5%. It also beat GPT-3 (175B) and Megatron-Turing NLG (530B). And being four times smaller, it was also much cheaper to run.

A compute-optimal model calculator in Java

With Chinchilla's rule of thumb, D = 20N, the compute formula becomes C = 6 × N × 20N = 120N², so the compute-optimal model size for a budget is N = √(C / 120). The program below applies this to a few compute budgets, and also shows the loss predicted by the 2020 paper's law for some model sizes -
Compute-optimal model size and data for a budget
//ComputeOptimal.java
public class ComputeOptimal {
  static final double TOKENS_PER_PARAM = 20;      // Chinchilla rule of thumb

  static String human(double x) {
    if (x >= 1e12) return String.format("%.1fT", x / 1e12);
    if (x >= 1e9) return String.format("%.0fB", x / 1e9);
    return String.format("%.0fM", x / 1e6);
  }

  public static void main(String[] args) {
    // C = 6 * N * D and D = 20 * N  =>  N = sqrt(C / 120)
    double gpt3 = 6 * 175e9 * 300e9;
    double[] budgets = {1e21, gpt3, 5.76e23, 1e25};
    String[] names = {"1e21 FLOPs", "GPT-3's budget", "Gopher's budget", "1e25 FLOPs"};
    System.out.printf("GPT-3 training compute: %.2e FLOPs%n%n", gpt3);
    System.out.printf("%-16s %-12s %-14s %s%n", "Budget", "FLOPs", "Optimal size", "Optimal tokens");
    for (int i = 0; i < budgets.length; i++) {
      double n = Math.sqrt(budgets[i] / (6 * TOKENS_PER_PARAM));
      System.out.printf("%-16s %-12.2e %-14s %s%n", names[i], budgets[i], human(n), human(TOKENS_PER_PARAM * n));
    }

    // Kaplan et al. (2020): L(N) = (Nc / N)^0.076
    System.out.println();
    for (double n : new double[] {1e8, 1e9, 1e10, 1e11}) {
      System.out.printf("Predicted loss for %-5s parameters: %.3f%n", human(n), Math.pow(8.8e13 / n, 0.076));
    }
  }
}
 OUTPUT of the above code
GPT-3 training compute: 3.15e+23 FLOPs

Budget           FLOPs        Optimal size   Optimal tokens
1e21 FLOPs       1.00e+21     3B             58B
GPT-3's budget   3.15e+23     51B            1.0T
Gopher's budget  5.76e+23     69B            1.4T
1e25 FLOPs       1.00e+25     289B           5.8T

Predicted loss for 100M  parameters: 2.830
Predicted loss for 1B    parameters: 2.376
Predicted loss for 10B   parameters: 1.994
Predicted loss for 100B  parameters: 1.674

Explanation of the code

  • For Gopher's budget, the formula gives about 69 billion parameters and 1.4 trillion tokens - almost exactly the size DeepMind chose for Chinchilla.
  • For GPT-3's budget, the compute-optimal model would have had about 51 billion parameters trained on about 1 trillion tokens, instead of 175 billion parameters on 300 billion tokens.
  • Because N grows with the square root of C, a 100× bigger budget buys only a 10× bigger model - the rest goes into data. This is why training datasets grew from hundreds of billions to tens of trillions of tokens.
  • The predicted losses fall by the same factor, about 0.84, for each 10× increase in parameters - the straight line in the chart above.

After Chinchilla: training past the optimum

"Compute-optimal" means the best model for a fixed training budget. But a popular model is run billions of times after it is trained, and a smaller model is cheaper to run. So many later models deliberately train a smaller model on far more than 20 tokens per parameter - Meta's Llama 3 8B, for example, was trained on over 15 trillion tokens, nearly 2,000 tokens per parameter. The loss keeps falling with more data, just more slowly, and the extra training cost is repaid by cheaper inference.

Mixture-of-experts models bend the formula in another way: only part of their parameters is used for each token, so the N in C ≈ 6ND is the active parameter count, and the model can hold far more parameters than its compute budget suggests - see mixture of expertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts.

Scaling laws have since been found for other axes too - for example, for the compute spent at inference time on reasoning, which is the idea behind models like DeepSeek-R1Click to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL.

Limitations

  • Scaling laws predict the loss, not specific abilities. Some abilities, such as multi-step arithmetic or chain-of-thought reasoningClick to read tutorial: Chain-of-Thought Prompting Paper Explained, appear fairly suddenly at certain scales, which a smooth loss curve does not show.
  • The fitted constants depend on the dataset, the tokenizer and the architecture, and do not carry over exactly to other setups.
  • The laws assume an unlimited supply of good data. High-quality text on the web is finite, which has made data quality, deduplication and synthetic data major research areas.

Summary

In this post we went through the scaling laws paper and the Chinchilla paper. We saw that a language model's test loss falls as a smooth power law in its size, its training data and its training compute, which made the results of very large training runs predictable. We used the formula C ≈ 6ND for training compute, and saw how the answer to "bigger model or more data?" changed from mostly bigger models in 2020 to equal growth - about 20 tokens per parameter - with Chinchilla in 2022. We wrote a Java calculator which reproduces Chinchilla's size from Gopher's budget, and lastly looked at why many models now train past the compute-optimal point, and at the limits of scaling laws. In the next post, we will see what scale made possible, in the GPT-3 paperClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners.