Chain-of-Thought Prompting Paper Explained

Introduction

GPT-3Click to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners showed that large language models can learn tasks from a few examples in their prompt. But on problems which need several steps of reasoning - maths word problems, for example - even the largest models did poorly, and making them bigger helped surprisingly little. The 2022 paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, from Google, found a remarkably simple fix: write the examples in the prompt so that they show the intermediate reasoning steps, not just the final answers. The model then writes out its own reasoning before answering, and gets far more problems right. This post explains the chain-of-thought prompting paper with diagrams. We will compare standard and chain-of-thought prompts, go through the results on arithmetic, commonsense and symbolic reasoning, see why the technique only works in large models, and look at the follow-up work it inspired - including self-consistency, which we will implement in Java.
The paper: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Authors: Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou (Google Research, Brain Team)
Published: January 2022 - NeurIPS 2022

Standard prompting vs chain-of-thought prompting

A chain of thought is a series of short, natural-language reasoning steps which lead to the final answer - the way a person would work through a problem on paper. In chain-of-thought prompting, each few-shot example in the prompt is a triple: the question, a chain of thought, and the answer. Nothing else changes - the model is not trained or fine-tuned.
With standard prompting the example shows only the answer and the model answers a new question wrongly; with chain-of-thought prompting the example shows reasoning steps and the model reasons its way to the right answer Standard prompting Example in the prompt: Q: A shop had 15 pens. It sold 6 and then got 10 more. How many pens now? A: The answer is 19. New question: Q: A library has 120 books. It lends 45 and receives 30 new ones. How many now? Model output: A: The answer is 195. ✗ Chain-of-thought prompting Example in the prompt: Q: A shop had 15 pens. It sold 6 and then got 10 more. How many pens now? A: 15 - 6 = 9 pens left. 9 + 10 = 19. The answer is 19. New question: Q: A library has 120 books. It lends 45 and receives 30 new ones. How many now? Model output: A: 120 - 45 = 75 books left. 75 + 30 = 105. The answer is 105. ✓ Orange: the chain of thought. The model copies the format of the examples, so a prompt that shows reasoning makes the model reason before answering. (An illustration of the technique, in the style of the paper's examples.)
The same few-shot prompt, with and without reasoning steps in the example.
The paper gives several reasons why writing out the steps helps -
  • More computation for harder problems: a language model does a fixed amount of computation per tokenClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations it generates. Producing the reasoning as extra tokens lets the model spend more computation on problems which need more steps.
  • Decomposition: a multi-step problem is broken into single steps, each of which is easy for the model.
  • Interpretability: the chain shows how the model reached its answer, which helps to find where it went wrong - although the paper notes that a chain of thought is not guaranteed to reflect what the model actually computed.
  • Generality: it works for any task a person could solve by reasoning in language, with no special training data or architecture.

Results

The authors tested chain-of-thought prompting on several model families - GPT-3, LaMDA, PaLM, UL2 and Codex - using 8 hand-written examples for the maths benchmarks. The headline result is on GSM8K, a benchmark of grade-school maths word problems -
PaLM 540B on GSM8K: 17.9 percent with standard prompting, 56.9 percent with chain-of-thought prompting; previous best 55 percent by fine-tuned GPT-3 with a verifier 0%20%40%60%80% 17.9% 56.9% 55% PaLM 540Bstandard prompting PaLM 540Bchain-of-thought Prior bestfine-tuned GPT-3 + verifier
Accuracy on GSM8K, as reported in the paper. Chain-of-thought prompting alone beat a model fine-tuned for the task.
  • Arithmetic reasoning: on GSM8K, PaLM 540B went from 17.9% to 56.9% - a new state of the art, beating a GPT-3 model which had been fine-tuned on the task and paired with a separate verifier model. Large gains were also seen on other maths benchmarks such as SVAMP and MAWPS.
  • Commonsense reasoning: it helped on StrategyQA (questions needing implicit multi-step reasoning), date understanding and sports understanding, where PaLM 540B scored 95.4%, above the 84% of an unaided sports enthusiast.
  • Symbolic reasoning: on toy tasks such as concatenating the last letters of words, or tracking whether a coin is still heads up after people flip it, chain-of-thought prompting let models handle longer inputs than any shown in the examples.

An emergent ability of model scale

The most striking finding is that chain-of-thought prompting only helps large models. For models below roughly 100 billion parameters, it did not help and could even hurt: small models wrote fluent chains of thought which were illogical, and reached wrong answers. Above that size, accuracy jumped. The paper calls this an emergent ability - one which is absent in smaller models and appears at scale, and which the smooth curves of the scaling lawsClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained would not have predicted.

The authors also checked that it is really the reasoning which helps. Prompts with only an equation instead of the reasoning in words, with dots instead of reasoning (to give the model extra tokens without content), or with the reasoning placed after the answer all did much worse than a real chain of thought before the answer.

Follow-up work

Self-consistency in Java

Self-consistency is easy to implement around any model: ask the same question several times, let the model reason differently each time, and keep the answer most chains agree on -
The same question is answered five times with different reasoning; three chains reach 105, two make different mistakes, and the majority vote picks 105 Library questionsampled 5 times,temperature > 0 120 − 45 = 75, 75 + 30 = 105105 lends 45 → 75, receives 30 → 105105 120 + 45 = 165, 165 + 30 = 195195 change −45 + 30 = −15, 120 − 15 = 105105 120 − 45 = 85, 85 + 30 = 115115 Majority vote 105: 3 votes195: 1, 115: 1 answer 105 Correct chains reach the same answer by different routes; the mistakes give different wrong answers.
Self-consistency on the library question: five sampled chains of thought, one vote.
The program below stands in for a model by using these five sampled chains of thought for the library question, as a model sampled with some randomness might produce them. It extracts the final answer from each chain and takes a majority vote -
Majority vote over sampled chains of thought
//SelfConsistency.java
import java.util.*;
import java.util.regex.*;
import java.util.stream.*;

public class SelfConsistency {
  static final Pattern ANSWER = Pattern.compile("The answer is (-?\\d+)");

  static Optional<Integer> finalAnswer(String chain) {
    Matcher m = ANSWER.matcher(chain);
    return m.find() ? Optional.of(Integer.parseInt(m.group(1))) : Optional.empty();
  }

  public static void main(String[] args) {
    // Five chains of thought sampled for: "A library has 120 books. It lends 45 and receives 30. How many now?"
    List<String> chains = List.of(
        "120 - 45 = 75 books left. 75 + 30 = 105. The answer is 105.",
        "It lends 45, so 120 - 45 = 75. Then it receives 30: 75 + 30 = 105. The answer is 105.",
        "120 + 45 = 165, and 165 + 30 = 195. The answer is 195.",
        "Change is -45 + 30 = -15. 120 - 15 = 105. The answer is 105.",
        "120 - 45 = 85. 85 + 30 = 115. The answer is 115.");

    Map<Integer, Long> votes = chains.stream()
        .map(SelfConsistency::finalAnswer)
        .flatMap(Optional::stream)
        .collect(Collectors.groupingBy(a -> a, TreeMap::new, Collectors.counting()));
    System.out.println("Votes per answer: " + votes);

    int majority = Collections.max(votes.entrySet(), Map.Entry.comparingByValue()).getKey();
    System.out.println("Had we sampled only chain 3: " + finalAnswer(chains.get(2)).orElseThrow());
    System.out.println("Self-consistency answer: " + majority);
  }
}
 OUTPUT of the above code
Votes per answer: {105=3, 115=1, 195=1}
Had we sampled only chain 3: 195
Self-consistency answer: 105

Explanation of the code

  • Each chain ends with "The answer is N" - the format taught by the examples in the prompt - which makes the final answer easy to extract with a regular expression.
  • The three correct chains reach 105 by different routes, while the two wrong ones make different mistakes (adding instead of subtracting, and a subtraction error) and so disagree with each other. The vote picks 105.
  • With a real model, the chains would come from calling the model several times with a sampling temperatureClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained above 0. The cost is one model call per chain, which is the price of the extra accuracy.

Summary

In this post we went through the chain-of-thought prompting paper. We saw that putting reasoning steps into the few-shot examples of a prompt makes a large language model reason step by step before answering, and why this helps - more computation, decomposition and interpretability. We went through the results on arithmetic, commonsense and symbolic reasoning, including PaLM 540B's jump from 17.9% to 56.9% on GSM8K, and the finding that the technique only works in models of around 100 billion parameters or more. Lastly, we looked at the follow-up work, from "Let's think step by step" to reasoning models, and implemented self-consistency's majority vote in Java. In the next post, we will see how GPT-3 was trained to follow instructions, in the InstructGPT paperClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF.