BERT Paper Explained: Bidirectional Transformers

Introduction

A year after the TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained was published, a team at Google showed what its encoder could do when it was first pre-trained on a large amount of plain text. Their model, BERT (Bidirectional Encoder Representations from Transformers), set new state-of-the-art results on eleven language understanding tasks, and made "pre-train, then fine-tune" the standard way to build language models. This post explains the BERT paper with diagrams. We will first see why reading text in both directions matters. We will then go through BERT's two pre-training tasks - the masked language model, whose masking procedure we will simulate in Java, and next sentence prediction - and its input representation. Lastly, we will see how BERT is fine-tuned, its results, and its legacy.
The paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (Google AI Language)
Published: October 2018 - NAACL 2019

Pre-training and fine-tuning

Before BERT, most language models were trained from scratch for each task, on that task's labelled data, which is expensive to collect and usually small. BERT splits training into two stages -
  1. Pre-training: the model learns how language works from a huge amount of unlabelled text - here, BooksCorpus (800 million words) and English Wikipedia (2,500 million words). No human labelling is needed, because the training examples are made from the text itself. This is called self-supervised learning.
  2. Fine-tuning: the pre-trained model is then trained a little further on a small labelled dataset for a specific task, such as sentiment classification or question answering. Almost all the parameters come from pre-training; only a small output layer is new. (For today's much larger models, even this step is often done cheaply with LoRAClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs.)
Pre-training once on unlabelled text gives one BERT model, which is copied and fine-tuned separately for each task 1. Pre-training (once) 2. Fine-tuning (once per task) Unlabelled text Wikipedia + BooksCorpus Pre-trained BERT trained with MLM + NSP BERT + classifier sentiment BERT + span output question answering BERT + token tagger named entities Labelled reviews a few thousand Questions + passages with answer spans Sentences with tagged names copy train on
Pre-training happens once and is expensive; each fine-tuning run is cheap and starts from a copy of the same pre-trained model.
This is transfer learning: knowledge learned once, on cheap data, is reused for many tasks. As the diagram shows, the expensive part happens only once, and every task starts from a copy of the same pre-trained model. OpenAI's first GPT model (2018) had already used this approach, but with a key limitation which BERT removed.

Why bidirectional?

GPT was trained as a language model in the usual sense: predict the next token from the tokens before it. That makes it read strictly left to right, so when it builds a representation of a word, it can only use the words on its left. For understanding tasks this throws away half of the context -
A left-to-right model predicts the masked word only from the words before it; BERT uses the words on both sides Left-to-right (GPT): predicts the blank from the words before it The man went to the [MASK] to buy milk Bidirectional (BERT): uses the words on both sides of the blank The man went to the [MASK] to buy milk
"buy milk" is the strongest clue that the blank is "store" - but a left-to-right model never sees it.
The obvious fix - letting each word see the whole sentence - does not work with ordinary language modelling: if the model can see every word, predicting a word is trivial, because the word itself is visible. BERT's solution is a different pre-training task.

Pre-training task 1: the masked language model

Instead of predicting the next word, BERT hides some words in the input and predicts them from the words around them, on both sides. This is the masked language model (MLM), an idea inspired by the "cloze" fill-in-the-blank test. In each training sequence, 15% of the tokens are chosen for prediction. Each chosen token is then -
  • replaced with a special [MASK] token 80% of the time,
  • replaced with a random token 10% of the time, and
  • left unchanged 10% of the time.
A token chosen for prediction is replaced by [MASK] 80% of the time, by a random token 10% of the time, and left unchanged 10% of the time; the model must predict the original token in every case Chosen token (15%) "store" 80%: replaced by [MASK] ... went to the [MASK] to buy ... 10%: a random token ... went to the red to buy ... 10%: left unchanged ... went to the store to buy ... Must predict "store"
What happens to a token chosen for prediction. In all three cases the model must output the original token.
The model must predict the original token at every chosen position. Why not always use [MASK]? Because [MASK] never appears when the model is later fine-tuned on real text. If every prediction were at a [MASK], the model would learn to build good representations only for masked positions. The random and unchanged tokens force it to keep a good representation of every token, since it never knows which ones it will be asked about. The program below simulates this procedure -
Simulating BERT's masking procedure
//MaskingDemo.java
import java.util.Random;

public class MaskingDemo {
  static final String[] VOCAB = {"the", "a", "dog", "cat", "store", "milk", "ran", "is", "big", "red"};

  public static void main(String[] args) {
    Random random = new Random(3);   // fixed seed, so the output is repeatable
    String[] tokens = "the man went to the store to buy a gallon of milk because the fridge was empty".split(" ");

    StringBuilder input = new StringBuilder(), targets = new StringBuilder();
    for (int i = 0; i < tokens.length; i++) {
      String shown = tokens[i];
      if (random.nextDouble() < 0.15) {                 // choose 15% of tokens for prediction
        double r = random.nextDouble();
        if (r < 0.8) shown = "[MASK]";                  // 80%: mask it
        else if (r < 0.9) shown = VOCAB[random.nextInt(VOCAB.length)];   // 10%: random token
        // else 10%: keep the original token
        targets.append(String.format("  position %2d: model sees %-7s must predict %s%n", i, shown, tokens[i]));
      }
      input.append(shown).append(' ');
    }
    System.out.println("Input : " + input.toString().trim());
    System.out.print(targets);

    // Over many tokens, the proportions match the paper
    int chosen = 0, masked = 0, randomized = 0, kept = 0, total = 1_000_000;
    for (int i = 0; i < total; i++) {
      if (random.nextDouble() < 0.15) {
        chosen++;
        double r = random.nextDouble();
        if (r < 0.8) masked++; else if (r < 0.9) randomized++; else kept++;
      }
    }
    System.out.printf("Of %,d tokens: %.1f%% chosen; of those %.1f%% [MASK], %.1f%% random, %.1f%% unchanged%n",
        total, 100.0 * chosen / total, 100.0 * masked / chosen, 100.0 * randomized / chosen, 100.0 * kept / chosen);
  }
}
 OUTPUT of the above code
Input : the [MASK] went to the store ran buy a gallon [MASK] milk because the fridge was empty
  position  1: model sees [MASK]  must predict man
  position  6: model sees ran     must predict to
  position 10: model sees [MASK]  must predict of
  position 16: model sees empty   must predict empty
Of 1,000,000 tokens: 15.0% chosen; of those 80.0% [MASK], 10.0% random, 10.0% unchanged

Explanation of the code

  • Four of the 17 tokens were chosen. Two became [MASK], "to" was replaced by the random token "ran", and "empty" was left unchanged - yet the model must still predict the original token at all four positions. From the input alone, it cannot tell that "empty" is one of them.
  • Each token is chosen independently with probability 0.15, so a short sentence may have more or fewer than 15% chosen; over a million tokens the proportions settle at the paper's numbers.
  • The model is trained only on the chosen positions - its loss is computed on those predictions, not on every token. This is one cost of the MLM approach: each sequence gives about 15% as many training signals as a left-to-right model, which predicts every token, so BERT needs more training steps.
  • The real BERT works on WordPiece tokens rather than whole words - a vocabulary of 30,000 sub-word units, in which a rare word such as "playing" may be split into "play" and "##ing". How sub-word tokenizers learn their pieces is explained in tokenization: byte pair encoding in JavaClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java.

Pre-training task 2: next sentence prediction

Many tasks, such as question answering and natural language inference, are about the relationship between two pieces of text. To teach this, BERT was also trained on next sentence prediction (NSP): it is given two sentences, A and B, and must say whether B actually followed A in the original text (50% of the examples) or is a random sentence from elsewhere in the corpus (the other 50%).

Note - Later work, notably RoBERTa (2019), found that NSP adds little or even hurts, and that training longer on more data with only MLM gives better results. The MLM idea is BERT's lasting contribution.

The input representation

Every input starts with a special [CLS] token, and the two sentences are separated and ended by [SEP]. The vector given to the Transformer for each token is the sum of three embeddings -
BERT input: token embeddings plus segment embeddings plus position embeddings Input Token Segment Position + + [CLS]mydogiscute[SEP]helikesplay##ing[SEP] E[CLS]EmyEdogEisEcuteE[SEP]EheElikesEplayE##ingE[SEP] EAEAEAEAEAEAEBEBEBEBEB E0E1E2E3E4E5E6E7E8E9E10 Sentence A (green segment) Sentence B (blue segment)
BERT's input representation, using the example from the paper.
  • Token embeddings represent the WordPiece token itself.
  • Segment embeddings tell the model which sentence a token belongs to - A or B.
  • Position embeddings give the token's position. Unlike the original Transformer's fixed sine waves, BERT learns them, for up to 512 positions.
The final vector for [CLS] is used as a summary of the whole input, for example to make the NSP prediction or to classify a sentence.

Model sizes

The paper trained two sizes of the Transformer encoder -
  • BERTBASE - 12 layers, a hidden size of 768, 12 attention heads, 110 million parameters. It was made the same size as OpenAI's GPT to allow a fair comparison.
  • BERTLARGE - 24 layers, a hidden size of 1024, 16 attention heads, 340 million parameters.
BERTLARGE was better on every task, even on tasks with very little labelled data, an early sign that larger pre-trained models generalize better.

Fine-tuning and results

Fine-tuning BERT is simple: add one small output layer and train the whole model on the task's data for a few epochs, which takes at most a few hours on one GPU. As in the pre-training and fine-tuning diagram at the start of this post, the same pre-trained model handles very different tasks -
  • Classification (sentiment, spam, entailment): a classifier on top of the [CLS] vector.
  • Question answering (SQuAD): the question is sentence A, the passage is sentence B, and the model predicts which tokens of the passage start and end the answer.
  • Token tagging (named entity recognition): a classifier on each token's vector.
BERT set new state-of-the-art results on all eleven tasks it was tested on, including a GLUE benchmark score of 80.5 (an absolute improvement of 7.7 points), 86.7% accuracy on MultiNLI, and an F1 score of 93.2 on SQuAD v1.1 - better than the human performance reported for that benchmark.

BERT's legacy

For generating text, however, the field moved to decoder-only models, which predict the next token and can be used directly through prompts, as the GPT-3 paperClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners showed.

Summary

In this post we went through the BERT paper. We saw how BERT splits training into self-supervised pre-training on unlabelled text and fine-tuning on small labelled datasets, and why reading text in both directions gives better representations than reading left to right. We went through its two pre-training tasks - the masked language model, whose 15% and 80/10/10 masking procedure we simulated in Java, and next sentence prediction - and its input representation of token, segment and position embeddings. Lastly, we looked at BERT's model sizes, how it is fine-tuned, its results, and its influence on search and embeddings. In the next post, we will look at the scaling lawsClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained which explain why language models kept getting bigger.