BERT Paper Explained: Bidirectional Transformers
Introduction
A year after the TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained was published, a team at Google showed what its encoder could do when it was first pre-trained on a large amount of plain text. Their model, BERT (Bidirectional Encoder Representations from Transformers), set new state-of-the-art results on eleven language understanding tasks, and made "pre-train, then fine-tune" the standard way to build language models. This post explains the BERT paper with diagrams. We will first see why reading text in both directions matters. We will then go through BERT's two pre-training tasks - the masked language model, whose masking procedure we will simulate in Java, and next sentence prediction - and its input representation. Lastly, we will see how BERT is fine-tuned, its results, and its legacy.The paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (Google AI Language)
Published: October 2018 - NAACL 2019
Authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (Google AI Language)
Published: October 2018 - NAACL 2019
Pre-training and fine-tuning
Before BERT, most language models were trained from scratch for each task, on that task's labelled data, which is expensive to collect and usually small. BERT splits training into two stages -- Pre-training: the model learns how language works from a huge amount of unlabelled text - here, BooksCorpus (800 million words) and English Wikipedia (2,500 million words). No human labelling is needed, because the training examples are made from the text itself. This is called self-supervised learning.
- Fine-tuning: the pre-trained model is then trained a little further on a small labelled dataset for a specific task, such as sentiment classification or question answering. Almost all the parameters come from pre-training; only a small output layer is new. (For today's much larger models, even this step is often done cheaply with LoRAClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs.)
Pre-training happens once and is expensive; each fine-tuning run is cheap and starts from a copy of the same pre-trained model.
Why bidirectional?
GPT was trained as a language model in the usual sense: predict the next token from the tokens before it. That makes it read strictly left to right, so when it builds a representation of a word, it can only use the words on its left. For understanding tasks this throws away half of the context -"buy milk" is the strongest clue that the blank is "store" - but a left-to-right model never sees it.
Pre-training task 1: the masked language model
Instead of predicting the next word, BERT hides some words in the input and predicts them from the words around them, on both sides. This is the masked language model (MLM), an idea inspired by the "cloze" fill-in-the-blank test. In each training sequence, 15% of the tokens are chosen for prediction. Each chosen token is then -- replaced with a special
[MASK]token 80% of the time, - replaced with a random token 10% of the time, and
- left unchanged 10% of the time.
What happens to a token chosen for prediction. In all three cases the model must output the original token.
[MASK]? Because [MASK] never appears when the model is later fine-tuned on real text. If every prediction were at a [MASK], the model would learn to build good representations only for masked positions. The random and unchanged tokens force it to keep a good representation of every token, since it never knows which ones it will be asked about. The program below simulates this procedure -
Simulating BERT's masking procedure
//MaskingDemo.java
import java.util.Random;
public class MaskingDemo {
static final String[] VOCAB = {"the", "a", "dog", "cat", "store", "milk", "ran", "is", "big", "red"};
public static void main(String[] args) {
Random random = new Random(3); // fixed seed, so the output is repeatable
String[] tokens = "the man went to the store to buy a gallon of milk because the fridge was empty".split(" ");
StringBuilder input = new StringBuilder(), targets = new StringBuilder();
for (int i = 0; i < tokens.length; i++) {
String shown = tokens[i];
if (random.nextDouble() < 0.15) { // choose 15% of tokens for prediction
double r = random.nextDouble();
if (r < 0.8) shown = "[MASK]"; // 80%: mask it
else if (r < 0.9) shown = VOCAB[random.nextInt(VOCAB.length)]; // 10%: random token
// else 10%: keep the original token
targets.append(String.format(" position %2d: model sees %-7s must predict %s%n", i, shown, tokens[i]));
}
input.append(shown).append(' ');
}
System.out.println("Input : " + input.toString().trim());
System.out.print(targets);
// Over many tokens, the proportions match the paper
int chosen = 0, masked = 0, randomized = 0, kept = 0, total = 1_000_000;
for (int i = 0; i < total; i++) {
if (random.nextDouble() < 0.15) {
chosen++;
double r = random.nextDouble();
if (r < 0.8) masked++; else if (r < 0.9) randomized++; else kept++;
}
}
System.out.printf("Of %,d tokens: %.1f%% chosen; of those %.1f%% [MASK], %.1f%% random, %.1f%% unchanged%n",
total, 100.0 * chosen / total, 100.0 * masked / chosen, 100.0 * randomized / chosen, 100.0 * kept / chosen);
}
}
Input : the [MASK] went to the store ran buy a gallon [MASK] milk because the fridge was empty position 1: model sees [MASK] must predict man position 6: model sees ran must predict to position 10: model sees [MASK] must predict of position 16: model sees empty must predict empty Of 1,000,000 tokens: 15.0% chosen; of those 80.0% [MASK], 10.0% random, 10.0% unchanged
Explanation of the code
- Four of the 17 tokens were chosen. Two became
[MASK], "to" was replaced by the random token "ran", and "empty" was left unchanged - yet the model must still predict the original token at all four positions. From the input alone, it cannot tell that "empty" is one of them. - Each token is chosen independently with probability 0.15, so a short sentence may have more or fewer than 15% chosen; over a million tokens the proportions settle at the paper's numbers.
- The model is trained only on the chosen positions - its loss is computed on those predictions, not on every token. This is one cost of the MLM approach: each sequence gives about 15% as many training signals as a left-to-right model, which predicts every token, so BERT needs more training steps.
- The real BERT works on WordPiece tokens rather than whole words - a vocabulary of 30,000 sub-word units, in which a rare word such as "playing" may be split into "play" and "##ing". How sub-word tokenizers learn their pieces is explained in tokenization: byte pair encoding in JavaClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java.
Pre-training task 2: next sentence prediction
Many tasks, such as question answering and natural language inference, are about the relationship between two pieces of text. To teach this, BERT was also trained on next sentence prediction (NSP): it is given two sentences, A and B, and must say whether B actually followed A in the original text (50% of the examples) or is a random sentence from elsewhere in the corpus (the other 50%).Note - Later work, notably RoBERTa (2019), found that NSP adds little or even hurts, and that training longer on more data with only MLM gives better results. The MLM idea is BERT's lasting contribution.
The input representation
Every input starts with a special[CLS] token, and the two sentences are separated and ended by [SEP]. The vector given to the Transformer for each token is the sum of three embeddings -
BERT's input representation, using the example from the paper.
- Token embeddings represent the WordPiece token itself.
- Segment embeddings tell the model which sentence a token belongs to - A or B.
- Position embeddings give the token's position. Unlike the original Transformer's fixed sine waves, BERT learns them, for up to 512 positions.
[CLS] is used as a summary of the whole input, for example to make the NSP prediction or to classify a sentence.
Model sizes
The paper trained two sizes of the Transformer encoder -- BERTBASE - 12 layers, a hidden size of 768, 12 attention heads, 110 million parameters. It was made the same size as OpenAI's GPT to allow a fair comparison.
- BERTLARGE - 24 layers, a hidden size of 1024, 16 attention heads, 340 million parameters.
Fine-tuning and results
Fine-tuning BERT is simple: add one small output layer and train the whole model on the task's data for a few epochs, which takes at most a few hours on one GPU. As in the pre-training and fine-tuning diagram at the start of this post, the same pre-trained model handles very different tasks -- Classification (sentiment, spam, entailment): a classifier on top of the
[CLS]vector. - Question answering (SQuAD): the question is sentence A, the passage is sentence B, and the model predicts which tokens of the passage start and end the answer.
- Token tagging (named entity recognition): a classifier on each token's vector.
BERT's legacy
- Search: in 2019 Google began using BERT in Google Search to better understand queries.
- Embeddings: BERT-style encoders, such as Sentence-BERT, became the standard way to turn text into embeddingsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations - vectors whose closeness reflects closeness in meaning, measured as explained in cosine similarity vs dot productClick to read tutorial: Cosine Similarity vs Dot Product for Embeddings. These power semantic search and the retrieval step of Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation, where BERT-style cross-encoders also re-rank the results (see RAG architecture explainedClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval).
- A family of models: RoBERTa, ALBERT, DistilBERT, ELECTRA and DeBERTa all refined BERT's recipe.
Summary
In this post we went through the BERT paper. We saw how BERT splits training into self-supervised pre-training on unlabelled text and fine-tuning on small labelled datasets, and why reading text in both directions gives better representations than reading left to right. We went through its two pre-training tasks - the masked language model, whose 15% and 80/10/10 masking procedure we simulated in Java, and next sentence prediction - and its input representation of token, segment and position embeddings. Lastly, we looked at BERT's model sizes, how it is fine-tuned, its results, and its influence on search and embeddings. In the next post, we will look at the scaling lawsClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained which explain why language models kept getting bigger.Continue Reading: AI and LLM Papers Explained
Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL
Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL