Attention Is All You Need: The Transformer Paper Explained
Introduction
Every large language model in use today - GPT, Claude, Gemini, Llama, DeepSeek - is built on one architecture, the Transformer. It was introduced in 2017 in a paper by eight Google researchers with a memorable title, Attention Is All You Need. This post explains the Transformer paper step by step, with diagrams. We will first see the problems with the recurrent neural networks which came before it. We will then go through the Transformer's encoder-decoder architecture, and its central idea, self-attention, which we will compute by hand in a small Java program. Next we will look at multi-head attention and positional encoding, and lastly at the paper's results and why the architecture took over AI.(Note - This post is part of the Large Language Models category, which explains how today's large language models work, starting with the research papers behind them. The paper posts can be read in the order the papers were published, starting with this one.)
The paper: Attention Is All You Need
Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin (Google Brain, Google Research)
Published: June 2017 - NeurIPS 2017
Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin (Google Brain, Google Research)
Published: June 2017 - NeurIPS 2017
The problem: recurrent networks read one word at a time
In 2017, the best models for language tasks such as machine translation were recurrent neural networks (RNNs), usually a variant called the LSTM. An RNN reads a sentence one token at a time, and carries what it has read so far in a hidden state, which it updates after each token. This has two problems -- No parallelism: the hidden state for word 10 needs the hidden state for word 9, which needs word 8, and so on. The tokens of a sentence must be processed one after another, so training cannot make full use of GPUs, which are built to do many calculations at once.
- Long-range dependencies are hard: information about the first word has to survive many updates of the hidden state to reach the fiftieth word, and much of it fades along the way.
The Transformer architecture
The Transformer was designed for translation, so it has two halves. The encoder reads the source sentence, say in English, and turns each token into a vector which represents its meaning in context. The decoder generates the translation, say in German, one token at a time, using the tokens generated so far and the encoder's output.The Transformer. Each sub-layer is also wrapped in a residual connection and layer normalization ("Add & Norm"), not drawn here.
- Embedding + positional encoding: each token - a word or a piece of a word, see tokenizationClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java - is turned into a vector of 512 numbers, its embedding, and information about its position in the sentence is added to it.
- Encoder layers: the encoder is a stack of 6 identical layers. In each, self-attention lets every token gather information from every other token in the sentence, and a small feed-forward network then processes each token's vector on its own. (Many large models today replace this one network with several smaller "experts" - see mixture of expertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts.)
- Decoder layers: the decoder is also a stack of 6 layers. Its self-attention is masked, so that a token can only look at the tokens before it - the model must not see the words it is about to predict. A second attention sub-layer, encoder-decoder attention, lets the decoder look at the encoder's output for the source sentence.
- Output: a final linear layer and a softmax turn the decoder's output into a probability for every token in the vocabulary. A token is chosen from these probabilities and appended to the output, and the decoder runs again for the next token. (The paper used beam search, which keeps the few most likely partial translations; chat models today usually sample, as explained in temperature, top-k and top-p samplingClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained.)
Scaled dot-product attention
Attention is the heart of the paper. For each token, the model computes three vectors from its embedding, by multiplying it with three learned weight matrices -- a query (Q) - what this token is looking for,
- a key (K) - what this token offers, to be matched against queries, and
- a value (V) - the information this token passes on when it is attended to.
dk, and turned into weights which add up to 1 using softmax (explained in softmax, cross-entropy and KL divergenceClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained). The token's new vector is the weighted sum of all the value vectors. For all tokens at once, in matrix form -
Attention(Q, K, V) = softmax(Q·KT / √dk) · VScaled dot-product attention. The mask hides future tokens in the decoder's self-attention.
√dk? With large vectors, dot products become large, which pushes softmax into regions where one weight is almost 1 and the rest almost 0, and the gradients used for training become tiny. Scaling keeps the scores in a range where softmax still behaves smoothly.
Computing attention in Java
The program below computes self-attention for a three-token sentence, "the cat sat", with 4-number embeddings. Real models learn the embeddings and the projection matricesWq, Wk and Wv during training; here they are fixed small numbers so that the arithmetic can be followed. It also computes the masked version used in the decoder.
Scaled dot-product self-attention in plain Java
//SelfAttention.java
public class SelfAttention {
static double[][] matmul(double[][] a, double[][] b) {
double[][] out = new double[a.length][b[0].length];
for (int i = 0; i < a.length; i++)
for (int j = 0; j < b[0].length; j++)
for (int k = 0; k < b.length; k++)
out[i][j] += a[i][k] * b[k][j];
return out;
}
static double[][] transpose(double[][] m) {
double[][] t = new double[m[0].length][m.length];
for (int i = 0; i < m.length; i++)
for (int j = 0; j < m[0].length; j++)
t[j][i] = m[i][j];
return t;
}
// softmax of each row; masked positions (j > i) get weight 0 when causal is true
static double[][] softmaxRows(double[][] scores, boolean causal) {
double[][] w = new double[scores.length][scores[0].length];
for (int i = 0; i < scores.length; i++) {
double sum = 0;
for (int j = 0; j < scores[i].length; j++) {
w[i][j] = (causal && j > i) ? 0 : Math.exp(scores[i][j]);
sum += w[i][j];
}
for (int j = 0; j < w[i].length; j++) w[i][j] /= sum;
}
return w;
}
static double[][] attentionWeights(double[][] q, double[][] k, boolean causal) {
double[][] scores = matmul(q, transpose(k));
double scale = Math.sqrt(k[0].length);
for (double[] row : scores)
for (int j = 0; j < row.length; j++) row[j] /= scale;
return softmaxRows(scores, causal);
}
static void print(String title, String[] tokens, double[][] m) {
System.out.println(title);
for (int i = 0; i < m.length; i++) {
StringBuilder sb = new StringBuilder(String.format(" %-4s", tokens[i]));
for (double v : m[i]) sb.append(String.format(" %6.3f", v));
System.out.println(sb);
}
}
public static void main(String[] args) {
String[] tokens = {"the", "cat", "sat"};
double[][] x = { // one embedding (row) per token
{1.0, 0.0, 1.0, 0.0},
{0.0, 2.0, 0.0, 2.0},
{1.0, 1.0, 1.0, 1.0}};
double[][] wq = {{1, 0, 0, 0}, {0, 1, 0, 0}, {0, 0, 1, 0}, {0, 0, 0, 1}};
double[][] wk = {{0, 1, 0, 0}, {1, 0, 0, 0}, {0, 0, 0, 1}, {0, 0, 1, 0}};
double[][] wv = {{1, 0, 0, 0}, {0, 1, 0, 0}, {0, 0, 1, 0}, {0, 0, 0, 1}};
double[][] q = matmul(x, wq), k = matmul(x, wk), v = matmul(x, wv);
double[][] weights = attentionWeights(q, k, false);
print("Attention weights (row = token, columns = the, cat, sat):", tokens, weights);
print("Output vectors = weights x V:", tokens, matmul(weights, v));
print("Masked (decoder) attention weights:", tokens, attentionWeights(q, k, true));
}
}
Attention weights (row = token, columns = the, cat, sat): the 0.090 0.665 0.245 cat 0.468 0.063 0.468 sat 0.155 0.422 0.422 Output vectors = weights x V: the 0.335 1.575 0.335 1.575 cat 0.937 0.595 0.937 0.595 sat 0.578 1.267 0.578 1.267 Masked (decoder) attention weights: the 1.000 0.000 0.000 cat 0.881 0.119 0.000 sat 0.155 0.422 0.422
Explanation of the code
- Each row of the weights matrix shows how much one token attends to each token of the sentence, and adds up to 1. With these numbers, "the" puts two-thirds of its attention (0.665) on "cat", whose key best matches its query, while "cat" splits its attention evenly between "the" and "sat".
- Each output vector is a blend of the value vectors, mixed in the proportions given by that token's row of weights. This is how a token's representation absorbs context from the rest of the sentence - in a real model, "bank" would absorb different information in "river bank" and "bank account".
- In the masked version, the upper triangle is 0. "the" can only attend to itself, "cat" to "the" and itself, and "sat" to all three. This is what lets a decoder be trained on whole sentences at once without seeing the future.
- All rows were computed together with matrix multiplications - no token waited for another. This is the parallelism which made Transformers so much faster to train than RNNs.
Multi-head attention
A single attention computation gives each token one blend of the others. But a word relates to its neighbours in several ways at once - grammatically, by meaning, by reference. The paper therefore uses multi-head attention: the model runs 8 attention computations, or heads, in parallel, as the diagram shows.Multi-head attention with the paper's 8 heads. All heads see the same input and run at the same time.
Wq, Wk and Wv, so each one computes a different set of attention weights over the same tokens. With a model width (dmodel) of 512, each head works on vectors of size 512 / 8 = 64. The outputs of the 8 heads are concatenated back into 512 numbers, and one more weight matrix, WO, mixes them. Different heads learn to track different relationships, and because each head is 8 times smaller, the total cost is about the same as one full-size head.
Positional encoding
Attention treats the input as a set: if you shuffled the words of a sentence, every token would get exactly the same attention scores. Word order obviously matters, so the Transformer adds a positional encoding to each token's embedding - a vector which is different for every position. The paper builds it from sine and cosine waves of different frequencies: each dimension of the vector follows its own wave, and a position's vector is the values of all the waves at that position.Three dimensions of the positional encoding. Read at position 5 and at position 11, the waves give two different vectors.
Results
- On the WMT 2014 English-to-German translation benchmark, the large Transformer scored 28.4 BLEU, beating the previous best results, including ensembles of models, by more than 2 BLEU.
- On English-to-French, it reached 41.8 BLEU, a new state of the art for a single model.
- It did this at a fraction of the training cost of earlier models - the large model trained in 3.5 days on 8 GPUs.
- The paper also showed the same architecture working well for English constituency parsing, a hint that it was not specific to translation.
Why the Transformer took over
Because every layer is made of matrix multiplications over all tokens at once, Transformers make full use of GPUs and TPUs, and they keep improving as they are made bigger and trained on more data - a property studied in the scaling laws paperClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained. Within two years, the two halves of the Transformer each became a family of models -- Encoder-only models such as BERTClick to read tutorial: BERT Paper Explained: Bidirectional Transformers, which read text in both directions and are used for understanding tasks such as search and classification.
- Decoder-only models such as GPT, which predict the next token and are used for generating text. All of today's chat assistants are decoder-only Transformers, scaled up to hundreds of billions of parameters, as in GPT-3Click to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners.
Summary
In this post we went through the paper Attention Is All You Need. We saw why recurrent networks were slow to train and struggled with long-range dependencies, and how the Transformer replaced recurrence with attention. We went through the encoder-decoder architecture, computed scaled dot-product attention and its masked form in Java, and saw how multi-head attention and positional encoding complete the design. Lastly, we looked at the paper's translation results and how the Transformer became the foundation of every modern large language model. In the next post, we will look at BERTClick to read tutorial: BERT Paper Explained: Bidirectional Transformers, which showed what a pre-trained Transformer encoder could do.Continue Reading: AI and LLM Papers Explained
BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL
BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL