Attention Is All You Need: The Transformer Paper Explained

Introduction

Every large language model in use today - GPT, Claude, Gemini, Llama, DeepSeek - is built on one architecture, the Transformer. It was introduced in 2017 in a paper by eight Google researchers with a memorable title, Attention Is All You Need. This post explains the Transformer paper step by step, with diagrams. We will first see the problems with the recurrent neural networks which came before it. We will then go through the Transformer's encoder-decoder architecture, and its central idea, self-attention, which we will compute by hand in a small Java program. Next we will look at multi-head attention and positional encoding, and lastly at the paper's results and why the architecture took over AI.

(Note - This post is part of the Large Language Models category, which explains how today's large language models work, starting with the research papers behind them. The paper posts can be read in the order the papers were published, starting with this one.)
The paper: Attention Is All You Need
Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin (Google Brain, Google Research)
Published: June 2017 - NeurIPS 2017

The problem: recurrent networks read one word at a time

In 2017, the best models for language tasks such as machine translation were recurrent neural networks (RNNs), usually a variant called the LSTM. An RNN reads a sentence one token at a time, and carries what it has read so far in a hidden state, which it updates after each token. This has two problems -
  • No parallelism: the hidden state for word 10 needs the hidden state for word 9, which needs word 8, and so on. The tokens of a sentence must be processed one after another, so training cannot make full use of GPUs, which are built to do many calculations at once.
  • Long-range dependencies are hard: information about the first word has to survive many updates of the hidden state to reach the fiftieth word, and much of it fades along the way.
Researchers had already added an attention mechanism to RNNs (Bahdanau et al., 2014), which let the translation model look back directly at any word of the source sentence. The Transformer paper's insight was radical: drop the recurrence entirely, and build the whole model out of attention. Every token can then look at every other token directly, and all tokens can be processed in parallel.

The Transformer architecture

The Transformer was designed for translation, so it has two halves. The encoder reads the source sentence, say in English, and turns each token into a vector which represents its meaning in context. The decoder generates the translation, say in German, one token at a time, using the tokens generated so far and the encoder's output.
The Transformer architecture: an encoder stack on the left and a decoder stack on the right Encoder layer × 6 Feed-forward network Multi-head self-attention Embedding + positional encoding Input tokens (source sentence) Decoder layer × 6 Feed-forward network Encoder-decoder attention Masked multi-head self-attention Embedding + positional encoding Output tokens so far (shifted right) Linear + softmax → next token encoder output (keys and values)
The Transformer. Each sub-layer is also wrapped in a residual connection and layer normalization ("Add & Norm"), not drawn here.
  • Embedding + positional encoding: each token - a word or a piece of a word, see tokenizationClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java - is turned into a vector of 512 numbers, its embedding, and information about its position in the sentence is added to it.
  • Encoder layers: the encoder is a stack of 6 identical layers. In each, self-attention lets every token gather information from every other token in the sentence, and a small feed-forward network then processes each token's vector on its own. (Many large models today replace this one network with several smaller "experts" - see mixture of expertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts.)
  • Decoder layers: the decoder is also a stack of 6 layers. Its self-attention is masked, so that a token can only look at the tokens before it - the model must not see the words it is about to predict. A second attention sub-layer, encoder-decoder attention, lets the decoder look at the encoder's output for the source sentence.
  • Output: a final linear layer and a softmax turn the decoder's output into a probability for every token in the vocabulary. A token is chosen from these probabilities and appended to the output, and the decoder runs again for the next token. (The paper used beam search, which keeps the few most likely partial translations; chat models today usually sample, as explained in temperature, top-k and top-p samplingClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained.)

Scaled dot-product attention

Attention is the heart of the paper. For each token, the model computes three vectors from its embedding, by multiplying it with three learned weight matrices -
  • a query (Q) - what this token is looking for,
  • a key (K) - what this token offers, to be matched against queries, and
  • a value (V) - the information this token passes on when it is attended to.
A token's query is compared with the key of every token using a dot product - the same measure often used to compare embeddings, see cosine similarity vs dot productClick to read tutorial: Cosine Similarity vs Dot Product for Embeddings - which gives a score of how relevant each token is. The scores are divided by the square root of the key size, dk, and turned into weights which add up to 1 using softmax (explained in softmax, cross-entropy and KL divergenceClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained). The token's new vector is the weighted sum of all the value vectors. For all tokens at once, in matrix form -
Attention(Q, K, V) = softmax(Q·KT / √dk) · V
Scaled dot-product attention: Q times K transposed, scaled, optionally masked, softmaxed, then multiplied by V Q K V Q · KT ÷ √dk Mask(decoder only) softmax × V out weights values
Scaled dot-product attention. The mask hides future tokens in the decoder's self-attention.
Why divide by √dk? With large vectors, dot products become large, which pushes softmax into regions where one weight is almost 1 and the rest almost 0, and the gradients used for training become tiny. Scaling keeps the scores in a range where softmax still behaves smoothly.

Computing attention in Java

The program below computes self-attention for a three-token sentence, "the cat sat", with 4-number embeddings. Real models learn the embeddings and the projection matrices Wq, Wk and Wv during training; here they are fixed small numbers so that the arithmetic can be followed. It also computes the masked version used in the decoder.
Scaled dot-product self-attention in plain Java
//SelfAttention.java
public class SelfAttention {

  static double[][] matmul(double[][] a, double[][] b) {
    double[][] out = new double[a.length][b[0].length];
    for (int i = 0; i < a.length; i++)
      for (int j = 0; j < b[0].length; j++)
        for (int k = 0; k < b.length; k++)
          out[i][j] += a[i][k] * b[k][j];
    return out;
  }

  static double[][] transpose(double[][] m) {
    double[][] t = new double[m[0].length][m.length];
    for (int i = 0; i < m.length; i++)
      for (int j = 0; j < m[0].length; j++)
        t[j][i] = m[i][j];
    return t;
  }

  // softmax of each row; masked positions (j > i) get weight 0 when causal is true
  static double[][] softmaxRows(double[][] scores, boolean causal) {
    double[][] w = new double[scores.length][scores[0].length];
    for (int i = 0; i < scores.length; i++) {
      double sum = 0;
      for (int j = 0; j < scores[i].length; j++) {
        w[i][j] = (causal && j > i) ? 0 : Math.exp(scores[i][j]);
        sum += w[i][j];
      }
      for (int j = 0; j < w[i].length; j++) w[i][j] /= sum;
    }
    return w;
  }

  static double[][] attentionWeights(double[][] q, double[][] k, boolean causal) {
    double[][] scores = matmul(q, transpose(k));
    double scale = Math.sqrt(k[0].length);
    for (double[] row : scores)
      for (int j = 0; j < row.length; j++) row[j] /= scale;
    return softmaxRows(scores, causal);
  }

  static void print(String title, String[] tokens, double[][] m) {
    System.out.println(title);
    for (int i = 0; i < m.length; i++) {
      StringBuilder sb = new StringBuilder(String.format("  %-4s", tokens[i]));
      for (double v : m[i]) sb.append(String.format(" %6.3f", v));
      System.out.println(sb);
    }
  }

  public static void main(String[] args) {
    String[] tokens = {"the", "cat", "sat"};
    double[][] x = {            // one embedding (row) per token
        {1.0, 0.0, 1.0, 0.0},
        {0.0, 2.0, 0.0, 2.0},
        {1.0, 1.0, 1.0, 1.0}};
    double[][] wq = {{1, 0, 0, 0}, {0, 1, 0, 0}, {0, 0, 1, 0}, {0, 0, 0, 1}};
    double[][] wk = {{0, 1, 0, 0}, {1, 0, 0, 0}, {0, 0, 0, 1}, {0, 0, 1, 0}};
    double[][] wv = {{1, 0, 0, 0}, {0, 1, 0, 0}, {0, 0, 1, 0}, {0, 0, 0, 1}};

    double[][] q = matmul(x, wq), k = matmul(x, wk), v = matmul(x, wv);

    double[][] weights = attentionWeights(q, k, false);
    print("Attention weights (row = token, columns = the, cat, sat):", tokens, weights);
    print("Output vectors = weights x V:", tokens, matmul(weights, v));
    print("Masked (decoder) attention weights:", tokens, attentionWeights(q, k, true));
  }
}
 OUTPUT of the above code
Attention weights (row = token, columns = the, cat, sat):
  the   0.090  0.665  0.245
  cat   0.468  0.063  0.468
  sat   0.155  0.422  0.422
Output vectors = weights x V:
  the   0.335  1.575  0.335  1.575
  cat   0.937  0.595  0.937  0.595
  sat   0.578  1.267  0.578  1.267
Masked (decoder) attention weights:
  the   1.000  0.000  0.000
  cat   0.881  0.119  0.000
  sat   0.155  0.422  0.422

Explanation of the code

  • Each row of the weights matrix shows how much one token attends to each token of the sentence, and adds up to 1. With these numbers, "the" puts two-thirds of its attention (0.665) on "cat", whose key best matches its query, while "cat" splits its attention evenly between "the" and "sat".
  • Each output vector is a blend of the value vectors, mixed in the proportions given by that token's row of weights. This is how a token's representation absorbs context from the rest of the sentence - in a real model, "bank" would absorb different information in "river bank" and "bank account".
  • In the masked version, the upper triangle is 0. "the" can only attend to itself, "cat" to "the" and itself, and "sat" to all three. This is what lets a decoder be trained on whole sentences at once without seeing the future.
  • All rows were computed together with matrix multiplications - no token waited for another. This is the parallelism which made Transformers so much faster to train than RNNs.

Multi-head attention

A single attention computation gives each token one blend of the others. But a word relates to its neighbours in several ways at once - grammatically, by meaning, by reference. The paper therefore uses multi-head attention: the model runs 8 attention computations, or heads, in parallel, as the diagram shows.
Multi-head attention: eight heads compute attention in parallel on 64-number vectors, their outputs are concatenated and mixed by W_O Input: a 512-number vector per token Head 1 own Wq, Wk, Wv attention on 64 dims Head 2 own Wq, Wk, Wv attention on 64 dims . . . Head 8 own Wq, Wk, Wv attention on 64 dims Concatenate: 8 heads × 64 = 512 numbers Multiply by WO to mix the heads Output: a 512-number vector per token
Multi-head attention with the paper's 8 heads. All heads see the same input and run at the same time.
Each head has its own learned Wq, Wk and Wv, so each one computes a different set of attention weights over the same tokens. With a model width (dmodel) of 512, each head works on vectors of size 512 / 8 = 64. The outputs of the 8 heads are concatenated back into 512 numbers, and one more weight matrix, WO, mixes them. Different heads learn to track different relationships, and because each head is 8 times smaller, the total cost is about the same as one full-size head.

Positional encoding

Attention treats the input as a set: if you shuffled the words of a sentence, every token would get exactly the same attention scores. Word order obviously matters, so the Transformer adds a positional encoding to each token's embedding - a vector which is different for every position. The paper builds it from sine and cosine waves of different frequencies: each dimension of the vector follows its own wave, and a position's vector is the values of all the waves at that position.
Positional encoding: each dimension is a wave of a different frequency, and a position's vector is the values of all waves at that position dimension 0 (fast) dimension 2 dimension 4 (slow) position 5 position 5 → [-0.96, 0.92, 0.61, ...] position 11 position 11 → [-1.00, -0.92, 0.99, ...] token position in the sentence →
Three dimensions of the positional encoding. Read at position 5 and at position 11, the waves give two different vectors.
The fast wave alone repeats quickly - it gives almost the same value at positions 5 and 11 - but the slower waves differ there, so the combination of all of them is unique for every position. The fast waves tell neighbouring positions apart, and the slow ones tell apart positions far from each other. Because the waves are regular, the encoding of position 11 can be computed from that of position 5 by a fixed transformation, which makes it easy for the model to learn relative distances such as "6 tokens earlier". (Many later models instead learn the position vectors, or use rotary position embeddings, RoPE, which rotate the queries and keys according to position.)

Results

  • On the WMT 2014 English-to-German translation benchmark, the large Transformer scored 28.4 BLEU, beating the previous best results, including ensembles of models, by more than 2 BLEU.
  • On English-to-French, it reached 41.8 BLEU, a new state of the art for a single model.
  • It did this at a fraction of the training cost of earlier models - the large model trained in 3.5 days on 8 GPUs.
  • The paper also showed the same architecture working well for English constituency parsing, a hint that it was not specific to translation.

Why the Transformer took over

Because every layer is made of matrix multiplications over all tokens at once, Transformers make full use of GPUs and TPUs, and they keep improving as they are made bigger and trained on more data - a property studied in the scaling laws paperClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained. Within two years, the two halves of the Transformer each became a family of models - The architecture has since spread beyond text, to images (Vision Transformers), audio, protein structure and more.

Summary

In this post we went through the paper Attention Is All You Need. We saw why recurrent networks were slow to train and struggled with long-range dependencies, and how the Transformer replaced recurrence with attention. We went through the encoder-decoder architecture, computed scaled dot-product attention and its masked form in Java, and saw how multi-head attention and positional encoding complete the design. Lastly, we looked at the paper's translation results and how the Transformer became the foundation of every modern large language model. In the next post, we will look at BERTClick to read tutorial: BERT Paper Explained: Bidirectional Transformers, which showed what a pre-trained Transformer encoder could do.

Attention Is All You Need: The Transformer Paper Explained - JavaBrahman