Softmax, Cross-Entropy and KL Divergence Explained
Introduction
Three small formulas appear again and again in the papers behind large language models: softmax, which turns a model's scores into probabilities, cross-entropy, the loss that pre-training minimizes, and KL divergence, which measures how far one model's probabilities have moved from another's and keeps RLHF and reasoning training in check. This post explains all three with diagrams and a Java program which computes them on a next-word example: what each formula does, where it appears in training an LLM, how loss and perplexity are related, why KL divergence is not a distance, and how the three fit together. No maths beyond logarithms is needed.(Note - How a model turns probabilities into the words it writes is explained in temperature, top-k and top-p samplingClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained.)
Where the three formulas appear
The three formulas in the life of a language model.
Softmax: from scores to probabilities
A model's last layer produces a score, a logit zi, for every token of its vocabulary. Softmax turns them into probabilities -pi = ezi / Σj ezjCross-entropy: the training loss
During pre-training, the real next token of the training text is known. Cross-entropy loss scores the model's probabilities against it -loss = −ln p(correct token)The cross-entropy loss for one token, with the four stages from the program below.
H(p, q) = −Σ pi ln qi. For a training token, p is 1 for the real token and 0 for all others, which leaves just −ln q(correct token). The cross-entropy of a distribution with itself, H(p) = −Σ pi ln pi, is called its entropy: the uncertainty that is in the data itself, which no model can remove. BERT'sClick to read tutorial: BERT Paper Explained: Bidirectional Transformers masked language model uses the same loss, computed only at the masked positions.Perplexity, often reported for language models, is the exponent of the average loss,
eloss. It reads as the number of tokens the model is effectively choosing between: a perplexity of 4 is as uncertain as a fair choice among 4 tokens.
KL divergence: how far a model has moved
Kullback-Leibler (KL) divergence measures how different one probability distribution q is from another, p -KL(q ‖ p) = Σ qi ln (qi / pi)The distributions from the program below, with their KL divergence from the reference.
Computing all three in Java
The program below follows one next-word prediction, "The cat sat on the ___", through four stages of training, and then compares a reference model's distribution with two trained models.Softmax, cross-entropy, perplexity and KL divergence
//LossFunctions.java
import java.util.*;
public class LossFunctions {
static double[] softmax(double[] logits) {
double max = Arrays.stream(logits).max().getAsDouble(), sum = 0;
double[] p = new double[logits.length];
for (int i = 0; i < p.length; i++) sum += p[i] = Math.exp(logits[i] - max);
for (int i = 0; i < p.length; i++) p[i] /= sum;
return p;
}
// Cross-entropy of a distribution q against the true next token: -ln q(true token)
static double crossEntropy(double[] q, int target) {
return -Math.log(q[target]);
}
// Entropy H(p) = -sum p ln p, and cross-entropy H(p, q) = -sum p ln q, for two whole distributions
static double entropy(double[] p) {
double h = 0;
for (double x : p) if (x > 0) h -= x * Math.log(x);
return h;
}
static double crossEntropy(double[] p, double[] q) {
double h = 0;
for (int i = 0; i < p.length; i++) if (p[i] > 0) h -= p[i] * Math.log(q[i]);
return h;
}
// KL divergence KL(p || q) = sum p ln(p / q): how much q differs from p, measured where p puts its probability
static double kl(double[] p, double[] q) {
double d = 0;
for (int i = 0; i < p.length; i++) if (p[i] > 0) d += p[i] * Math.log(p[i] / q[i]);
return d;
}
static String fmt(double[] p) {
StringBuilder sb = new StringBuilder("[");
for (int i = 0; i < p.length; i++) sb.append(i == 0 ? "" : ", ").append(String.format("%.3f", p[i]));
return sb.append("]").toString();
}
public static void main(String[] args) {
String[] vocab = {"mat", "sofa", "roof", "moon"};
int target = 0; // the real next word after "The cat sat on the" is "mat"
System.out.println("1) Softmax and cross-entropy for the next word after \"The cat sat on the\" (true word: mat)");
double[][] stages = {{0, 0, 0, 0}, {1.0, 0.5, 0.2, -1.0}, {3.0, 1.0, 0.0, -2.0}, {6.0, 2.0, 1.0, -3.0}};
String[] names = {"untrained", "early", "later", "trained"};
for (int s = 0; s < stages.length; s++) {
double[] q = softmax(stages[s]);
double loss = crossEntropy(q, target);
System.out.printf(" %-9s logits=%-22s p=%s p(mat)=%.3f loss=%.3f perplexity=%.2f%n",
names[s], Arrays.toString(stages[s]), fmt(q), q[target], loss, Math.exp(loss));
}
System.out.println();
System.out.println("2) KL divergence between a reference model and a model being trained");
double[] reference = {0.50, 0.30, 0.15, 0.05};
double[] close = {0.55, 0.28, 0.13, 0.04};
double[] drifted = {0.90, 0.08, 0.01, 0.01};
System.out.printf(" reference %s%n", fmt(reference));
System.out.printf(" close %s KL(close || reference) = %.4f%n", fmt(close), kl(close, reference));
System.out.printf(" drifted %s KL(drifted || reference) = %.4f%n", fmt(drifted), kl(drifted, reference));
System.out.printf(" same %s KL(reference || reference) = %.4f%n", fmt(reference), kl(reference, reference));
System.out.printf(" not symmetric: KL(reference || drifted) = %.4f%n", kl(reference, drifted));
System.out.println();
System.out.println("3) Cross-entropy = entropy + KL divergence");
System.out.printf(" H(reference, drifted) = %.4f%n", crossEntropy(reference, drifted));
System.out.printf(" H(reference) + KL(reference || drifted) = %.4f + %.4f = %.4f%n",
entropy(reference), kl(reference, drifted), entropy(reference) + kl(reference, drifted));
}
}
1) Softmax and cross-entropy for the next word after "The cat sat on the" (true word: mat) untrained logits=[0.0, 0.0, 0.0, 0.0] p=[0.250, 0.250, 0.250, 0.250] p(mat)=0.250 loss=1.386 perplexity=4.00 early logits=[1.0, 0.5, 0.2, -1.0] p=[0.456, 0.277, 0.205, 0.062] p(mat)=0.456 loss=0.784 perplexity=2.19 later logits=[3.0, 1.0, 0.0, -2.0] p=[0.839, 0.114, 0.042, 0.006] p(mat)=0.839 loss=0.176 perplexity=1.19 trained logits=[6.0, 2.0, 1.0, -3.0] p=[0.975, 0.018, 0.007, 0.000] p(mat)=0.975 loss=0.025 perplexity=1.03 2) KL divergence between a reference model and a model being trained reference [0.500, 0.300, 0.150, 0.050] close [0.550, 0.280, 0.130, 0.040] KL(close || reference) = 0.0056 drifted [0.900, 0.080, 0.010, 0.010] KL(drifted || reference) = 0.3801 same [0.500, 0.300, 0.150, 0.050] KL(reference || reference) = 0.0000 not symmetric: KL(reference || drifted) = 0.5893 3) Cross-entropy = entropy + KL divergence H(reference, drifted) = 1.7314 H(reference) + KL(reference || drifted) = 1.1421 + 0.5893 = 1.7314
Explanation of the code
- Untrained, all logits are equal, so softmax spreads the probability evenly: 0.25 each. The loss is ln 4 = 1.386 and the perplexity exactly 4 - the model is choosing blindly among four words.
- As training raises the logit of "mat", its probability rises to 0.456, 0.839 and 0.975, and the loss falls to 0.784, 0.176 and 0.025. The last steps change the probability little, but each still lowers the loss.
- The slightly changed model has a KL divergence of only 0.0056 from the reference; the drifted model, which put 0.90 on one word, has 0.3801. This is the difference a KL penalty is designed to notice.
- The reverse direction, KL(reference ‖ drifted), is 0.5893 - not 0.3801. The drifted model gives "roof" and "moon" almost no probability, and KL in that direction punishes heavily any token the reference finds likely but the other model does not.
- The last lines check the identity which connects the three: cross-entropy = entropy + KL divergence, 1.1421 + 0.5893 = 1.7314. The entropy of the data is fixed, so minimizing cross-entropy during training is the same as minimizing the KL divergence between the model and the data.
Summary
In this post we went through the three formulas behind training large language models. Softmax turns logits into probabilities, and only the differences between logits matter. Cross-entropy, −ln of the probability given to the correct token, is the loss of pre-training, and perplexity is its exponent. KL divergence measures how far a model's distribution has moved from a reference, which RLHF, GRPO and distillation use to keep training in check, and it is not symmetric. In Java we computed all three, and checked that cross-entropy equals entropy plus KL divergence. In the next post, we will look at how large models save compute with mixture of expertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts.Continue Reading: AI and LLM Papers Explained
Mixture of Experts (MoE) Explained: Routers and ExpertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts Cosine Similarity vs Dot Product for EmbeddingsClick to read tutorial: Cosine Similarity vs Dot Product for Embeddings Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained
Mixture of Experts (MoE) Explained: Routers and ExpertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts Cosine Similarity vs Dot Product for EmbeddingsClick to read tutorial: Cosine Similarity vs Dot Product for Embeddings Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained