Softmax, Cross-Entropy and KL Divergence Explained

Introduction

Three small formulas appear again and again in the papers behind large language models: softmax, which turns a model's scores into probabilities, cross-entropy, the loss that pre-training minimizes, and KL divergence, which measures how far one model's probabilities have moved from another's and keeps RLHF and reasoning training in check. This post explains all three with diagrams and a Java program which computes them on a next-word example: what each formula does, where it appears in training an LLM, how loss and perplexity are related, why KL divergence is not a distance, and how the three fit together. No maths beyond logarithms is needed.

(Note - How a model turns probabilities into the words it writes is explained in temperature, top-k and top-p samplingClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained.)

Where the three formulas appear

The model outputs logits, softmax turns them into probabilities; in pre-training, cross-entropy compares them with the real next token; in RLHF and RL for reasoning, KL divergence compares them with a frozen reference model's probabilities Model→ logits softmax→ probabilities cross-entropyvs the real next tokenpre-training, fine-tuning KL divergencevs a frozen reference modelRLHF, GRPO, distillation "mat"the training text referenceprobabilities Softmax runs in every step; cross-entropy and KL divergence turn its output into a training signal.
The three formulas in the life of a language model.
Softmax also appears inside every attention layer of the TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained, where it turns attention scores into weights. Each formula is short; the sections below take them one at a time.

Softmax: from scores to probabilities

A model's last layer produces a score, a logit zi, for every token of its vocabulary. Softmax turns them into probabilities -
pi = ezi / Σj ezj
Raising e to each score makes every value positive and stretches the gaps between scores, and dividing by the total makes the values add up to 1. Only differences between logits matter: adding the same number to every logit gives the same probabilities, which is why implementations subtract the largest logit first, to keep the exponents from overflowing.

Cross-entropy: the training loss

During pre-training, the real next token of the training text is known. Cross-entropy loss scores the model's probabilities against it -
loss = −ln p(correct token)
If the model gave the correct token probability 1, the loss would be 0; the lower the probability, the higher the loss, without limit. Training adjusts the model's weights to lower the average loss over billions of tokens - that is, to give the real next tokens ever higher probability. This average is the "test loss" of the scaling lawsClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained. The curve shows how sharply the loss punishes a low probability -
The loss -ln p falls steeply as the probability of the correct token rises: 1.386 at 0.250, 0.784 at 0.456, 0.176 at 0.839 and 0.025 at 0.975 0 1 2 3 4 0 0.25 0.5 0.75 1 probability the model gives the correct next token loss = −ln p 1 1 untrained: p = 0.250, loss 1.386 2 2 early: p = 0.456, loss 0.784 3 3 later: p = 0.839, loss 0.176 4 4 trained: p = 0.975, loss 0.025
The cross-entropy loss for one token, with the four stages from the program below.
The general form compares two whole distributions, a true one p and the model's q: H(p, q) = −Σ pi ln qi. For a training token, p is 1 for the real token and 0 for all others, which leaves just −ln q(correct token). The cross-entropy of a distribution with itself, H(p) = −Σ pi ln pi, is called its entropy: the uncertainty that is in the data itself, which no model can remove. BERT'sClick to read tutorial: BERT Paper Explained: Bidirectional Transformers masked language model uses the same loss, computed only at the masked positions.

Perplexity, often reported for language models, is the exponent of the average loss, eloss. It reads as the number of tokens the model is effectively choosing between: a perplexity of 4 is as uncertain as a fair choice among 4 tokens.

KL divergence: how far a model has moved

Kullback-Leibler (KL) divergence measures how different one probability distribution q is from another, p -
KL(q ‖ p) = Σ qi ln (qi / pi)
It is 0 when the two are identical and grows as they differ. In RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF, the model being trained is q and the frozen supervised model is p: a KL penalty is subtracted from the reward, so the model improves its answers without drifting so far from its starting point that it learns to fool the reward model. GRPO in DeepSeek-R1Click to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL uses a KL term in the same way, and distillation trains a small model to bring its distribution close to a large one's -
A model close to the reference has a KL divergence of 0.0056; a model which has moved most probability onto one token has 0.3801 reference p 0.50mat 0.30sofa 0.15roof 0.05moon close q: KL = 0.0056 0.55mat 0.28sofa 0.13roof 0.04moon drifted q: KL = 0.3801 0.90mat 0.08sofa 0.01roof 0.01moon A KL penalty lets a model move a little, as in the middle, and makes moving far, as on the right, expensive.
The distributions from the program below, with their KL divergence from the reference.
KL divergence is not symmetric: KL(q ‖ p) and KL(p ‖ q) are generally different, so it is not a distance in the everyday sense. Which direction a method uses matters, and the papers state it.

Computing all three in Java

The program below follows one next-word prediction, "The cat sat on the ___", through four stages of training, and then compares a reference model's distribution with two trained models.
Softmax, cross-entropy, perplexity and KL divergence
//LossFunctions.java
import java.util.*;

public class LossFunctions {

  static double[] softmax(double[] logits) {
    double max = Arrays.stream(logits).max().getAsDouble(), sum = 0;
    double[] p = new double[logits.length];
    for (int i = 0; i < p.length; i++) sum += p[i] = Math.exp(logits[i] - max);
    for (int i = 0; i < p.length; i++) p[i] /= sum;
    return p;
  }

  // Cross-entropy of a distribution q against the true next token: -ln q(true token)
  static double crossEntropy(double[] q, int target) {
    return -Math.log(q[target]);
  }

  // Entropy H(p) = -sum p ln p, and cross-entropy H(p, q) = -sum p ln q, for two whole distributions
  static double entropy(double[] p) {
    double h = 0;
    for (double x : p) if (x > 0) h -= x * Math.log(x);
    return h;
  }

  static double crossEntropy(double[] p, double[] q) {
    double h = 0;
    for (int i = 0; i < p.length; i++) if (p[i] > 0) h -= p[i] * Math.log(q[i]);
    return h;
  }

  // KL divergence KL(p || q) = sum p ln(p / q): how much q differs from p, measured where p puts its probability
  static double kl(double[] p, double[] q) {
    double d = 0;
    for (int i = 0; i < p.length; i++) if (p[i] > 0) d += p[i] * Math.log(p[i] / q[i]);
    return d;
  }

  static String fmt(double[] p) {
    StringBuilder sb = new StringBuilder("[");
    for (int i = 0; i < p.length; i++) sb.append(i == 0 ? "" : ", ").append(String.format("%.3f", p[i]));
    return sb.append("]").toString();
  }

  public static void main(String[] args) {
    String[] vocab = {"mat", "sofa", "roof", "moon"};
    int target = 0;   // the real next word after "The cat sat on the" is "mat"

    System.out.println("1) Softmax and cross-entropy for the next word after \"The cat sat on the\" (true word: mat)");
    double[][] stages = {{0, 0, 0, 0}, {1.0, 0.5, 0.2, -1.0}, {3.0, 1.0, 0.0, -2.0}, {6.0, 2.0, 1.0, -3.0}};
    String[] names = {"untrained", "early", "later", "trained"};
    for (int s = 0; s < stages.length; s++) {
      double[] q = softmax(stages[s]);
      double loss = crossEntropy(q, target);
      System.out.printf("  %-9s logits=%-22s p=%s  p(mat)=%.3f  loss=%.3f  perplexity=%.2f%n",
          names[s], Arrays.toString(stages[s]), fmt(q), q[target], loss, Math.exp(loss));
    }

    System.out.println();
    System.out.println("2) KL divergence between a reference model and a model being trained");
    double[] reference = {0.50, 0.30, 0.15, 0.05};
    double[] close = {0.55, 0.28, 0.13, 0.04};
    double[] drifted = {0.90, 0.08, 0.01, 0.01};
    System.out.printf("  reference %s%n", fmt(reference));
    System.out.printf("  close     %s  KL(close || reference)   = %.4f%n", fmt(close), kl(close, reference));
    System.out.printf("  drifted   %s  KL(drifted || reference) = %.4f%n", fmt(drifted), kl(drifted, reference));
    System.out.printf("  same      %s  KL(reference || reference) = %.4f%n", fmt(reference), kl(reference, reference));
    System.out.printf("  not symmetric: KL(reference || drifted) = %.4f%n", kl(reference, drifted));

    System.out.println();
    System.out.println("3) Cross-entropy = entropy + KL divergence");
    System.out.printf("  H(reference, drifted) = %.4f%n", crossEntropy(reference, drifted));
    System.out.printf("  H(reference) + KL(reference || drifted) = %.4f + %.4f = %.4f%n",
        entropy(reference), kl(reference, drifted), entropy(reference) + kl(reference, drifted));
  }
}
 OUTPUT of the above code
1) Softmax and cross-entropy for the next word after "The cat sat on the" (true word: mat)
  untrained logits=[0.0, 0.0, 0.0, 0.0]   p=[0.250, 0.250, 0.250, 0.250]  p(mat)=0.250  loss=1.386  perplexity=4.00
  early     logits=[1.0, 0.5, 0.2, -1.0]  p=[0.456, 0.277, 0.205, 0.062]  p(mat)=0.456  loss=0.784  perplexity=2.19
  later     logits=[3.0, 1.0, 0.0, -2.0]  p=[0.839, 0.114, 0.042, 0.006]  p(mat)=0.839  loss=0.176  perplexity=1.19
  trained   logits=[6.0, 2.0, 1.0, -3.0]  p=[0.975, 0.018, 0.007, 0.000]  p(mat)=0.975  loss=0.025  perplexity=1.03

2) KL divergence between a reference model and a model being trained
  reference [0.500, 0.300, 0.150, 0.050]
  close     [0.550, 0.280, 0.130, 0.040]  KL(close || reference)   = 0.0056
  drifted   [0.900, 0.080, 0.010, 0.010]  KL(drifted || reference) = 0.3801
  same      [0.500, 0.300, 0.150, 0.050]  KL(reference || reference) = 0.0000
  not symmetric: KL(reference || drifted) = 0.5893

3) Cross-entropy = entropy + KL divergence
  H(reference, drifted) = 1.7314
  H(reference) + KL(reference || drifted) = 1.1421 + 0.5893 = 1.7314

Explanation of the code

  • Untrained, all logits are equal, so softmax spreads the probability evenly: 0.25 each. The loss is ln 4 = 1.386 and the perplexity exactly 4 - the model is choosing blindly among four words.
  • As training raises the logit of "mat", its probability rises to 0.456, 0.839 and 0.975, and the loss falls to 0.784, 0.176 and 0.025. The last steps change the probability little, but each still lowers the loss.
  • The slightly changed model has a KL divergence of only 0.0056 from the reference; the drifted model, which put 0.90 on one word, has 0.3801. This is the difference a KL penalty is designed to notice.
  • The reverse direction, KL(reference ‖ drifted), is 0.5893 - not 0.3801. The drifted model gives "roof" and "moon" almost no probability, and KL in that direction punishes heavily any token the reference finds likely but the other model does not.
  • The last lines check the identity which connects the three: cross-entropy = entropy + KL divergence, 1.1421 + 0.5893 = 1.7314. The entropy of the data is fixed, so minimizing cross-entropy during training is the same as minimizing the KL divergence between the model and the data.

Summary

In this post we went through the three formulas behind training large language models. Softmax turns logits into probabilities, and only the differences between logits matter. Cross-entropy, −ln of the probability given to the correct token, is the loss of pre-training, and perplexity is its exponent. KL divergence measures how far a model's distribution has moved from a reference, which RLHF, GRPO and distillation use to keep training in check, and it is not symmetric. In Java we computed all three, and checked that cross-entropy equals entropy plus KL divergence. In the next post, we will look at how large models save compute with mixture of expertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts.