Knowledge Distillation Explained: Teacher and Student Models

Introduction

The best language models are large, slow and expensive to run; many applications need something small and fast. Knowledge distillation trains a small student model to imitate a large teacher model, so that the student learns more than it would from the training data alone. It is how DistilBERT came out 40% smaller than BERT while keeping most of its accuracy, and how DeepSeek put the reasoning of its 671-billion-parameter R1 into models as small as 1.5 billion parameters. This post explains knowledge distillation with diagrams: the teacher-student setup, why the teacher's probabilities teach more than the correct answer alone - shown in a Java program - the role of temperature, the two ways distillation is done for LLMs, and its limits.

(Note - Distillation is built on softmax, cross-entropy and KL divergence, which are explained in softmax, cross-entropy and KL divergenceClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained.)

Teacher and student

In normal training, a model learns from hard labels: for each example, the one correct answer. In distillation, the student also learns from the teacher's output for the same example - its full probability distribution over all possible answers, called soft targets -
The same input goes to a frozen large teacher and a small student; the student's loss combines the KL divergence from the teacher's softened probabilities with the cross-entropy against the hard label, and only the student is updated Training inputa photo, a prompt Teacherlarge, already trainedfrozen Studentsmall, being trainedupdated soft targetscat 0.725, lynx 0.208,dog 0.059, car 0.008 student'sprobabilities LossKL(teacher ‖ student)+ cross-entropyvs the hard label gradients update the student only
Knowledge distillation: the student learns from the teacher's probabilities as well as from the correct answer.
The method was set out by Hinton, Vinyals and Dean in 2015 in "Distilling the Knowledge in a Neural Network". The student's loss is usually a mix of two parts: the KL divergenceClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained between the teacher's and the student's distributions, and the ordinary cross-entropy against the hard label -
loss = α · T² · KL(teacherT ‖ studentT) + (1 − α) · cross-entropy(student, hard label)
Here teacherT and studentT are the two models' probabilities at temperature T (explained next), and α, between 0 and 1, sets how much the student listens to the teacher rather than to the label.

Why soft targets teach more

A hard label says only "this photo is a cat". The teacher's probabilities say much more: that it is a cat, that it looks somewhat like a lynx, much less like a dog, and nothing like a car. Hinton called this information in the small probabilities dark knowledge. It tells the student how the classes relate to each other - something it would otherwise have to discover from far more data.

The teacher is usually very confident, so these small probabilities are tiny. Distillation therefore reads both models' softmax at a temperature T above 1, the same temperature as in samplingClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained: dividing the logits by T flattens the distribution and makes the differences between the unlikely answers visible -
A hard label puts all probability on cat; the teacher's softmax at temperature 1, 2 and 4 spreads more and more probability to lynx, dog and car, in that order hard label 1.000cat 0.000lynx 0.000dog 0.000car teacher, T = 1 0.918cat 0.075lynx 0.006dog 0.000car teacher, T = 2 0.725cat 0.208lynx 0.059dog 0.008car teacher, T = 4 0.519cat 0.278lynx 0.149dog 0.055car The soft targets keep the order lynx > dog > car - knowledge which the hard label throws away.
The training targets for one photo of a cat, from the program below.
At T = 1, "car" gets 0.000 and the order of the wrong answers is almost invisible; at T = 2 and T = 4, it stands out. Because softening also shrinks the gradients, Hinton et al. multiply the soft-target loss by T² to keep its weight balanced against the hard-label loss.

Why not simply train the small model on the same data as the teacher? Two reasons make distillation worth it. First, a soft target is a much richer signal than a hard label: every example tells the student about all the answers, not just the right one, so it learns more from the same data. Second, a teacher can produce training data - a model which writes thousands of worked answers is a better source of examples than raw web text. Training a small model for much longer than the compute-optimal amount, as described in the scaling laws postClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained, is the other way to get a strong small model; distillation improves what each training example gives it.

Distilling a classifier in Java

The program below trains a student's four logits by gradient descent, once on the hard label and once on the teacher's probabilities at temperature 2, and then reads both students at temperature 1. The gradient of cross-entropy with respect to the logits is, up to a constant factor, the student's probabilities minus the target (at temperature T the factor is 1/T, which the step size absorbs here), which keeps the training loop to two lines.
Training a student on hard labels and on a teacher's soft targets
//Distillation.java
import java.util.*;

public class Distillation {
  static final String[] CLASSES = {"cat", "lynx", "dog", "car"};

  static double[] softmax(double[] logits, double temperature) {
    double max = Arrays.stream(logits).max().getAsDouble(), sum = 0;
    double[] p = new double[logits.length];
    for (int i = 0; i < p.length; i++) sum += p[i] = Math.exp((logits[i] - max) / temperature);
    for (int i = 0; i < p.length; i++) p[i] /= sum;
    return p;
  }

  // Trains the student's logits towards a target distribution with gradient descent on cross-entropy.
  // The gradient of cross-entropy with respect to the logits is (student probabilities - target), up to a factor 1/T
  // which the step size absorbs here.
  static double[] train(double[] target, double temperature, int steps) {
    double[] logits = new double[CLASSES.length];          // the student starts knowing nothing
    for (int step = 0; step < steps; step++) {
      double[] p = softmax(logits, temperature);
      for (int i = 0; i < logits.length; i++) logits[i] -= 0.5 * (p[i] - target[i]);
    }
    return logits;
  }

  static String fmt(double[] p) {
    StringBuilder sb = new StringBuilder();
    for (int i = 0; i < p.length; i++) sb.append(String.format("%s=%.3f ", CLASSES[i], p[i]));
    return sb.toString().trim();
  }

  public static void main(String[] args) {
    // The teacher's logits for a photo of a cat: it is fairly sure, and it knows a lynx looks more like a cat than a car does
    double[] teacher = {6.0, 3.5, 1.0, -3.0};
    double[] hardLabel = {1, 0, 0, 0};

    System.out.println("Targets for one training example (a photo of a cat):");
    System.out.println("  hard label            " + fmt(hardLabel));
    for (double t : new double[] {1, 2, 4}) System.out.println("  teacher, T = " + (int) t + "          " + fmt(softmax(teacher, t)));

    System.out.println();
    System.out.println("Student after 300 training steps, read at temperature 1:");
    System.out.println("  trained on the hard label     " + fmt(softmax(train(hardLabel, 1, 300), 1)));
    double[] soft = softmax(teacher, 2);
    System.out.println("  trained on teacher at T = 2   " + fmt(softmax(train(soft, 2, 300), 1)));
  }
}
 OUTPUT of the above code
Targets for one training example (a photo of a cat):
  hard label            cat=1.000 lynx=0.000 dog=0.000 car=0.000
  teacher, T = 1          cat=0.918 lynx=0.075 dog=0.006 car=0.000
  teacher, T = 2          cat=0.725 lynx=0.208 dog=0.059 car=0.008
  teacher, T = 4          cat=0.519 lynx=0.278 dog=0.149 car=0.055

Student after 300 training steps, read at temperature 1:
  trained on the hard label     cat=0.995 lynx=0.002 dog=0.002 car=0.002
  trained on teacher at T = 2   cat=0.919 lynx=0.074 dog=0.006 car=0.000

Explanation of the code

  • The teacher's targets at temperatures 1, 2 and 4 keep the same order - cat, lynx, dog, car - while spreading more probability to the wrong answers as T rises.
  • The student trained on the hard label became confident about "cat" (0.995), but gave lynx, dog and car the same 0.002 each: from one hard label, it cannot learn that a lynx is more cat-like than a car.
  • The student trained on the teacher at T = 2 ended with cat 0.919, lynx 0.074, dog 0.006 and car 0.000 - almost exactly the teacher's own T = 1 distribution of 0.918, 0.075, 0.006 and 0.000. This is no accident: matching the teacher's softened distribution at T = 2 forces the student's logits to match the teacher's (up to a constant), so read at T = 1 it reproduces the teacher's view of how the classes relate.
  • Here the student is just four numbers for one example; in real distillation, the student is a full network trained on many examples, and this extra information per example is what lets it reach the teacher's accuracy with less data and fewer parameters.

Two ways to distil a language model

For LLMs, every next-token prediction is a classification over the model's whole vocabulary of tokensClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java, so the same idea applies. In practice it is done in two ways -
Logit distillation trains the student on the teacher's full next-token probabilities at every position, which needs access to the teacher's outputs; sequence-level distillation has the teacher write answers, and the student is fine-tuned on that text Logit (soft-label) distillation Teacherreads the text full probabilities over the vocabularyat every position Student matches themKL loss, e.g. DistilBERT Sequence-level distillation Teacherwrites answers a dataset of generated textfiltered for quality Student fine-tuned on ite.g. DeepSeek-R1-Distill The second needs only the teacher's text, so it also works through an API - where the provider's terms allow it.
Two ways of distilling a large language model.

Limits and practical points

Summary

In this post we went through knowledge distillation. We saw how a small student model is trained to match a large teacher's probability distribution as well as the correct answers, and why those soft targets carry more information - Hinton's dark knowledge - than hard labels. We saw that a temperature above 1 brings out the differences between the unlikely answers, and in Java a student trained on a teacher's soft targets learned the teacher's view of which wrong answers are close, which one trained on the hard label could not. Lastly, we went through logit and sequence-level distillation, with DistilBERT and DeepSeek-R1's distilled models as examples, and the limits and terms to keep in mind. Distillation makes the model itself smaller; quantizationClick to read tutorial: LLM Quantization Explained: 8-bit, 4-bit and GGUF shrinks the numbers inside a model, and the two are often used together. In the next post, we will look at how to choose between fine-tuning, RAG and prompt engineeringClick to read tutorial: Fine-Tuning vs RAG vs Prompt Engineering: Which to Use.