LLM Temperature, Top-k and Top-p Sampling Explained

Introduction

Send the same prompt to a large language model twice, and you may get two different answers. Set one parameter to 0, and the answers become nearly identical. That parameter is the temperature, and with top-k and top-p it controls how the model chooses each word it writes. This post explains LLM sampling with diagrams: how a model turns a prompt into probabilities for the next token, greedy decoding, what temperature does to those probabilities, how top-k and top-p cut off unlikely tokens - all computed in a Java program on a real-shaped example - and which settings suit which tasks.

(Note - Tokens, and the loop in which a model writes one token at a time, are explained in LLM tokens, context windows, embeddings and hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations.)

From scores to a chosen token

A language model writes one token at a time. For each step, its last layer produces a score, called a logit, for every token in its vocabularyClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java - tens of thousands of numbers. The softmax function turns these scores into probabilities that add up to 1, and a decoding strategy then picks one token, which is appended to the text before the next step. Temperature, top-k and top-p are settings of that last part -
The model outputs logits; they are divided by the temperature, turned into probabilities by softmax, filtered by top-k and top-p, and one token is sampled and appended to the text Text so far"The coffee was" Model→ logits ÷ Ttemperature softmax→ probabilities top-k / top-pdrop unlikely ones samplepick one "hot" append the token and repeat for the next one The orange steps are the sampling settings; the model itself is the same for every setting.
How one token is chosen. Temperature, top-k and top-p only change the orange steps.
Softmax is the same function that turns attention scores into weights in the TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained: it raises e to the power of each score and divides by the total, so higher scores get much larger shares - see softmax, cross-entropy and KL divergenceClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained.

Greedy decoding

The simplest strategy, greedy decoding, always picks the most likely token. It gives the same answer for the same prompt, which is useful for extraction, classification and tests. For longer text it has a known weakness: always taking the single most likely word makes text bland and prone to repeating itself, because the most likely continuation of a repeated phrase is often the phrase again. Setting the temperature to 0 means greedy decoding in most APIs. Even then, hosted models are not always perfectly deterministic, because the order of floating-point calculations on shared hardware can vary slightly between requests.

Temperature

Sampling picks the next token at random, in proportion to its probability: a token with probability 0.4 is chosen about 40% of the time. The temperature T reshapes the probabilities first, by dividing every logit by T before the softmax -
pi = exp(zi / T) / Σj exp(zj / T)
A temperature below 1 makes the differences between logits bigger, so the likely tokens become even more likely; above 1 it makes them smaller, so unlikely tokens get a real chance. For the next token after "The coffee was", with the logits from the program below -
At temperature 0.5 hot gets 0.65; at 1.0 0.43; at 1.5 0.34, while the unlikely tokens gain probability temperature 0.5 0.65 hot 0.24 strong 0.09 delicious 0.01 cold 0.01 bitter 0.00 perfect 0.00 purple temperature 1.0 0.43 hot 0.26 strong 0.16 delicious 0.06 cold 0.05 bitter 0.04 perfect 0.00 purple temperature 1.5 0.34 hot 0.24 strong 0.17 delicious 0.09 cold 0.08 bitter 0.06 perfect 0.01 purple
The same seven logits at three temperatures, as computed by the program below.
At temperature 0.5, "hot" gets 65% of the probability and the last four tokens almost none: answers are focused and repeatable. At 1.5, "hot" falls to 34%, and even "purple" - a token which makes no sense here - gets a small chance. That is the trade-off temperature controls: higher temperatures give more varied text, and more chances of a poor choice.

Top-k and top-p

Pure sampling has a problem: the vocabulary has tens of thousands of tokens, and together the many unlikely ones hold a noticeable share of the probability. Sooner or later one of them is picked, and the text goes off course. Two filters remove the tail before sampling -
  • Top-k keeps only the k most likely tokens. It is simple, but k is fixed: when the model is sure, k = 50 still lets 49 poor candidates in; when many words fit, a small k cuts off good ones.
  • Top-p, or nucleus sampling (Holtzman et al., 2019), keeps the most likely tokens until their probabilities add up to p, such as 0.9. The number of candidates adapts: few when the model is confident, many when it is not.
Top-k 3 keeps hot, strong and delicious; top-p 0.9 keeps hot, strong, delicious and cold, whose probabilities add up to 0.91 top-k = 3: keep the 3 most likely 0.43 hot 0.26 strong 0.16 delicious 0.06 cold 0.05 bitter 0.04 perfect 0.00 purple always exactly 3 candidates top-p = 0.9: keep until they add up to 0.9 0.43 hot 0.26 strong 0.16 delicious 0.06 cold 0.05 bitter 0.04 perfect 0.00 purple 0.43 + 0.26 + 0.16 + 0.06 = 0.91
Top-k and top-p applied to the temperature-1.0 probabilities. Grey tokens can no longer be chosen; the kept ones are rescaled to add up to 1.
Top-p 0.9 kept four tokens here because "cold" was needed to reach 0.9; on a step where the model is sure of one word, the same setting would keep only that word. In practice, change either the temperature or top-p, not both at once - their effects overlap, and adjusting both makes results hard to reason about.

Temperature, top-k and top-p in Java

The program below implements the whole pipeline for one step: softmax with temperature, top-k and top-p filtering, and sampling. It prints the probabilities, then draws ten next tokens for each setting with the same random seed, so that the settings can be compared.
Softmax with temperature, top-k, top-p and sampling
//NextTokenSampling.java
import java.util.*;

public class NextTokenSampling {

  // The model's raw scores (logits) for the next token after "The coffee was" - made up, but shaped like real ones
  static final String[] TOKENS = {"hot", "strong", "delicious", "cold", "bitter", "perfect", "purple"};
  static final double[] LOGITS = {4.0, 3.5, 3.0, 2.0, 1.8, 1.5, -1.0};

  // Softmax with temperature: divide every logit by T, then turn the scores into probabilities that add up to 1
  static double[] softmax(double[] logits, double temperature) {
    double max = Arrays.stream(logits).max().getAsDouble();
    double[] p = new double[logits.length];
    double sum = 0;
    for (int i = 0; i < p.length; i++) {
      p[i] = Math.exp((logits[i] - max) / temperature);   // subtracting max keeps exp() from overflowing
      sum += p[i];
    }
    for (int i = 0; i < p.length; i++) p[i] /= sum;
    return p;
  }

  // Top-k: keep the k most likely tokens; top-p: keep the most likely tokens until their probabilities add up to p.
  // Everything else gets probability 0, and the rest is renormalized.
  static double[] filter(double[] probs, int topK, double topP) {
    Integer[] order = new Integer[probs.length];
    for (int i = 0; i < order.length; i++) order[i] = i;
    Arrays.sort(order, (a, b) -> Double.compare(probs[b], probs[a]));
    double[] kept = new double[probs.length];
    double cumulative = 0, sum = 0;
    for (int rank = 0; rank < order.length; rank++) {
      int i = order[rank];
      if (rank >= topK || cumulative >= topP) break;
      kept[i] = probs[i];
      cumulative += probs[i];
      sum += probs[i];
    }
    for (int i = 0; i < kept.length; i++) kept[i] /= sum;
    return kept;
  }

  static int sample(double[] probs, Random random) {
    double r = random.nextDouble(), cumulative = 0;
    for (int i = 0; i < probs.length; i++) {
      cumulative += probs[i];
      if (r < cumulative) return i;
    }
    return probs.length - 1;
  }

  static void show(String label, double[] probs) {
    StringBuilder sb = new StringBuilder(String.format("%-24s", label));
    for (int i = 0; i < TOKENS.length; i++) sb.append(String.format(" %s=%.2f", TOKENS[i], probs[i]));
    System.out.println(sb);
  }

  public static void main(String[] args) {
    System.out.println("Probabilities for the next token after \"The coffee was\":");
    for (double t : new double[] {0.5, 1.0, 1.5}) show("temperature " + t, softmax(LOGITS, t));
    double[] base = softmax(LOGITS, 1.0);
    show("top-k 3", filter(base, 3, 1.0));
    show("top-p 0.9", filter(base, TOKENS.length, 0.9));

    System.out.println();
    System.out.println("Ten sampled continuations for each setting (fixed seed):");
    Object[][] settings = {{"greedy (temperature 0)", 0.0, 7, 1.0}, {"temperature 0.5", 0.5, 7, 1.0},
        {"temperature 1.0", 1.0, 7, 1.0}, {"temperature 1.5", 1.5, 7, 1.0}, {"temperature 1.5, top-p 0.9", 1.5, 7, 0.9}};
    for (Object[] s : settings) {
      double t = (double) s[1];
      Random random = new Random(7);
      StringBuilder sb = new StringBuilder(String.format("%-28s", s[0]));
      Set<String> distinct = new TreeSet<>();
      for (int n = 0; n < 10; n++) {
        String token;
        if (t == 0) {                                     // greedy: always the most likely token
          int best = 0;
          for (int i = 1; i < LOGITS.length; i++) if (LOGITS[i] > LOGITS[best]) best = i;
          token = TOKENS[best];
        } else {
          token = TOKENS[sample(filter(softmax(LOGITS, t), (int) s[2], (double) s[3]), random)];
        }
        sb.append(token).append(' ');
        distinct.add(token);
      }
      System.out.println(sb.toString().trim() + "   (" + distinct.size() + " different)");
    }
  }
}
 OUTPUT of the above code
Probabilities for the next token after "The coffee was":
temperature 0.5          hot=0.65 strong=0.24 delicious=0.09 cold=0.01 bitter=0.01 perfect=0.00 purple=0.00
temperature 1.0          hot=0.43 strong=0.26 delicious=0.16 cold=0.06 bitter=0.05 perfect=0.04 purple=0.00
temperature 1.5          hot=0.34 strong=0.24 delicious=0.17 cold=0.09 bitter=0.08 perfect=0.06 purple=0.01
top-k 3                  hot=0.51 strong=0.31 delicious=0.19 cold=0.00 bitter=0.00 perfect=0.00 purple=0.00
top-p 0.9                hot=0.47 strong=0.29 delicious=0.17 cold=0.06 bitter=0.00 perfect=0.00 purple=0.00

Ten sampled continuations for each setting (fixed seed):
greedy (temperature 0)      hot hot hot hot hot hot hot hot hot hot   (1 different)
temperature 0.5             strong strong hot delicious strong hot hot strong hot delicious   (3 different)
temperature 1.0             delicious delicious hot cold delicious hot hot delicious hot bitter   (4 different)
temperature 1.5             delicious delicious strong bitter delicious strong hot bitter hot perfect   (5 different)
temperature 1.5, top-p 0.9  delicious delicious hot cold delicious hot hot cold hot bitter   (4 different)

Explanation of the code

  • softmax() divides each logit by the temperature before the exponent. Subtracting the largest logit first does not change the result, but keeps Math.exp() from overflowing - real implementations do the same.
  • After top-k or top-p, the kept probabilities are divided by their sum, so they add up to 1 again: with top-k 3, "hot" rose from 0.43 to 0.51.
  • Greedy decoding chose "hot" ten times out of ten. Temperature 0.5 produced 3 different tokens, 1.0 produced 4 and 1.5 produced 5, including "perfect", which had only a 6% chance.
  • Temperature 1.5 with top-p 0.9 gave 4 different tokens and never "perfect" or "purple": top-p removed the tail which the high temperature had raised. This combination - some randomness, but no unlikely tokens - is what top-p is for.
  • All runs used the same seed, so they are repeatable. APIs which accept a seed parameter use it the same way, to make sampled answers more reproducible.

Choosing the settings

TaskTypical settingWhy
Extraction, classification, structured outputClick to read tutorial: Prompt Engineering Techniques Explained with Examples, codeTemperature 0 to 0.3There is one right answer; variation only adds errors
Chat assistants, explanations, summariesTemperature around 0.7, or the provider's defaultNatural-sounding text with few odd word choices
Brainstorming, stories, marketing copyTemperature around 1, top-p 0.9 to 0.95Variety is the goal; top-p keeps it sensible
Several samples for voting, as in self-consistencyClick to read tutorial: Chain-of-Thought Prompting Paper ExplainedTemperature above 0, e.g. 0.7Different reasoning paths are needed to vote on
Three practical notes - Other common settings work on the same loop: a maximum number of output tokens ends generation after a limit, stop sequences end it when a given text appears, and frequency or presence penalties lower the logits of tokens which have already been used, to reduce repetition.

Summary

In this post we went through how a large language model chooses each token. We saw that the model produces a logit for every token, softmax turns the logits into probabilities, and a decoding strategy picks one token. Greedy decoding always takes the most likely token; sampling picks at random by probability, with the temperature making the distribution sharper below 1 and flatter above 1. Top-k keeps a fixed number of candidates and top-p keeps just enough to reach a probability mass, which adapts to how sure the model is. In Java, we computed all of this for one step and compared ten sampled tokens per setting, and lastly we went through which settings suit which tasks. How to write the prompt that the model continues is explained in prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples.