LLM Temperature, Top-k and Top-p Sampling Explained
Introduction
Send the same prompt to a large language model twice, and you may get two different answers. Set one parameter to 0, and the answers become nearly identical. That parameter is the temperature, and with top-k and top-p it controls how the model chooses each word it writes. This post explains LLM sampling with diagrams: how a model turns a prompt into probabilities for the next token, greedy decoding, what temperature does to those probabilities, how top-k and top-p cut off unlikely tokens - all computed in a Java program on a real-shaped example - and which settings suit which tasks.(Note - Tokens, and the loop in which a model writes one token at a time, are explained in LLM tokens, context windows, embeddings and hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations.)
From scores to a chosen token
A language model writes one token at a time. For each step, its last layer produces a score, called a logit, for every token in its vocabularyClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java - tens of thousands of numbers. The softmax function turns these scores into probabilities that add up to 1, and a decoding strategy then picks one token, which is appended to the text before the next step. Temperature, top-k and top-p are settings of that last part -How one token is chosen. Temperature, top-k and top-p only change the orange steps.
Greedy decoding
The simplest strategy, greedy decoding, always picks the most likely token. It gives the same answer for the same prompt, which is useful for extraction, classification and tests. For longer text it has a known weakness: always taking the single most likely word makes text bland and prone to repeating itself, because the most likely continuation of a repeated phrase is often the phrase again. Setting the temperature to 0 means greedy decoding in most APIs. Even then, hosted models are not always perfectly deterministic, because the order of floating-point calculations on shared hardware can vary slightly between requests.Temperature
Sampling picks the next token at random, in proportion to its probability: a token with probability 0.4 is chosen about 40% of the time. The temperature T reshapes the probabilities first, by dividing every logit by T before the softmax -pi = exp(zi / T) / Σj exp(zj / T)The same seven logits at three temperatures, as computed by the program below.
Top-k and top-p
Pure sampling has a problem: the vocabulary has tens of thousands of tokens, and together the many unlikely ones hold a noticeable share of the probability. Sooner or later one of them is picked, and the text goes off course. Two filters remove the tail before sampling -- Top-k keeps only the k most likely tokens. It is simple, but k is fixed: when the model is sure, k = 50 still lets 49 poor candidates in; when many words fit, a small k cuts off good ones.
- Top-p, or nucleus sampling (Holtzman et al., 2019), keeps the most likely tokens until their probabilities add up to p, such as 0.9. The number of candidates adapts: few when the model is confident, many when it is not.
Top-k and top-p applied to the temperature-1.0 probabilities. Grey tokens can no longer be chosen; the kept ones are rescaled to add up to 1.
Temperature, top-k and top-p in Java
The program below implements the whole pipeline for one step: softmax with temperature, top-k and top-p filtering, and sampling. It prints the probabilities, then draws ten next tokens for each setting with the same random seed, so that the settings can be compared.Softmax with temperature, top-k, top-p and sampling
//NextTokenSampling.java
import java.util.*;
public class NextTokenSampling {
// The model's raw scores (logits) for the next token after "The coffee was" - made up, but shaped like real ones
static final String[] TOKENS = {"hot", "strong", "delicious", "cold", "bitter", "perfect", "purple"};
static final double[] LOGITS = {4.0, 3.5, 3.0, 2.0, 1.8, 1.5, -1.0};
// Softmax with temperature: divide every logit by T, then turn the scores into probabilities that add up to 1
static double[] softmax(double[] logits, double temperature) {
double max = Arrays.stream(logits).max().getAsDouble();
double[] p = new double[logits.length];
double sum = 0;
for (int i = 0; i < p.length; i++) {
p[i] = Math.exp((logits[i] - max) / temperature); // subtracting max keeps exp() from overflowing
sum += p[i];
}
for (int i = 0; i < p.length; i++) p[i] /= sum;
return p;
}
// Top-k: keep the k most likely tokens; top-p: keep the most likely tokens until their probabilities add up to p.
// Everything else gets probability 0, and the rest is renormalized.
static double[] filter(double[] probs, int topK, double topP) {
Integer[] order = new Integer[probs.length];
for (int i = 0; i < order.length; i++) order[i] = i;
Arrays.sort(order, (a, b) -> Double.compare(probs[b], probs[a]));
double[] kept = new double[probs.length];
double cumulative = 0, sum = 0;
for (int rank = 0; rank < order.length; rank++) {
int i = order[rank];
if (rank >= topK || cumulative >= topP) break;
kept[i] = probs[i];
cumulative += probs[i];
sum += probs[i];
}
for (int i = 0; i < kept.length; i++) kept[i] /= sum;
return kept;
}
static int sample(double[] probs, Random random) {
double r = random.nextDouble(), cumulative = 0;
for (int i = 0; i < probs.length; i++) {
cumulative += probs[i];
if (r < cumulative) return i;
}
return probs.length - 1;
}
static void show(String label, double[] probs) {
StringBuilder sb = new StringBuilder(String.format("%-24s", label));
for (int i = 0; i < TOKENS.length; i++) sb.append(String.format(" %s=%.2f", TOKENS[i], probs[i]));
System.out.println(sb);
}
public static void main(String[] args) {
System.out.println("Probabilities for the next token after \"The coffee was\":");
for (double t : new double[] {0.5, 1.0, 1.5}) show("temperature " + t, softmax(LOGITS, t));
double[] base = softmax(LOGITS, 1.0);
show("top-k 3", filter(base, 3, 1.0));
show("top-p 0.9", filter(base, TOKENS.length, 0.9));
System.out.println();
System.out.println("Ten sampled continuations for each setting (fixed seed):");
Object[][] settings = {{"greedy (temperature 0)", 0.0, 7, 1.0}, {"temperature 0.5", 0.5, 7, 1.0},
{"temperature 1.0", 1.0, 7, 1.0}, {"temperature 1.5", 1.5, 7, 1.0}, {"temperature 1.5, top-p 0.9", 1.5, 7, 0.9}};
for (Object[] s : settings) {
double t = (double) s[1];
Random random = new Random(7);
StringBuilder sb = new StringBuilder(String.format("%-28s", s[0]));
Set<String> distinct = new TreeSet<>();
for (int n = 0; n < 10; n++) {
String token;
if (t == 0) { // greedy: always the most likely token
int best = 0;
for (int i = 1; i < LOGITS.length; i++) if (LOGITS[i] > LOGITS[best]) best = i;
token = TOKENS[best];
} else {
token = TOKENS[sample(filter(softmax(LOGITS, t), (int) s[2], (double) s[3]), random)];
}
sb.append(token).append(' ');
distinct.add(token);
}
System.out.println(sb.toString().trim() + " (" + distinct.size() + " different)");
}
}
}
Probabilities for the next token after "The coffee was": temperature 0.5 hot=0.65 strong=0.24 delicious=0.09 cold=0.01 bitter=0.01 perfect=0.00 purple=0.00 temperature 1.0 hot=0.43 strong=0.26 delicious=0.16 cold=0.06 bitter=0.05 perfect=0.04 purple=0.00 temperature 1.5 hot=0.34 strong=0.24 delicious=0.17 cold=0.09 bitter=0.08 perfect=0.06 purple=0.01 top-k 3 hot=0.51 strong=0.31 delicious=0.19 cold=0.00 bitter=0.00 perfect=0.00 purple=0.00 top-p 0.9 hot=0.47 strong=0.29 delicious=0.17 cold=0.06 bitter=0.00 perfect=0.00 purple=0.00 Ten sampled continuations for each setting (fixed seed): greedy (temperature 0) hot hot hot hot hot hot hot hot hot hot (1 different) temperature 0.5 strong strong hot delicious strong hot hot strong hot delicious (3 different) temperature 1.0 delicious delicious hot cold delicious hot hot delicious hot bitter (4 different) temperature 1.5 delicious delicious strong bitter delicious strong hot bitter hot perfect (5 different) temperature 1.5, top-p 0.9 delicious delicious hot cold delicious hot hot cold hot bitter (4 different)
Explanation of the code
softmax()divides each logit by the temperature before the exponent. Subtracting the largest logit first does not change the result, but keepsMath.exp()from overflowing - real implementations do the same.- After top-k or top-p, the kept probabilities are divided by their sum, so they add up to 1 again: with top-k 3, "hot" rose from 0.43 to 0.51.
- Greedy decoding chose "hot" ten times out of ten. Temperature 0.5 produced 3 different tokens, 1.0 produced 4 and 1.5 produced 5, including "perfect", which had only a 6% chance.
- Temperature 1.5 with top-p 0.9 gave 4 different tokens and never "perfect" or "purple": top-p removed the tail which the high temperature had raised. This combination - some randomness, but no unlikely tokens - is what top-p is for.
- All runs used the same seed, so they are repeatable. APIs which accept a
seedparameter use it the same way, to make sampled answers more reproducible.
Choosing the settings
| Task | Typical setting | Why |
|---|---|---|
| Extraction, classification, structured outputClick to read tutorial: Prompt Engineering Techniques Explained with Examples, code | Temperature 0 to 0.3 | There is one right answer; variation only adds errors |
| Chat assistants, explanations, summaries | Temperature around 0.7, or the provider's default | Natural-sounding text with few odd word choices |
| Brainstorming, stories, marketing copy | Temperature around 1, top-p 0.9 to 0.95 | Variety is the goal; top-p keeps it sensible |
| Several samples for voting, as in self-consistencyClick to read tutorial: Chain-of-Thought Prompting Paper Explained | Temperature above 0, e.g. 0.7 | Different reasoning paths are needed to vote on |
- Ranges differ between providers: some accept temperatures from 0 to 2, others from 0 to 1, and the defaults differ too. The same number does not mean exactly the same thing across providers or models.
- Reasoning models, such as DeepSeek-R1Click to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL, often fix their sampling settings themselves, and some APIs do not accept a temperature for them at all.
- Low temperature is not accuracy: it makes answers repeatable, not correct. A model at temperature 0 can repeat the same hallucinationClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations every time; grounding with RAGClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval and validation are what reduce wrong answers.
Summary
In this post we went through how a large language model chooses each token. We saw that the model produces a logit for every token, softmax turns the logits into probabilities, and a decoding strategy picks one token. Greedy decoding always takes the most likely token; sampling picks at random by probability, with the temperature making the distribution sharper below 1 and flatter above 1. Top-k keeps a fixed number of candidates and top-p keeps just enough to reach a probability mass, which adapts to how sure the model is. In Java, we computed all of this for one step and compared ten sampled tokens per setting, and lastly we went through which settings suit which tasks. How to write the prompt that the model continues is explained in prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples.Continue Reading: AI and LLM Papers Explained
Tokenization Explained: Byte Pair Encoding in JavaClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java Prompt Engineering Techniques Explained with ExamplesClick to read tutorial: Prompt Engineering Techniques Explained with Examples Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained
Tokenization Explained: Byte Pair Encoding in JavaClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java Prompt Engineering Techniques Explained with ExamplesClick to read tutorial: Prompt Engineering Techniques Explained with Examples Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained