LLM Tokens, Context Windows, Embeddings and Hallucinations
Introduction
Calling a large language model from Java takes a few lines of code: send some text to an API, get some text back. Using one well takes four ideas which the APIs, their prices and their failures are all built around - tokens, the context window, embeddings and hallucinations. This post explains these four concepts for Java developers, with diagrams and small runnable Java programs. We will first see what a generative model actually does, then how text is cut into tokens and why everything is counted in them, how the context window limits what a model can see in one call, how embeddings turn meaning into numbers, and lastly why models hallucinate and what can be done about it.(Note - This post explains concepts used throughout the Large Language Models category. The paper posts, starting with the Transformer paperClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained, show where each idea came from.)
What a generative model does
A large language model (LLM) such as GPT, Claude, Gemini or Llama is a neural network trained on a huge amount of text to do one thing: given the text so far, predict what comes next. To write an answer, it predicts one piece of text, appends it to its input, and predicts again - the autoregressive loop shown in the GPT-3 postClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners. Everything a chat assistant does - answering, summarizing, writing code - is this loop, run on a prompt that makes the right continuation likely. (Writing such prompts is covered in prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples.) Three consequences matter for a developer -- The model reads and writes tokens, not characters or words.
- Each call sees only what fits in its context window; it has no other memory of the conversation.
- It produces likely text, not checked text - which is where hallucinations come from.
Tokens
A token is the unit of text a model works with: a whole common word, a piece of a longer word, a punctuation mark, or a space folded into the next word. A tokenizer splits the text into tokens from a fixed vocabulary of pieces, learned from data before the model is trained (how, is shown in tokenization: byte pair encoding in JavaClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java) - about 50,000 pieces for GPT-3, and up to a few hundred thousand in newer models. Common words become one token; rare and long words are split into several -The tokens produced by the small tokenizer in the program below. A real tokenizer's pieces differ, but the pattern is the same.
- Cost: API providers charge per token, with separate prices for the tokens you send (input) and the tokens the model writes (output).
- Limits: the context window and the maximum length of an answer are measured in tokens.
- Speed: the model writes one token per step, so a long answer takes proportionally longer.
Splitting text into tokens and fitting a chat into a token budget
//TokensAndContext.java
import java.util.*;
public class TokensAndContext {
// A tiny sub-word vocabulary. Real tokenizers learn about 50,000 to 200,000 pieces from data.
static final Set<String> VOCAB = Set.of(
"the", "jvm", "run", "s", "java", "byte", "code", "token", "iz", "ation", "un", "believ", "ab", "ly",
"fast", "is", "?", ".", ",", " ");
// Greedy longest match, as WordPiece does: take the longest vocabulary piece at each position.
static List<String> tokenize(String text) {
List<String> tokens = new ArrayList<>();
String s = text.toLowerCase();
int i = 0;
while (i < s.length()) {
int end = s.length();
while (end > i && !VOCAB.contains(s.substring(i, end))) end--;
if (end == i) end = i + 1; // unknown character: one token on its own
tokens.add(s.substring(i, end));
i = end;
}
tokens.removeIf(t -> t.equals(" ")); // spaces are folded into the next token by real tokenizers
return tokens;
}
record Message(String role, String text) {}
// Rough estimate used when no tokenizer is at hand: about 4 characters per token for English text.
static int estimateTokens(String text) {
return Math.max(1, (int) Math.ceil(text.length() / 4.0));
}
// Keeps the system prompt and as many of the newest messages as fit in the token budget.
static List<Message> fitToWindow(List<Message> history, int budget) {
Message system = history.get(0);
int used = estimateTokens(system.text());
Deque<Message> kept = new ArrayDeque<>();
for (int i = history.size() - 1; i > 0; i--) {
int cost = estimateTokens(history.get(i).text());
if (used + cost > budget) break;
used += cost;
kept.addFirst(history.get(i));
}
List<Message> result = new ArrayList<>(List.of(system));
result.addAll(kept);
System.out.printf("kept %d of %d messages, about %d tokens of a %d-token budget%n",
result.size(), history.size(), used, budget);
return result;
}
public static void main(String[] args) {
for (String text : List.of("The JVM runs Java bytecode.", "Tokenization is unbelievably fast.")) {
List<String> tokens = tokenize(text);
System.out.printf("%-36s chars=%d words=%d tokens=%d %s%n", "\"" + text + "\"",
text.length(), text.split(" ").length, tokens.size(), tokens);
}
List<Message> history = List.of(
new Message("system", "You are a helpful assistant for a Java team. Answer briefly."),
new Message("user", "What does the JIT compiler do?"),
new Message("assistant", "It compiles frequently used bytecode to native machine code while the program runs."),
new Message("user", "And what is the difference between the heap and the stack?"),
new Message("assistant", "Objects live on the heap, shared by all threads; each thread has its own stack for method calls."),
new Message("user", "Which garbage collector is the default in Java 21?"));
System.out.println();
for (Message m : fitToWindow(history, 60)) System.out.println(" " + m.role() + ": " + m.text());
}
}
"The JVM runs Java bytecode." chars=27 words=5 tokens=8 [the, jvm, run, s, java, byte, code, .] "Tokenization is unbelievably fast." chars=34 words=4 tokens=10 [token, iz, ation, is, un, believ, ab, ly, fast, .] kept 3 of 6 messages, about 52 tokens of a 60-token budget system: You are a helpful assistant for a Java team. Answer briefly. assistant: Objects live on the heap, shared by all threads; each thread has its own stack for method calls. user: Which garbage collector is the default in Java 21?
Explanation of the code
tokenize()takes the longest piece of the vocabulary that matches at each position. "runs" becomes "run" + "s", and "bytecode" becomes "byte" + "code" - sub-word pieces let a fixed vocabulary spell any word, including words it has never seen.- The first sentence has 5 words but 8 tokens; the second has 4 words but 10 tokens. Counting words or characters is only an estimate of what a model will be charged for.
estimateTokens()uses the 4-characters rule. When the exact count matters, use the tokenizer of the model you call - providers publish them, and their APIs report the exact token counts of each request.fitToWindow()is explained in the next section.
The context window
The context window is the maximum number of tokens a model can take into account in one call - its input and its answer together. It has grown from 2,048 tokens for GPT-3 in 2020 to hundreds of thousands of tokens and more in current models. Everything the model is to use must fit in it -What a chat application puts into the context window on every call.
- The model has no memory between calls. A chat API does not remember your earlier messages; a chat application sends the whole conversation, or the part of it that fits, with every new message. That is also why each turn of a long conversation costs more input tokens than the one before.
- A full window is not free. More tokens cost more and take longer to process, and models tend to use information in the middle of a very long context less reliably than information near its start or end. A smaller, relevant context often gives better answers than a full one.
fitToWindow() above, keeps the system prompt and the newest messages and drops the oldest ones. With a budget of 60 tokens, the output shows its flaw: it kept the assistant's answer about the heap and the stack, but dropped the user's question which that answer belonged to. Better strategies drop whole question-and-answer pairs, or replace old messages with a short summary written by the model itself.
Embeddings
An embedding is a list of numbers - a vector - which represents the meaning of a piece of text. An embedding model, a neural network trained for this purpose, turns any text into a vector of the same length, typically 384 to 3,072 numbers. It is trained so that texts with similar meanings get vectors that point in similar directions, even when they share no words -An embedding space: the two sentences about dogs are neighbours although they share no words.
- semantic search - find documents by meaning rather than by matching words,
- retrieval for RAG - find the passages an LLM should read before answering, as in the RAG paperClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation, and
- clustering, recommendations and duplicate detection - anything that needs "how similar are these two texts?".
Comparing texts by the cosine similarity of their embeddings
//EmbeddingSimilarity.java
import java.util.*;
public class EmbeddingSimilarity {
// Hand-made 4-number "embeddings". Each number stands for a direction of meaning:
// [programming, animals, food, transport]. Real embedding models produce 384 to 3,072 numbers,
// learned from data, and no single number has a meaning a person could name.
static final Map<String, double[]> EMBEDDINGS = new LinkedHashMap<>();
static {
EMBEDDINGS.put("How do I sort a list in Java?", new double[] {0.90, 0.02, 0.05, 0.03});
EMBEDDINGS.put("Ordering an ArrayList with Comparator", new double[] {0.85, 0.01, 0.03, 0.05});
EMBEDDINGS.put("My puppy will not stop barking", new double[] {0.03, 0.92, 0.10, 0.02});
EMBEDDINGS.put("Training a young dog to be quiet", new double[] {0.05, 0.88, 0.06, 0.04});
EMBEDDINGS.put("Best pasta recipes for dinner", new double[] {0.02, 0.04, 0.95, 0.03});
}
static double cosine(double[] a, double[] b) {
double dot = 0, na = 0, nb = 0;
for (int i = 0; i < a.length; i++) {
dot += a[i] * b[i];
na += a[i] * a[i];
nb += b[i] * b[i];
}
return dot / (Math.sqrt(na) * Math.sqrt(nb));
}
public static void main(String[] args) {
String query = "How do I sort a list in Java?";
double[] q = EMBEDDINGS.get(query);
System.out.println("Similarity to: \"" + query + "\"");
EMBEDDINGS.entrySet().stream()
.filter(e -> !e.getKey().equals(query))
.sorted((a, b) -> Double.compare(cosine(q, b.getValue()), cosine(q, a.getValue())))
.forEach(e -> System.out.printf(" %.3f %s%n", cosine(q, e.getValue()), e.getKey()));
String a = "My puppy will not stop barking", b = "Training a young dog to be quiet";
System.out.printf("%nNo words in common, similar meaning:%n %.3f \"%s\" vs \"%s\"%n",
cosine(EMBEDDINGS.get(a), EMBEDDINGS.get(b)), a, b);
}
}
Similarity to: "How do I sort a list in Java?" 0.999 Ordering an ArrayList with Comparator 0.084 Training a young dog to be quiet 0.078 Best pasta recipes for dinner 0.061 My puppy will not stop barking No words in common, similar meaning: 0.999 "My puppy will not stop barking" vs "Training a young dog to be quiet"
Explanation of the code
- The question about sorting is closest to the sentence about ordering an
ArrayListwith aComparator, which shares only "list" with it as a word - and even that only inside "ArrayList". - The two sentences about dogs share no words at all, yet their similarity is 0.999. Word matching would never find this; embeddings find it because both point in the "animals" direction.
- Cosine similarity ignores the length of the vectors and compares only their direction, so a long document and a short question about the same topic can still match closely.
Hallucinations
A hallucination is an answer which is fluent and confident but wrong: an invented fact, a citation to a paper that does not exist, a Java method that is not in the API. It follows directly from what the model is: it generates the most likely continuation of its context, and nothing in that process looks a fact up or checks it -The same question answered from the model's training alone, and from documents placed in its context.
- Ground the answer: put the relevant documents into the context and tell the model to answer only from them - retrieval-augmented generation, explained step by step in RAG architecture explainedClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval.
- Ask for sources: answers which must quote or cite the provided text are easier to check, by a person or by code.
- Allow "I don't know": say in the prompt that the model should say so when the context does not contain the answer.
- Check what can be checked: compile generated code, run its tests, validate generated JSON against a schema, and look up facts that drive decisions.
- Use better-trained models: training with human feedback reduced made-up information considerably in the InstructGPT paperClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF, and newer models hallucinate less than older ones - but no model is free of hallucinations.
Summary
In this post we went through the four ideas behind working with large language models from Java. We saw that a generative model predicts text one token at a time, and that tokens - sub-word pieces from a fixed vocabulary - are the unit of cost, limits and speed, which we showed with a small tokenizer in Java. We saw that the context window is all a model can see in one call, that the model keeps nothing between calls, and how a conversation is trimmed to a token budget. We then saw how embeddings turn meaning into vectors that can be compared with cosine similarity, and lastly why models hallucinate and how grounding, sources and checks reduce it. In the next post, we will put tokens, embeddings and the context window together in the architecture of a RAG systemClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval.Continue Reading: AI and LLM Papers Explained
RAG Architecture Explained: Chunking, Embeddings, RetrievalClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained
RAG Architecture Explained: Chunking, Embeddings, RetrievalClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained