GPT-3 Paper Explained: Language Models Are Few-Shot Learners
Introduction
Until 2020, using a language model for a new task meant fine-tuning it on thousands of labelled examples of that task, as with BERTClick to read tutorial: BERT Paper Explained: Bidirectional Transformers. The GPT-3 paper, Language Models are Few-Shot Learners, showed something different: a big enough language model can perform a new task just from a description and a few examples written into its prompt - with no training at all. This ability, called in-context learning, is how we use language models today. This post explains the GPT-3 paper with diagrams. We will first go through the model - its decoder-only architecture, its size, which we will reproduce in Java, and its training data. We will then see zero-shot, one-shot and few-shot learning, the paper's results, and lastly its limitations and impact.The paper: Language Models are Few-Shot Learners
Authors: Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal and 25 others (OpenAI)
Published: May 2020 - NeurIPS 2020
Authors: Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal and 25 others (OpenAI)
Published: May 2020 - NeurIPS 2020
The model: a scaled-up decoder
GPT-3 is a decoder-only Transformer - the decoder half of the original TransformerClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained, without the encoder and without encoder-decoder attention. It is trained on one task only: given the tokens so far, predict the next token. To generate text, it predicts a token, appends it to its input, and runs again - an autoregressive loop (how each token is picked is explained in temperature, top-k and top-p samplingClick to read tutorial: LLM Temperature, Top-k and Top-p Sampling Explained) -How a decoder-only model generates text, one token at a time.
- 96 layers, each with 96 attention heads of size 128,
- a model width (
dmodel) of 12,288, - a context windowClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations of 2,048 tokens, and
- 175 billion parameters - more than 10 times any previous non-sparse language model.
The weight matrices of one GPT-3 layer. Biases and layer-normalization weights are too small to matter.
d × d matrices (queries, keys, values and output). The feed-forward block widens each token's vector to 4d numbers and shrinks it back, with two matrices of d × 4d. That is 4d² + 8d² = 12d² parameters per layer - two thirds of them in the feed-forward block, which is why later models made that block sparse with mixture of expertsClick to read tutorial: Mixture of Experts (MoE) Explained: Routers and Experts. Adding the token and position embeddings gives the total -
Reproducing GPT-3's parameter counts
//Gpt3Params.java
public class Gpt3Params {
static final int VOCAB = 50_257, CONTEXT = 2_048;
static double params(int layers, int dModel) {
double perLayer = 12.0 * dModel * dModel; // attention 4d^2 + feed-forward 8d^2
double embeddings = (double) (VOCAB + CONTEXT) * dModel; // token + position embeddings
return layers * perLayer + embeddings;
}
public static void main(String[] args) {
String[] names = {"GPT-3 Small", "GPT-3 XL", "GPT-3 13B", "GPT-3 175B"};
int[] layers = {12, 24, 40, 96};
int[] widths = {768, 2048, 5140, 12288};
for (int i = 0; i < names.length; i++) {
System.out.printf("%-12s layers=%-3d d_model=%-6d -> %,.0f parameters%n",
names[i], layers[i], widths[i], params(layers[i], widths[i]));
}
}
}
GPT-3 Small layers=12 d_model=768 -> 125,104,896 parameters GPT-3 XL layers=24 d_model=2048 -> 1,315,080,192 parameters GPT-3 13B layers=40 d_model=5140 -> 12,950,255,700 parameters GPT-3 175B layers=96 d_model=12288 -> 174,588,899,328 parameters
GPT-3 was trained on 300 billion tokens - pieces from its byte-level BPE vocabulary, see tokenization explainedClick to read tutorial: Tokenization Explained: Byte Pair Encoding in Java - sampled from a mixture of a filtered version of Common Crawl (60% of the mix), an expanded WebText dataset (22%), two book corpora (16%) and English Wikipedia (3%). Higher-quality sources were sampled more often than their size alone would suggest. Training took about 3.14 × 1023 FLOPs on a cluster of NVIDIA V100 GPUs.
In-context learning: zero-shot, one-shot and few-shot
The paper's central question was how well the model can learn a task from its context, without any change to its weights. It tested three settings, which differ only in how many solved examples are written into the prompt before the actual question -The three in-context learning settings, using the translation example from the paper.
- Zero-shot: only a natural-language description of the task. This is the hardest setting, but the most convenient for users.
- One-shot: the description and one example, as you might show a person once what you want.
- Few-shot: the description and as many examples as fit in the context window - typically 10 to 100.
Results
- Question answering: on TriviaQA, answering from memory alone with no documents, few-shot GPT-3 scored 71.2%, beating fine-tuned models which were allowed to search for documents.
- Language modelling: on LAMBADA, which asks for the last word of a paragraph, few-shot accuracy reached 86.4%, an 18-point jump over the previous state of the art.
- Arithmetic: the 175B model added and subtracted 2-digit numbers almost perfectly, and 3-digit numbers about 80-94% of the time, while the smaller models largely failed - an ability which appeared with scale.
- Novel tasks: the model could unscramble words, use a made-up word in a sentence after seeing its definition once, and correct grammar.
- News generation: people asked to tell GPT-3's short news articles from real ones were right only about 52% of the time - barely better than guessing.
Limitations noted in the paper
The paper was unusually frank about what GPT-3 could not do -- It was still weak on some tasks, such as natural language inference (deciding whether one sentence implies another) and some reading comprehension benchmarks, where fine-tuned models remained far better.
- Long generations could repeat themselves, lose coherence, or contradict themselves, and answers could state made-up facts with confidence - hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations.
- Some benchmark questions had leaked into the web-scale training data; the authors built tools to measure this data contamination and flagged affected results.
- The model reflected social biases present in its training data, which the paper measured for gender, race and religion.
- The model was expensive to train and to run, and it is not clear whether it learns new tasks "from scratch" in context, or recognizes tasks it saw during training.
Impact
GPT-3 changed how people use language models. Instead of training a model for each task, developers write prompts, and "prompt engineering" became a skill of its own - its main techniques are explained in prompt engineering techniques explainedClick to read tutorial: Prompt Engineering Techniques Explained with Examples. OpenAI made GPT-3 available through an API in 2020, the first time many developers could build on a large language model. Its weaknesses also set the agenda for the papers that followed: few-shot examples which show reasoning steps gave chain-of-thought promptingClick to read tutorial: Chain-of-Thought Prompting Paper Explained; the gap between predicting internet text and following a user's instructions led to InstructGPTClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF and then ChatGPT; and the cost of adapting such a large model led to methods like LoRAClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs.Summary
In this post we went through the GPT-3 paper. We saw that GPT-3 is a decoder-only Transformer which generates text one token at a time, and reproduced its 175 billion parameters in Java from its 96 layers and width of 12,288. We looked at its 300-billion-token training mix, and at the paper's main idea, in-context learning, with zero-shot, one-shot and few-shot prompts and no weight updates. We then went through the results, which improved sharply with model size, the limitations the paper reported, and how GPT-3 made prompting the main way to use language models. In the next post, we will see how a language model can answer from documents, in the Retrieval-Augmented Generation paperClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation.Continue Reading: AI and LLM Papers Explained
RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL
RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL