InstructGPT Paper Explained: Training LLMs with RLHF
Introduction
GPT-3Click to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners was trained to predict the next token of internet text - not to do what its user asks. Ask it to "explain photosynthesis to a child" and it might continue with a list of other homework questions, because that is what such text often looks like on the web. It could also make up facts - hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations - or produce toxic text. The 2022 OpenAI paper Training language models to follow instructions with human feedback fixed this with reinforcement learning from human feedback (RLHF), producing the InstructGPT models. The same method was used, months later, to create ChatGPT. This post explains the InstructGPT paper with diagrams. We will see the alignment problem it addresses, go through the three steps of RLHF - supervised fine-tuning, training a reward model (whose loss we will compute in Java), and optimizing the model with PPO - and then look at the results and limitations.The paper: Training language models to follow instructions with human feedback
Authors: Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin and 14 others (OpenAI)
Published: March 2022 - NeurIPS 2022
Authors: Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin and 14 others (OpenAI)
Published: March 2022 - NeurIPS 2022
The alignment problem
The paper calls the gap between what a model is trained to do and what its users want a problem of alignment. Next-token prediction on web text is a misaligned objective for an assistant. The authors wanted models which are -- helpful - they do the task the user intends, including when it is only implied,
- honest - they do not make up information or mislead, and
- harmless - they do not cause physical, psychological or social harm.
RLHF in three steps
The training data came from prompts which users had submitted to the OpenAI API, plus prompts written by a team of about 40 contractors, the labelers, who were selected and trained for the job. Starting from a pre-trained GPT-3 model, training has three steps -The three steps of RLHF as used for InstructGPT.
- Supervised fine-tuning (SFT): labelers wrote high-quality answers for about 13,000 prompts, and GPT-3 was fine-tuned on these demonstrations. This teaches the model the basic format of following an instruction. (For open models, this step is often done with LoRAClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs adapters instead of updating every weight.)
- Reward model (RM): for about 33,000 prompts, the model generated several answers, and labelers ranked them from best to worst. A separate model - a 6-billion-parameter GPT-3 with its output layer replaced by one which outputs a single number - was trained to give higher scores to the answers the labelers preferred. Ranking is much faster and more consistent for people than writing perfect answers, so this collects far more signal per hour of human work.
- Reinforcement learning with PPO: the SFT model, now called the policy, writes answers to new prompts, the reward model scores them, and the policy's weights are updated with Proximal Policy Optimization (PPO) to make high-scoring answers more likely.
Training the reward model
The reward modelr(x, y) gives a score to an answer y for a prompt x. The scores themselves have no meaning; only the differences matter. For each pair of answers where the labeler preferred yw (the "winner") over yl (the "loser"), the loss is -
loss = -log σ( r(x, yw) - r(x, yl) )How one labeler ranking becomes six training examples for the reward model.
The pairwise ranking loss of the reward model
//RewardModelLoss.java
import java.util.List;
public class RewardModelLoss {
static double sigmoid(double z) { return 1 / (1 + Math.exp(-z)); }
// ranking: answers from best to worst; scores: the reward model's score for answers A, B, C, D
static double rankingLoss(List<Character> ranking, double[] scores) {
double total = 0;
int pairs = 0;
for (int i = 0; i < ranking.size(); i++) {
for (int j = i + 1; j < ranking.size(); j++) {
double winner = scores[ranking.get(i) - 'A'], loser = scores[ranking.get(j) - 'A'];
total += -Math.log(sigmoid(winner - loser));
pairs++;
}
}
return total / pairs;
}
public static void main(String[] args) {
List<Character> labelerRanking = List.of('D', 'C', 'A', 'B'); // D > C > A > B
double[] agrees = {0.5, -1.0, 1.5, 2.5}; // scores for A, B, C, D: same order as the labeler
double[] disagrees = {2.0, 1.0, -0.5, -1.0}; // roughly the reverse order
double[] untrained = {0.0, 0.0, 0.0, 0.0}; // all answers scored equally
System.out.printf("Pairs compared per ranking : %d%n", 4 * 3 / 2);
System.out.printf("Loss, RM agrees with labeler : %.3f%n", rankingLoss(labelerRanking, agrees));
System.out.printf("Loss, untrained RM (all equal) : %.3f%n", rankingLoss(labelerRanking, untrained));
System.out.printf("Loss, RM disagrees with labeler: %.3f%n", rankingLoss(labelerRanking, disagrees));
System.out.printf("P(D preferred over B) under the agreeing RM: %.2f%n", sigmoid(agrees[3] - agrees[1]));
}
}
Pairs compared per ranking : 6 Loss, RM agrees with labeler : 0.177 Loss, untrained RM (all equal) : 0.693 Loss, RM disagrees with labeler: 1.791 P(D preferred over B) under the agreeing RM: 0.97
Explanation of the code
- An untrained reward model which scores everything equally has a loss of log 2 ≈ 0.693 per pair - it is guessing 50/50.
- A reward model whose scores follow the labeler's order has a much lower loss, and one which reverses the order has a much higher loss. Training moves the scores to reduce the loss, i.e. towards the labelers' preferences.
- The last line shows how a score difference reads as a probability: with a difference of 3.5, the model is 97% confident that a labeler would prefer D over B.
Optimizing the policy with PPO
In step 3, generating an answer is treated as a reinforcement learning problem: the policy takes "actions" (choosing tokens), and receives a reward from the reward model when the answer is complete. There is a catch: the reward model is only an approximation of human judgement, and a policy optimized hard against it will find answers which score highly but are actually bad - known as reward hacking. To prevent this, the reward is reduced by a penalty for drifting too far from the SFT model, measured with the KL divergenceClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained between the two models' token probabilities -The RL step: maximize the reward model's score while staying close to the SFT model.
Results
- Preferred by people: labelers preferred the answers of the 1.3-billion-parameter InstructGPT over those of the 175-billion-parameter GPT-3, despite it having over 100 times fewer parameters. The 175B InstructGPT's answers were preferred to GPT-3's 85% of the time, and to GPT-3 with a carefully written few-shot prompt (see prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples) 71% of the time.
- More truthful: on the TruthfulQA benchmark, InstructGPT gave truthful and informative answers about twice as often as GPT-3, and on tasks where it should answer only from given text, it made up information about half as often (21% vs 41% of the time).
- Less toxic: when asked to be respectful, it produced about 25% fewer toxic outputs than GPT-3. It did not, however, show a significant improvement on measures of bias.
- Generalization: it followed instructions in settings which were rare in its fine-tuning data, such as code questions and instructions in other languages.
- Cheap: the whole RLHF process used a small fraction of the compute of GPT-3's pre-training.
Limitations
- The models are aligned to the preferences of a small group of labelers and researchers, mostly English-speaking, not to "human values" in general.
- InstructGPT still made simple mistakes, could invent facts, and would follow instructions to produce harmful content.
- RLHF can encourage answers which sound good to a rater over answers which are correct, and can make models overly verbose or hedging.
Impact
In November 2022, OpenAI released ChatGPT, trained with the same method on conversations. RLHF became the standard final stage of training for chat assistants. The pipeline has since been simplified and extended: Direct Preference Optimization (DPO), from 2023, learns from the same preference pairs without a separate reward model or PPO, and methods such as Constitutional AI use feedback generated by an AI model, guided by written principles, in place of some human labels (RLAIF). Reinforcement learning itself became central again for reasoning, with rewards from checking answers automatically, as in DeepSeek-R1Click to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL.Summary
In this post we went through the InstructGPT paper. We saw why next-token prediction makes a model misaligned with what its users want, and how RLHF uses human judgements to fix this in three steps: supervised fine-tuning on labeler demonstrations, training a reward model on labeler rankings, and optimizing the model against the reward model with PPO, with a KL penalty to keep it close to the SFT model. We computed the reward model's pairwise ranking loss in Java. Lastly, we went through the results - a 1.3B InstructGPT preferred over the 175B GPT-3, more truthful and less toxic answers - the limitations, and how the method led to ChatGPT. In the next post, we will see how language models started to take actions, in the ReAct paperClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs.Continue Reading: AI and LLM Papers Explained
ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL
ReAct Paper Explained: Reasoning and Acting in LLMsClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL