InstructGPT Paper Explained: Training LLMs with RLHF

Introduction

GPT-3Click to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners was trained to predict the next token of internet text - not to do what its user asks. Ask it to "explain photosynthesis to a child" and it might continue with a list of other homework questions, because that is what such text often looks like on the web. It could also make up facts - hallucinationsClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations - or produce toxic text. The 2022 OpenAI paper Training language models to follow instructions with human feedback fixed this with reinforcement learning from human feedback (RLHF), producing the InstructGPT models. The same method was used, months later, to create ChatGPT. This post explains the InstructGPT paper with diagrams. We will see the alignment problem it addresses, go through the three steps of RLHF - supervised fine-tuning, training a reward model (whose loss we will compute in Java), and optimizing the model with PPO - and then look at the results and limitations.
The paper: Training language models to follow instructions with human feedback
Authors: Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin and 14 others (OpenAI)
Published: March 2022 - NeurIPS 2022

The alignment problem

The paper calls the gap between what a model is trained to do and what its users want a problem of alignment. Next-token prediction on web text is a misaligned objective for an assistant. The authors wanted models which are -
  • helpful - they do the task the user intends, including when it is only implied,
  • honest - they do not make up information or mislead, and
  • harmless - they do not cause physical, psychological or social harm.
These qualities are hard to write down as a loss function, but people can easily recognize them when they compare two answers. RLHF turns those human judgements into a training signal.

RLHF in three steps

The training data came from prompts which users had submitted to the OpenAI API, plus prompts written by a team of about 40 contractors, the labelers, who were selected and trained for the job. Starting from a pre-trained GPT-3 model, training has three steps -
Step 1: labelers write demonstrations used for supervised fine-tuning. Step 2: labelers rank model outputs and a reward model is trained on the rankings. Step 3: the model is optimized against the reward model with PPO. Step 1: Supervised fine-tuning A prompt is sampled A labeler writes thedesired answer Fine-tune GPT-3 ~13k prompts→ SFT model Step 2: Reward model The model writes 4 to 9answers to a prompt A labeler ranks thembest to worst: D > C > A > B Train a model to scoreanswers like the labelers ~33k prompts→ reward model (RM) Step 3: Optimize with PPO A new prompt is sampled The policy (starting fromSFT) writes an answer RM scores it; PPOupdates the policy ~31k prompts→ InstructGPT
The three steps of RLHF as used for InstructGPT.
  1. Supervised fine-tuning (SFT): labelers wrote high-quality answers for about 13,000 prompts, and GPT-3 was fine-tuned on these demonstrations. This teaches the model the basic format of following an instruction. (For open models, this step is often done with LoRAClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs adapters instead of updating every weight.)
  2. Reward model (RM): for about 33,000 prompts, the model generated several answers, and labelers ranked them from best to worst. A separate model - a 6-billion-parameter GPT-3 with its output layer replaced by one which outputs a single number - was trained to give higher scores to the answers the labelers preferred. Ranking is much faster and more consistent for people than writing perfect answers, so this collects far more signal per hour of human work.
  3. Reinforcement learning with PPO: the SFT model, now called the policy, writes answers to new prompts, the reward model scores them, and the policy's weights are updated with Proximal Policy Optimization (PPO) to make high-scoring answers more likely.

Training the reward model

The reward model r(x, y) gives a score to an answer y for a prompt x. The scores themselves have no meaning; only the differences matter. For each pair of answers where the labeler preferred yw (the "winner") over yl (the "loser"), the loss is -
loss = -log σ( r(x, yw) - r(x, yl) )
where σ is the sigmoid function, which turns the score difference into a probability that the winner is preferred. The loss is small when the winner's score is well above the loser's, and large when the model gets the order wrong. A ranking of K answers gives K × (K - 1) / 2 pairs - 6 pairs for 4 answers -
The ranking D, C, A, B gives the pairs D over C, D over A, D over B, C over A, C over B and A over B; each pair adds one loss term A labeler's ranking D > C > A > B bestworst 6 training pairs (winner over loser) D over C D over A D over B C over A C over B A over B each pair adds -log σ( r(winner) - r(loser) ) the six terms are averaged into one loss for this prompt One ranking, made in one sitting, gives six comparisons for training.
How one labeler ranking becomes six training examples for the reward model.
The paper trains on all the pairs from one prompt together, which makes training faster and avoids overfitting. The program below computes this loss for a labeler's ranking of 4 answers, for a reward model which agrees with the labeler and for one which does not -
The pairwise ranking loss of the reward model
//RewardModelLoss.java
import java.util.List;

public class RewardModelLoss {
  static double sigmoid(double z) { return 1 / (1 + Math.exp(-z)); }

  // ranking: answers from best to worst; scores: the reward model's score for answers A, B, C, D
  static double rankingLoss(List<Character> ranking, double[] scores) {
    double total = 0;
    int pairs = 0;
    for (int i = 0; i < ranking.size(); i++) {
      for (int j = i + 1; j < ranking.size(); j++) {
        double winner = scores[ranking.get(i) - 'A'], loser = scores[ranking.get(j) - 'A'];
        total += -Math.log(sigmoid(winner - loser));
        pairs++;
      }
    }
    return total / pairs;
  }

  public static void main(String[] args) {
    List<Character> labelerRanking = List.of('D', 'C', 'A', 'B');   // D > C > A > B
    double[] agrees = {0.5, -1.0, 1.5, 2.5};       // scores for A, B, C, D: same order as the labeler
    double[] disagrees = {2.0, 1.0, -0.5, -1.0};   // roughly the reverse order
    double[] untrained = {0.0, 0.0, 0.0, 0.0};     // all answers scored equally

    System.out.printf("Pairs compared per ranking : %d%n", 4 * 3 / 2);
    System.out.printf("Loss, RM agrees with labeler   : %.3f%n", rankingLoss(labelerRanking, agrees));
    System.out.printf("Loss, untrained RM (all equal) : %.3f%n", rankingLoss(labelerRanking, untrained));
    System.out.printf("Loss, RM disagrees with labeler: %.3f%n", rankingLoss(labelerRanking, disagrees));
    System.out.printf("P(D preferred over B) under the agreeing RM: %.2f%n", sigmoid(agrees[3] - agrees[1]));
  }
}
 OUTPUT of the above code
Pairs compared per ranking : 6
Loss, RM agrees with labeler   : 0.177
Loss, untrained RM (all equal) : 0.693
Loss, RM disagrees with labeler: 1.791
P(D preferred over B) under the agreeing RM: 0.97

Explanation of the code

  • An untrained reward model which scores everything equally has a loss of log 2 ≈ 0.693 per pair - it is guessing 50/50.
  • A reward model whose scores follow the labeler's order has a much lower loss, and one which reverses the order has a much higher loss. Training moves the scores to reduce the loss, i.e. towards the labelers' preferences.
  • The last line shows how a score difference reads as a probability: with a difference of 3.5, the model is 97% confident that a labeler would prefer D over B.

Optimizing the policy with PPO

In step 3, generating an answer is treated as a reinforcement learning problem: the policy takes "actions" (choosing tokens), and receives a reward from the reward model when the answer is complete. There is a catch: the reward model is only an approximation of human judgement, and a policy optimized hard against it will find answers which score highly but are actually bad - known as reward hacking. To prevent this, the reward is reduced by a penalty for drifting too far from the SFT model, measured with the KL divergenceClick to read tutorial: Softmax, Cross-Entropy and KL Divergence Explained between the two models' token probabilities -
The policy writes an answer, the reward model scores it, a KL penalty for drifting from the SFT model is subtracted, and PPO updates the policy Prompt x Policy(being trained) Reward modelscore r(x, y) SFT model (frozen)KL penalty reward =r(x, y) - β · KL PPO updates the policy's weights to increase this reward answer y
The RL step: maximize the reward model's score while staying close to the SFT model.
The authors also found that RLHF made the model slightly worse on some standard benchmarks - an alignment tax. Their final models, called PPO-ptx, reduce this by mixing in updates from the original pre-training objective during PPO.

Results

  • Preferred by people: labelers preferred the answers of the 1.3-billion-parameter InstructGPT over those of the 175-billion-parameter GPT-3, despite it having over 100 times fewer parameters. The 175B InstructGPT's answers were preferred to GPT-3's 85% of the time, and to GPT-3 with a carefully written few-shot prompt (see prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples) 71% of the time.
  • More truthful: on the TruthfulQA benchmark, InstructGPT gave truthful and informative answers about twice as often as GPT-3, and on tasks where it should answer only from given text, it made up information about half as often (21% vs 41% of the time).
  • Less toxic: when asked to be respectful, it produced about 25% fewer toxic outputs than GPT-3. It did not, however, show a significant improvement on measures of bias.
  • Generalization: it followed instructions in settings which were rare in its fine-tuning data, such as code questions and instructions in other languages.
  • Cheap: the whole RLHF process used a small fraction of the compute of GPT-3's pre-training.

Limitations

  • The models are aligned to the preferences of a small group of labelers and researchers, mostly English-speaking, not to "human values" in general.
  • InstructGPT still made simple mistakes, could invent facts, and would follow instructions to produce harmful content.
  • RLHF can encourage answers which sound good to a rater over answers which are correct, and can make models overly verbose or hedging.

Impact

In November 2022, OpenAI released ChatGPT, trained with the same method on conversations. RLHF became the standard final stage of training for chat assistants. The pipeline has since been simplified and extended: Direct Preference Optimization (DPO), from 2023, learns from the same preference pairs without a separate reward model or PPO, and methods such as Constitutional AI use feedback generated by an AI model, guided by written principles, in place of some human labels (RLAIF). Reinforcement learning itself became central again for reasoning, with rewards from checking answers automatically, as in DeepSeek-R1Click to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL.

Summary

In this post we went through the InstructGPT paper. We saw why next-token prediction makes a model misaligned with what its users want, and how RLHF uses human judgements to fix this in three steps: supervised fine-tuning on labeler demonstrations, training a reward model on labeler rankings, and optimizing the model against the reward model with PPO, with a KL penalty to keep it close to the SFT model. We computed the reward model's pairwise ranking loss in Java. Lastly, we went through the results - a 1.3B InstructGPT preferred over the 175B GPT-3, more truthful and less toxic answers - the limitations, and how the method led to ChatGPT. In the next post, we will see how language models started to take actions, in the ReAct paperClick to read tutorial: ReAct Paper Explained: Reasoning and Acting in LLMs.

InstructGPT Paper Explained: Training LLMs with RLHF - JavaBrahman