Training language models to follow instructions with human feedback
Starting from a pre-trained model, the authors fine-tune on human demonstrations, train a reward model on human rankings of outputs, and then optimise the policy against that reward with PPO. Raters preferred the outputs of a small tuned model over those of a much larger untuned one.
Key ideas
- Demonstrations, then comparisons, then reinforcement learning
- A reward model turns sparse human judgement into a dense training signal
- A KL penalty keeps the tuned policy near the pre-trained one
Why read it now
This three-stage recipe is what turned raw language models into assistants, and it is the reference point for every later alignment method.
Question to keep in mind
What does the reward model actually learn to predict, and what failure modes follow from optimising against it too hard?