Direct Preference Optimization: Your Language Model is Secretly a Reward Model
DPO shows that the reinforcement learning step of RLHF has a closed-form solution, so the policy can be trained directly on preference pairs with a simple classification-style loss. No separate reward model, no sampling loop, and results that match PPO-based training on the tasks studied.
Key ideas
- The optimal RLHF policy can be written in terms of the reward and the reference model
- Reparameterising the reward removes the need to learn it
- The loss raises the likelihood of preferred answers relative to rejected ones, scaled by the reference
Why read it now
It replaced a fragile RL pipeline with one loss function and is now the default starting point for preference tuning.
Question to keep in mind
What does DPO give up compared with on-policy RL, and when would that matter?