thought

RLHF

RLHF is Reinforcement Learning from Human Feedback. The three-stage InstructGPT pipeline: SFT on demonstrations → reward model on human pairwise comparisons (per Bradley-Terry: comparing “which is better” is more reliable than absolute ratings) → PPO with RM evaluation. The KL penalty keeps the policy close to the reference model: not drifting too far, otherwise the RM evaluation is an unjustified extrapolation. Reverse KL is mode-seeking: it preserves several peaks of high reward. RM overoptimization: the surrogate reward increases, while real quality first increases and then decreases (Goodhart’s Law).