Training & Adaptationintermediate13 min · 9 steps
RLHF and DPO
How preference data steers a model: the classic RLHF pipeline, why DPO replaced it for most teams, online alignment, and where rubric and reasoning rewards go wrong.
requesterrordataresponsesignalheavy worktoken
ProblemStep 1 of 9
Pretrained models know a lot and obey nothing
A pretrained model is knowledgeable but uncontrolled. Ask it something and it may produce harmful content, ignore the instruction, or hallucinate freely.
Alignment steers behaviour toward human intent, using a reward signal and policy updates.
What changed
- +Three ways an unaligned model fails
Lesson 4 of 8 in Training & Adaptation