Skip to content
anythingai.space
Training & Adaptationadvanced15 min · 9 steps

RLVR and Reasoning Models

How reasoning models are trained with verifiable rewards and GRPO, how they game imperfect checkers, why you never reward the chain of thought, and why most teams should distill instead.

CoT + answerreward 1 / 0updateCheckable promptsmath · code · logicPolicy LLMwrites long CoTVerifieranswer match · tests passGradient updateadvantages → policy
requesttokensignalheavy workresponseerrortool calldata
1 / 9
Key ideaStep 1 of 9

Reward the answer with a checker

RLVR (reinforcement learning with verifiable rewards) trains reasoning models without a learned reward model. The policy writes a long chain of thought and an answer; a deterministic verifier checks it: did the answer match, did the tests pass, does the proof check.

The reward is often just 1 for correct, 0 for wrong, sometimes plus a small format reward. Long chains of thought, self-verification and backtracking emerge from this loop rather than being supervised. DeepSeek-R1 showed reasoning can be incentivised through pure RL with no human-labelled reasoning traces.

InsightThe strongest verifiers are formal. A proof checker accepts nothing it can't verify, so it scales to outputs no human could review.
What changed
  • +Checkable prompts
  • +Policy
  • +Verifier
  • +Gradient update

Lesson 8 of 8 in Training & Adaptation