Skip to content
anythingai.space
Training & Adaptationintermediate12 min · 8 steps

Knowledge Distillation

How a large teacher model trains a small student: soft labels, temperature, on-policy and self-distillation, and why API operators now treat distillation as a threat.

outputs · logits · tracesDOCUMENTED DISTILLED STUDENTSGemma 2 & 3Llama 3.2 1B/3BLlama 4 MaverickR1-Distill Qwen/LlamaNemotron coderTeacher100B+ params, often MoEStudent1B–30B params
datatokensignalresponserequesterror
1 / 8
Key ideaStep 1 of 8

Small models learn from big ones

Many small models aren't trained on raw web data alone. They learn from a larger teacher: its outputs, its logits, or its reasoning traces. That's a big reason today's small open models punch above their size.

The documented cases are open models. Closed labs don't publish small-model recipes, so "Haiku is distilled from Opus" is plausible, not documented.

StudentTeacher signal
Gemma 2 (2B, 9B), all Gemma 3Distilled from a larger teacher in pretraining
Llama 3.2 1B / 3BLogits from Llama 3.1 8B and 70B
DeepSeek-R1 distills~800K samples generated by R1
NVIDIA competitive-coding model477,642 traces from GLM-5.2 (MIT)
What changed
  • +Teacher
  • +Student
  • +Knowledge flowing between them

Lesson 5 of 8 in Training & Adaptation