Skip to content
anythingai.space
Training & Adaptationintermediate12 min · 8 steps

Synthetic Data Generation

With high-quality human text running out, models now train largely on model-generated data. How to evolve it, critique it, verify it, filter it, and keep its lineage.

not enoughPHI-4 PRETRAINING TOKENSSynthetic 40%Web, code, books, other 60%Human web textfinite, licensing riskNext modelwants 15T–45T tokens
datarequestresponsesignalerror
1 / 8
ProblemStep 1 of 8

The data wall

Disclosed pretraining budgets already reach 15T–45T tokens, and the supply of high-quality, legally clean human text isn't growing that fast. That's the data wall.

So post-training sets are now mostly model-generated: reasoning traces, rewrites, verified solutions. Synthetic shares of pretraining are rising too. Microsoft's Phi-4 report gives 40% of pretraining tokens to synthetic data. Most labs don't publish their mix.

Human dataSynthetic data
VolumeFiniteUnbounded
QualityVariable, noisyControllable
CostHigh (labellers)Cheap (GPU inference)
BiasMirrors the internetCan be balanced
What changed
  • +Human text supply (strained)
  • +Phi-4's synthetic share

Lesson 6 of 8 in Training & Adaptation