Training & Adaptation
Pretraining, fine-tuning, LoRA, RLHF/DPO, distillation, synthetic data, quantization and reasoning models.
- lessons
- 8
- steps
- 69
- time
- 103 min
Pretraining Basics
How a model soaks up language and knowledge from trillions of tokens, and why data, compute, memory and stability are the hard parts, not the objective.
- The single objective behind every base model: predict the next token
- Why data mixture and curriculum matter more than raw volume
- Chinchilla vs inference-optimal training, and why small models are overtrained
- Estimate training compute with 6 × N × D
Fine-Tuning Strategies
When to fine-tune, which kind (SFT, continued pretraining, PEFT), how to set it up, and where fine-tuning can still run in 2026.
- Decide between prompting, RAG, fine-tuning and continued pretraining
- Why fine-tuning false facts into weights is dangerous
- SFT data: why 1,000 perfect examples beat 1,000,000 noisy ones
- Full fine-tuning vs PEFT, and the hyperparameters that matter
LoRA, QLoRA and PEFT
Freeze the giant matrix, train two skinny ones: how LoRA and QLoRA make fine-tuning a 70B model fit on one GPU, and how adapters are served at scale.
- Why full fine-tuning of a large model doesn't fit
- The LoRA equation: frozen W plus a trainable low-rank B·A
- How QLoRA fits a 70B fine-tune on one 80 GB GPU
- When to try DoRA, VeRA or rsLoRA
RLHF and DPO
How preference data steers a model: the classic RLHF pipeline, why DPO replaced it for most teams, online alignment, and where rubric and reasoning rewards go wrong.
- The three stages of RLHF and why PPO is heavy
- How DPO removes the reward model
- Why online alignment beats static preference data
- How rubric rewards reintroduce reward hacking, and the fixes
Knowledge Distillation
How a large teacher model trains a small student: soft labels, temperature, on-policy and self-distillation, and why API operators now treat distillation as a threat.
- Why small models are trained on a bigger model's outputs
- Soft labels and temperature: what the student actually learns
- On-policy distillation and its contested mechanism
- Self-distillation via rejection sampling
Synthetic Data Generation
With high-quality human text running out, models now train largely on model-generated data. How to evolve it, critique it, verify it, filter it, and keep its lineage.
- Why synthetic data became the main engine after the data wall
- Evol-Instruct and Constitutional AI loops
- What model collapse is and how verification prevents it
- A multi-stage filtering pipeline for millions of rows
Quantization Deep Dive
Fewer bits per weight: how BF16 becomes FP8, 4-bit and below, which methods keep quality, why kernels decide speed, and why the KV cache matters as much as the weights.
- Weight size and quality across BF16, FP8, 4-bit, 2-bit and ternary
- Why bit width alone doesn't predict quality
- How AWQ protects salient channels and how FP4 block scaling works
- Why lower precision isn't automatically faster
RLVR and Reasoning Models
How reasoning models are trained with verifiable rewards and GRPO, how they game imperfect checkers, why you never reward the chain of thought, and why most teams should distill instead.
- The RLVR loop and why a verifier is harder to hack than a reward model
- How GRPO replaces the critic with a group baseline
- Zero-variance collapse, length bias, and the variants that fix them
- Why you never put reward pressure on the chain of thought