Skip to content
anythingai.space
Module 04

Training & Adaptation

Pretraining, fine-tuning, LoRA, RLHF/DPO, distillation, synthetic data, quantization and reasoning models.

lessons
8
steps
69
time
103 min
FSDP / ZERO: STATE SHARDED ACROSS GPUSPER PARAMETER ≈ 16 BYTES (TYPICAL SPLIT)BF16 weights · 2 BBF16 gradients · 2 BFP32 master copy · 4 BAdam momentum · 4 BAdam variance · 4 BGPU 11/4 of the stateGPU 21/4 of the stateGPU 31/4 of the stateGPU 41/4 of the state
01beginner·12 min·9 steps

Pretraining Basics

How a model soaks up language and knowledge from trillions of tokens, and why data, compute, memory and stability are the hard parts, not the objective.

  • The single objective behind every base model: predict the next token
  • Why data mixture and curriculum matter more than raw volume
  • Chinchilla vs inference-optimal training, and why small models are overtrained
  • Estimate training compute with 6 × N × D
Start →
SFTCONTINUED-PRETRAINING MIXDomain text 90–95%Rehearsal 5–10%Raw domain textall SEC filingsContinued PTLR ≈ 1/10 of originalDomain basespeaks financeSFT pairssmall task setFinance assistantfollows instructions
02intermediate·12 min·9 steps

Fine-Tuning Strategies

When to fine-tune, which kind (SFT, continued pretraining, PEFT), how to set it up, and where fine-tuning can still run in 2026.

  • Decide between prompting, RAG, fine-tuning and continued pretraining
  • Why fine-tuning false facts into weights is dangerous
  • SFT data: why 1,000 perfect examples beat 1,000,000 noisy ones
  • Full fine-tuning vs PEFT, and the hyperparameters that matter
Start →
B (D × R)r1r2123456A (R × D)123456r1r2ΔW = B · A (RANK 2)123456123456d = 4096, r = 8 → 2·r·d = 65,536 params vs d² = 16.8M (≈0.4%)
03intermediate·13 min·8 steps

LoRA, QLoRA and PEFT

Freeze the giant matrix, train two skinny ones: how LoRA and QLoRA make fine-tuning a 70B model fit on one GPU, and how adapters are served at scale.

  • Why full fine-tuning of a large model doesn't fit
  • The LoRA equation: frozen W plus a trainable low-rank B·A
  • How QLoRA fits a 70B fine-tune on one 80 GB GPU
  • When to try DoRA, VeRA or rsLoRA
Start →
ALIGNMENT STAGESSFT5k–10k samplesDPO 1instruction followingDPO 2safety + tonepost-training →Base modelSFT modelAligned v1Aligned v2
04intermediate·13 min·9 steps

RLHF and DPO

How preference data steers a model: the classic RLHF pipeline, why DPO replaced it for most teams, online alignment, and where rubric and reasoning rewards go wrong.

  • The three stages of RLHF and why PPO is heavy
  • How DPO removes the reward model
  • Why online alignment beats static preference data
  • How rubric rewards reintroduce reward hacking, and the fixes
Start →
PROMPTThecapitalofAustraliais?HARD LABELCanberra1Sydney0Melbourne0Perth0SOFT LABELS (ILLUSTRATIVE)Canberra.71Sydney.20Melbourne.07Perth.02Teacher100B+ params, often MoE
05intermediate·12 min·8 steps

Knowledge Distillation

How a large teacher model trains a small student: soft labels, temperature, on-policy and self-distillation, and why API operators now treat distillation as a threat.

  • Why small models are trained on a bigger model's outputs
  • Soft labels and temperature: what the student actually learns
  • On-policy distillation and its contested mechanism
  • Self-distillation via rejection sampling
Start →
Seed: “Write a function to add two numbers.”Evolved: “Write a thread-safe Python class for matrix addition with error handling andunit tests, following PEP 8.”Seed taskEvolver LLMIn-depthconstraints · stepsIn-breadthnew, rarer taskEliminatedrop failures
06intermediate·12 min·8 steps

Synthetic Data Generation

With high-quality human text running out, models now train largely on model-generated data. How to evolve it, critique it, verify it, filter it, and keep its lineage.

  • Why synthetic data became the main engine after the data wall
  • Evol-Instruct and Constitutional AI loops
  • What model collapse is and how verification prevents it
  • A multi-stage filtering pipeline for millions of rows
Start →
NVFP4: ONE FP8 SCALE PER 16 VALUESscale E4M3vvvvvvvvvvvvvvvvMXFP4: ONE POWER-OF-TWO SCALE PER 32 VALUESscale E8M0vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvFP8 VARIANTSE4M3 · weights + activationsE5M2 · gradients (more range)
07advanced·14 min·9 steps

Quantization Deep Dive

Fewer bits per weight: how BF16 becomes FP8, 4-bit and below, which methods keep quality, why kernels decide speed, and why the KV cache matters as much as the weights.

  • Weight size and quality across BF16, FP8, 4-bit, 2-bit and ternary
  • Why bit width alone doesn't predict quality
  • How AWQ protects salient channels and how FP4 block scaling works
  • Why lower precision isn't automatically faster
Start →
RLHF: LEARNED REWARDRLVR: PROGRAMMATIC CHECKexploitable scoregrounded rewardRemoves the easiest exploit, not all ofthem.PolicyReward modelneural net · blind spotsPolicy LLMwrites long CoTVerifierexternal oracle
08advanced·15 min·9 steps

RLVR and Reasoning Models

How reasoning models are trained with verifiable rewards and GRPO, how they game imperfect checkers, why you never reward the chain of thought, and why most teams should distill instead.

  • The RLVR loop and why a verifier is harder to hack than a reward model
  • How GRPO replaces the critic with a group baseline
  • Zero-variance collapse, length bias, and the variants that fix them
  • Why you never put reward pressure on the chain of thought
Start →
Read deeper

Guide chapters · Training & Adaptation

Next module · 05
Inference & Serving