Skip to content
anythingai.space
Training & Adaptationintermediate12 min · 9 steps

Fine-Tuning Strategies

When to fine-tune, which kind (SFT, continued pretraining, PEFT), how to set it up, and where fine-tuning can still run in 2026.

New factspolicy changed last weekStrict JSON outputTone / personaDomain vocabularymedical, legal, SECRAGupdate a documentFine-tune (SFT)format + behaviourContinued pretrainingraw domain text
requesterrordataresponseheavy work
1 / 9
ProblemStep 1 of 9

Fine-tuning teaches form, not facts

Before training anything, ask: can prompting or RAG solve this? Fine-tuning is now less about teaching facts and more about teaching format and behaviour.

LLMs are bad at memorising facts from fine-tuning, and RAG is far easier to update. But a reliable JSON schema, a consistent persona, or dropping a long few-shot prompt to cut latency are all good fine-tuning jobs. Specialised vocabulary calls for continued pretraining.

RequirementBetter solution
New facts / knowledgeRAG
Specific output formatFine-tuning
Tone / personaFine-tuning
Latency (drop few-shot prompt)Fine-tuning
Private domain languageContinued pretraining
What changed
  • +Four common needs
  • +The right tool for each

Lesson 2 of 8 in Training & Adaptation