Skip to content
anythingai.space
Generative AIbeginner12 min · 9 steps

Language Model Families

How text generation works — next-token prediction — and the families built on it: encoder-only BERT, encoder-decoder T5, the decoder-only GPT line through ChatGPT, and open LLaMA.

A TRIGRAM MODEL PREDICTS FROM THE LAST TWO WORDS ONLYThebankraiseditsratessotheriver→ ???Everything greyed out is invisible to themodel.
signalresponserequestdataheavy work
1 / 9
ProblemStep 1 of 9

Rules and n-grams only see a few words back

Older text generators were hand-written rules or statistical n-gram models that pick the next word from the last two or three. They have no idea what the sentence, let alone the document, is about.

Ask one to continue “The bank raised its rates, so the river…” and it can't tell which bank it's in. The meaning lives further back than its window reaches.

Watch outN-gram models are now considered outdated: they can't use long-range context.

Lesson 2 of 8 in Generative AI