Skip to content
anythingai.space
How LLMs Workbeginner11 min · 9 steps

Tokenization Deep Dive

Models read subword tokens, not letters or words — and that one fact drives cost, context limits, multilingual fairness and some famous failures.

How many r’s in“strawberry”?WHAT YOU SEEstrawberryWHAT THE MODEL SEESstrawberryThe three r’s are buried inside two tokens
datatoken
1 / 9
ProblemStep 1 of 9

The model never sees the letters

Ask a model “How many r’s are in strawberry?” and it often gets it wrong. Not because it can’t count — because it never saw the letters.

A tokenizer turns the word into subword pieces like str · aw · berry. Counting letters means reasoning about the inside of tokens the model only knows as whole units.

TipWorkaround from the guide: ask the model to spell the word out character by character first, then count. That forces one token per letter.

Lesson 2 of 6 in How LLMs Work