How LLMs Workbeginner12 min · 8 steps
LLM Internals
Why transformers replaced RNNs, how Mixture-of-Experts splits memory from compute, and why serving a model is a bandwidth problem.
datarequesttoken
ProblemStep 1 of 8
Before 2017: one token at a time
RNNs and LSTMs read a sentence left to right, carrying a hidden state from step to step. Step t can't start until step t−1 finishes, so training can't be parallelised.
Long-range links are hard too. In “The animal didn't cross the street because it* was too tired”, the meaning of it* has to survive six hops through hidden states to reach animal.
InsightThink of an RNN as a single-threaded pipeline where every stage waits on the previous one.
← Previous lessonPrivacy-Preserving ML: Federated Learning and Differential PrivacyNext lesson →Tokenization Deep Dive
Lesson 1 of 6 in How LLMs Work