Skip to content
anythingai.space
AI Foundationsintermediate13 min · 9 steps

Attention, Self-Attention and Multi-Head Attention

How a model learns where to look: queries score keys, softmax turns scores into weights, and many heads look in different ways at once.

SOURCE SENTENCE — EARLY WORDS FADETheblackcatthatweadoptedlastyearsatEncoderreads the sourceContext vectorfixed size1 vectorDecoderwrites the target
datarequest
1 / 9
ProblemStep 1 of 9

One vector can't hold a whole sentence

Early neural translators read the source sentence with a recurrent encoder and squeezed it into one fixed-size vector. The decoder then had to write the whole translation from that single summary.

Short sentences survived. Long ones lost detail — by the time the encoder reached the last word, the first ones had faded.

Watch outA fixed-size summary means a fixed amount of memory, no matter how long the input is.

Lesson 21 of 34 in AI Foundations