Attention is the mechanism that lets a model decide, for every word it is processing, which earlier words actually matter. It is the idea that made modern language models possible, and it can be explained without a single equation.
Third in a series on how language models actually work.
The problem it solves
Consider: The trophy did not fit in the suitcase because it was too big.
What does it refer to? The trophy. Change one word — because it was too small — and now it is the suitcase. Nothing about the word it changed. What changed is which other word it needs to be read against.
Earlier architectures processed text strictly in order, carrying a running summary forward. Information from far back had to survive every intervening step, and it usually did not. Attention removes the relay entirely: any position can look directly at any earlier position, however distant.
What actually happens
For each word, the model produces three things. A query, meaning roughly “what am I looking for?”. A key, meaning “what do I offer?”. And a value — the content it passes on if selected.
Every word’s query is compared against every earlier word’s key. Strong matches score highly. Those scores are normalised into weights that sum to one, and the result is a blend of the values, weighted accordingly. A word that matters heavily contributes most of the blend; irrelevant words contribute nearly nothing.
The useful mental image is a soft lookup. Not “fetch record 4”, but “fetch 70 percent of the trophy, 20 percent of suitcase, and a scattering of everything else”.
Many heads, looking for different things
One set of comparisons can only track one kind of relationship. Real models run dozens in parallel — attention heads — each with its own queries and keys, and therefore its own notion of what is worth looking at.
When researchers inspect trained models they find heads that have specialised without being told to: some follow grammatical agreement, some track quotation boundaries, some simply look at the previous token. Nobody assigned those roles. They emerged because they reduced prediction error.
Why this made models scale
The practical breakthrough was not only quality — it was that every position can be computed at the same time. Sequential architectures had to finish word one before starting word two. Attention compares everything against everything in one go, which maps beautifully onto GPUs. That parallelism is what made it economic to train on internet-scale text.
The bill arrives with length. Comparing every position against every other means cost grows with roughly the square of the sequence. Double the input and the attention work quadruples. This is the direct reason long context windows were hard to obtain, expensive to serve, and the subject of a decade of engineering — sparse patterns, sliding windows, caching — all aimed at avoiding the full comparison.
Only backwards
While generating, a word may attend only to what precedes it. Future positions are masked, because at that point they do not exist. This is what makes generation causal, and it connects back to the first article in this series: the model commits to each token without seeing what follows, because it structurally cannot.
What to take away
Attention lets every position pull directly from every earlier position, weighted by relevance the model learned to judge. It is the reason models handle long-range dependencies well, the reason they parallelise, and — through its quadratic cost — the reason context length is expensive. Almost every architectural argument in modern AI is about how to keep the benefit while avoiding that bill.
Next in the series: temperature and top-p, and how to control the model’s willingness to gamble.