01The problem attention was built to solve

Before attention, sequence models read a sentence one word at a time and squeezed everything they had read into a single fixed-length summary vector. For a long sentence, that bottleneck threw away detail — the model had to remember the whole input through one cramped representation. Bahdanau, Cho & Bengio (2014) introduced attention to fix exactly this: instead of relying on one summary, the model is allowed to look back at every input word and decide, for each step, which words matter most right now. They called the learned weighting a soft alignment between output and input.

The Transformer (Vaswani et al., 2017) took this idea to its logical extreme. Its paper title — "Attention Is All You Need" — says it plainly: it threw out recurrence and convolution entirely and built the whole model out of attention. To understand a Transformer, you really only need to understand this one operation well.

• The bottleneck: older models compressed a whole sentence into one fixed vector, losing detail on long inputs.
• The fix: attention lets the model look back at all words and weight them per step — a learned "soft alignment" (Bahdanau et al., 2014).
• The leap: the Transformer is built almost entirely from attention, dispensing with recurrence and convolution (Vaswani et al., 2017).

02Queries, keys & values — the three roles of a word

Attention reframes each word as playing three roles at once. A useful mental model is a library lookup. Every word produces a query ("what am I looking for?"), a key ("what do I offer as a match?"), and a value ("what information do I carry?"). All three are just linear projections of the word's representation — the same vector pushed through three different learned weight matrices (Vaswani et al., 2017; Jurafsky & Martin, SLP3 ch. 9).

To compute attention for one word, you take its query and compare it against every word's key. A close match means "this word is relevant to me." Those match scores become weights, and the output is a weighted sum of the value vectors — you pull in more of the values from words you matched strongly, and less from the rest. That weighted blend, not any single word, is what attention produces.

• Query (Q): what this word is looking for in the others.
• Key (K): what each word advertises, to be matched against queries.
• Value (V): the actual information a word contributes once it is selected.
• Q, K, and V are three learned linear projections of the same input — and the output is a weighted sum of values, weighted by query-key match.

03Scaled dot-product attention, step by step

Here is the whole operation in one line, exactly as the Transformer defines it (Vaswani et al., 2017):

Read it left to right. Q Kₜ takes the dot product of each query with every key — a raw compatibility score: bigger means more aligned. Those scores are then divided by √dₖ (the square root of the key dimension). This scaling matters: with large key dimensions the dot products grow large, which pushes the softmax into regions where its gradients are tiny; dividing by √dₖ keeps them in a healthy range. softmax turns the scaled scores into a set of positive weights that sum to one — the attention distribution. Finally, multiplying by V produces the weighted sum of value vectors.

Earlier work explored other ways to score query-key compatibility — Bahdanau et al. (2014) used an additive (concatenation) score, and Luong et al. (2015) compared multiplicative (dot-product) scoring plus global vs. local attention. The Transformer settled on scaled dot-product attention because it is fast and parallelizable as plain matrix multiplication.

• Score: dot-product of each query with every key (Q Kₜ).
• Scale: divide by √dₖ to keep softmax gradients stable at large dimensions.
• Normalize: softmax turns scores into weights summing to 1.
• Blend: multiply weights by the value vectors to get the output.

04Self-attention, masking & why position has to be added

Self-attention is the case where Q, K, and V all come from the same sequence — every position attends to every other position in the same sentence, letting a word gather context from its neighbours and beyond. Cross-attention instead lets one sequence attend to another, e.g. a decoder attending to an encoder's output during translation (Vaswani et al., 2017; Hugging Face docs). When a model generates text left to right, it uses masked (causal) attention so a position can only attend to earlier positions, never to words it has not produced yet.

One subtle but important consequence: attention is permutation-invariant over the set of tokens. The operation itself has no notion of word order — shuffle the inputs and the math does not change. So position must be added explicitly. The original Transformer used fixed sinusoidal positional encodings; later work added relative position representations (Shaw et al., 2018) and rotary embeddings / RoPE (Su et al., 2021), which encode relative position by rotating the query and key vectors.

• Self vs cross: self-attention stays within one sequence; cross-attention bridges two (e.g. decoder → encoder).
• Causal masking: in generation, a token may only attend to earlier tokens.
• Position is bolted on: attention ignores order, so positional information is added — sinusoidal, relative, or rotary (RoPE).

05Multi-head attention — and a live look inside

A single attention calculation can only capture one kind of relationship at a time. Multi-head attention runs several attention operations in parallel, each on its own lower-dimensional projection of Q, K, and V, then concatenates and re-projects the results (Vaswani et al., 2017). The point is specialization: different heads can learn to attend to different kinds of relations — one might track which verb a subject belongs to, another which noun a pronoun refers to. No head is told what to look for; the roles emerge from training.

The interactive below makes this concrete. Pick a query word in the sentence and the chart shows the attention weights that word places on every other word — computed live from toy query and key vectors with the real scaled-dot-product-then-softmax pipeline. Switch heads to see how different heads concentrate on different words. The numbers here are illustrative, designed to show the shape of attention; they are not from a trained model.

• Many heads, one layer: several attention operations run in parallel on separate projections, then are concatenated and re-projected.
• Specialization emerges: different heads learn to attend to different relations — this is learned, not hand-designed.
• Each head still uses the same softmax(QKₜ/√dₖ)V math — multi-head just runs it several times in different subspaces.

06The catch: attention scales quadratically

Because every token attends to every other token, standard self-attention's time and memory cost grows quadratically with sequence length — double the context and you roughly quadruple the work. This is the main bottleneck for long inputs (Dao et al., 2022; Beltagy et al., 2020), and a great deal of research targets it. The variants split into two camps that you should not conflate:

07Check your understanding

08Take it with you & go deeper