Transformers ·Attention ·Self-Attention

Attention

On its own, a token embedding is context-blind: "bit" is the same vector whether a dog did it or you only want a little. Attention is the step that lets each token look at the others and rewrite itself. Here is how, one matrix at a time.

June 28, 2026interactive explainer

the problem: context-blind tokens

One vector. Three meanings.

Before attention, every token is just a lookup. The word "bit" maps to the exact same vector in all three sentences below, even though it means something different in each.

context_blind

the dog bitmeans: bit as in bite: the past tense of a dog biting

token ID 2699

a little bitmeans: a small amount of something

token ID 2699

set the bitmeans: a single bit of data, a 0 or a 1

token ID 2699

context-blindSame word, same ID, same vector. The meaning lives in the neighbors, and nothing has read them yet.

01 / 04 · look up row in W_E

Three sentences, one word

"bit" shows up in three sentences. To a human, context settles the meaning instantly. The model has not looked at the context yet.

embedding · 1024-dim

Three sentences, one token. Token ID 2699 and the embedding are illustrative values; what matters is that all three start identical.

the big picture

One head, end to end.

Before we zoom in, here is the whole thing in one diagram. A single attention head turns each token into a Query, Key, and Value, scores every pair, masks the future, softmaxes into weights, and mixes the Values into a new, context-aware vector. Every scene below zooms into one box.

head_pipeline

01 / 07

Tokens go in

The sentence enters as one vector per token: the, dog, bit.

One attention head as a pipeline. Press play to follow the sentence through each stage; the scenes below zoom into each box.

the machinery: Q, K, V

One vector becomes three.

Attention starts by multiplying each token's embedding by three learned matrices, producing three small vectors: a Query (what it is looking for), a Key (what it advertises), and a Value (what it carries). For example, bit's Query asks "who did the biting?", dog's Key matches it well, and dog's Value carries the content that flows into bit.

recapNew to Q, K, V? The intuition, with the same "the dog bit" example, is in Positional Encoding.

project_qkv

bit embedding · 1024

QQuery q · 64

× W_Q · 1024 × 64

KKey k · 64

× W_K · 1024 × 64

VValue v · 64

× W_V · 1024 × 64

01 / 04

Start from the embedding

Take "bit" as a 1024-dim vector. Every token in the sentence does the same three projections; we follow "bit".

QQuery (Q)

what a token is looking for

e.g. bit: a verb hunting for its subject, who did the biting?

KKey (K)

what a token advertises about itself, like a billboard

e.g. dog: I am a noun, an agent.

VValue (V)

the content a token carries, blended in when it is attended to

e.g. dog's meaning gets pulled into bit.

the projectionEach of the 64 output numbers is the embedding dotted with one column of W_Q. Stack 64 columns and you get the 64-dim Query; the Key and Value work the same way.

embedding

Query

col 1 / 8

Three learned matrices turn one 1024-dim embedding into a 64-dim Query, Key, and Value. Q rose, K violet, V teal. Sizes are illustrative (d_model 1024, head size 64).

why a score is high

A score is just alignment.

A score is a dot product, and a dot product measures alignment: two vectors pointing the same way score high, at a right angle score zero, opposite score negative. Drag dog's Key and watch bit's score rise as the two arrows line up.

dot_product

q\cdot k = |q|\,|k|\cos\theta

drag dog's Key

bit (Query)dog (Key)

score (cosine)0.44

angle 64°

right angle: score near zero

The next scene fills a whole matrix of these scores. Here you can move one pair by hand.

bit's Query is fixed; drag dog's Key around the circle. The closer the arrows point, the larger the dot product. This is why aligned means relevant.

step 1: attention scores

How well does every pair match?

To decide where "bit" should look, attention compares its Query against every Key with a dot product. Do that for all pairs and you get a score for each (Query, Key) cell.

scores · Q·Kᵀ

S = Q\,K^{\top}

Key (looked at)

Query (looking)

thedogbit

the

1.2

-0.5

0.1

dog

bit

low / negativehigh (aligned)

01 / 03

the looks around

First row: "the" compares its Query against every Key. It is a determiner, so nothing lines up strongly.

Q·KᵀRead each row as one token looking out at all three. Brighter means a higher dot product.

Every cell is one dot product: how aligned is Query i with Key j.

The 3 × 3 score matrix for "the dog bit", one attention head. Scores are illustrative values; the pattern (bit aligns with dog) is the point.

step 2: scale and mask

Shrink the scores, mask later tokens.

Two quick fixes before the scores become weights. Divide them down so they stay stable, then block every token from looking ahead.

scale_and_mask

\dfrac{Q\,K^{\top}}{\sqrt{d_k}}

thedogbit

the

1.2

-0.5

0.1

dog

0.8

2.1

-0.3

bit

0.3

1.8

0.9

raw scoresThe raw dot products. Big values would make the next step (softmax) too spiky, pushing almost all the weight onto one token.

left to right

thedogbit

Generation fills the rows one token at a time. Each new token attends to itself and the past, never ahead, so the upper triangle stays dark. That is what makes the model autoregressive.

Scaling keeps softmax well-behaved; the causal mask enforces autoregressive order. Toggle the three states to compare. Values are illustrative.

step 3: softmax

Scores become attention.

Softmax turns each row of masked scores into a clean set of weights: all positive, summing to 1. The -inf cells fall to 0. Pick a token to see its row.

softmax

\text{softmax}(z_i)=\dfrac{e^{z_i}}{\sum_n e^{z_n}}

whose row?

the

30%

dog

37%

bit

33%

row sums to1.00

row · bit

"bit" can see all three. Most of its attention, 37%, goes to "dog", with the rest split across "the" and itself.

Softmax exponentiates each score and normalizes, so the row becomes a probability distribution.

Each row of weights is a probability distribution over what that token attends to. Weights follow an illustrative worked example for "the dog bit".

who attends to whom

The attention map.

Put the weights back onto the words. Each token sends its attention along edges to the tokens it looks at; a thicker, brighter edge means more weight. This is the picture people mean when they say attention. Hover or tap a token to see only its attention.

attention_map

hover a token

attentionbit's strongest edge lands on dog: the verb pulls in its subject. "the" can only attend to itself.

Attention for "the dog bit", one head. Edge thickness is the softmax weight. Causal: no token sends an edge to a later token. Hover or tap a token to isolate its row.

step 4: weighted sum

bit, rewritten by its neighbors.

The final step: take "bit"'s attention weights and use them to mix the Values. The output is a weighted average, so the tokens "bit" attended to flow into its new vector.

weighted_sum · A·V

\text{out} = A\,V

bit's attention · Values (V)

the30%

dog37%

bit33%

new bit · 64

01 / 04

Start with bit's weights

From the softmax step, "bit" attends 30% to "the", 37% to "dog", 33% to itself.

context-blind bit

context-aware bit

A·V37% of "dog" flows into "bit". Same input token, new meaning, decided by the neighbors.

Output = weighted sum of Values. The before/after bars show "bit"'s vector being rewritten. Numbers are illustrative values.

all at once: 16 heads

Sixteen heads, one forward pass.

Everything so far was one head. A real layer runs many in parallel, each with its own W_Q, W_K, W_V. The model here uses 16 heads of size 64. Their outputs are concatenated back to 1024.

forward passA forward pass is one run of the input through the model, from tokens to the next-token prediction. All 16 heads run inside this single pass; no weights change.

multi_head

input · 3 × 1024

what different heads attend to

verb ↔ subject

nearby words

coreference

Same sentence, three heads. Each learned its own attention pattern: one tracks the verb and its subject, one stays local, one links a word back to what it refers to. Concatenating their outputs gives the layer 16 views at once.

concat · 16 × 64 = 1024

context-aware output · 3 × 1024

01 / 03

One input, many heads

The same "the dog bit" goes into all 16 heads. Each head has its own projection matrices, so each builds a different Query, Key, Value.

Sizes at a glance

d_model (embedding): 1024
heads: 16
head size dₖ: 64
√dₖ (scale): 8

16 × 6416 heads × 64 dims = 1024. The split costs nothing in size; it buys 16 independent views of the same sentence.

+ W_OOne detail the diagram skips: the concatenated 1024 vector is then multiplied by a learned output matrix W_O (1024 × 1024). That final projection lets the 16 heads' views mix before leaving the layer.

Multi-head attention: 16 independent heads, each 64-dim, concatenated to 1024. Head roles are illustrative; in practice they emerge and are rarely this clean.

back to the start

Same token, three vectors.

Remember the three sentences where "bit" started as the exact same vector. After attention, each "bit" has attended to different neighbors, so the three come out as three different vectors. The context-blind input became a context-aware output.

same_token_three_vectors

before attention

the dog bit

a little bit

set the bit

01 / 02

Three identical starts

All three "bit" tokens begin as the same vector, exactly as at the top of the page.

attentionThree identical inputs, three different outputs. That is the whole job of attention.

The same "bit" token before attention (identical) and after (three different vectors), one per sentence. Numbers are illustrative.

the takeaway

Same token, new meaning.

Attention projects each token into Query, Key, Value, scores every pair with a scaled dot product, masks the future, softmaxes into weights, and mixes the Values. The context-blind vector comes out context-aware, and 16 heads do it from 16 angles at once, blended back together by a final learned projection.

One thing attention can't do on its own: tell word order apart. "the dog bit the man" and "the man bit the dog" look identical to it. That is what Positional Encoding fixes.