Positional Encoding
Attention works out which words matter to each other, but on its own it cannot tell "the dog bit the man" from "the man bit the dog". Here is how a transformer is told where each token sits.
same word, three meanings
One word. Three meanings.
Before we get to order, a quick recap of what attention already solved. The token "bit" shows up in three sentences and means something different each time.
starting embedding
means: a single bit of data, a 0 or a 1
starting embedding
means: bit as in bite, the past tense of a dog biting
starting embedding
means: a small amount of something
Attention fixed meaning. But it still has a blind spot: word order.
the machinery: Q, K, V
Query, Key, Value.
Attention "looks at neighbors", but how? Every token produces three vectors: a Query (what it is looking for), a Key (what it advertises), and a Value (what it carries). Here is one attention step on "the dog bit", with bit doing the looking.
01 / 04 · match (Q · K)
Query: what bit is looking for
Every token casts a Query, the thing it wants to find. bit is a verb, so its Query hunts for a subject: who did the biting?
what a token is looking for
e.g. bit: a verb hunting for its subject, who did the biting?
what a token advertises about itself, like a billboard
e.g. dog: I am a noun, an agent.
the content a token carries, blended in when it is attended to
e.g. dog's meaning gets pulled into bit.
the blind spot: word order
Same words. Opposite meaning.
"The dog bit the man" and "the man bit the dog" use the exact same tokens. Only the order differs. Flip between them and watch what attention outputs.
tokenized
The number under each word is its token ID, the integer index of that word in the vocabulary (not a 0-1 value). Both sentences use the exact same token IDs, only the order differs. (IDs are the course's illustrative values.)
Output Embeddings
Additive Positional Embeddings
Give every token its address.
The first fix: before a token enters the transformer, add a second vector that says where it sits. Token embedding plus position vector equals a position-aware embedding.
Look at "man": position 4 in one sentence, position 1 in the other. Different position, different sum, so order finally reaches the model.
Each position vector is the same size as the token embedding (1024-dim here), so the two can be added element by element.
Fixed: produced by a set formula (the original Transformer's sinusoidal encoding). Never trained, defined for any position.
but language is relative
Position is relative, not absolute.
Absolute positions ("you are token 3") are a clumsy fit for language. What usually matters is the distance between words, and that is exactly what absolute encoding misses.
Push "Neither" and "nor" apart and the absolute gap grows, but the grammatical link is identical. Relative distance is what carries the meaning.
"is" must agree with its subject "cat", not with the closer noun "mat". The link reaches across the sentence, no matter the absolute slots.
Rotary Positional Embeddings
RoPE: rotate, don't add.
Most large language models today (Llama, Qwen, Mistral, DeepSeek) use Rotary Positional Embeddings. Position is not added to the input. It is applied inside attention, by rotating the Query and Key vectors by an angle set by each token's position. Here is what that buys you, step by step.
01 / 05
An attention score is alignment
Take the Query of "bit" and the Key of "dog". Their attention score is just how aligned the two vectors are: a smaller angle means a higher score, which means pay more attention here.
gap 20°
Inside one attention head
Q and K get rotated. V does not: position should decide who looks at whom, not what content gets carried.
Rotate q by its position, rotate k by its position. Their dot product then depends only on the gap m - n, so attention reads relative distance directly, no extra step.
- q, k
- the Query and Key vectors
- R_m, R_n
- rotate by position m and n
- ⟨ , ⟩
- dot product, the attention score
- m - n
- the gap between the two positions (relative distance)
- g(...)
- depends only on q, k and that gap
No Positional Embedding
NoPE: maybe the mask already knows.
A research finding: under some conditions a model learns word order with no positional encoding at all. The causal mask on its own leaks position.
The causal mask decides who may look at whom. Each row is a token doing the looking (its Query); each column is a token it might look at (its Key). A token may attend to itself and everything before it, never to a token in the future.
It is still an open question: how much position information must we hand the model, and how much can it infer from the input's own structure?
the four methods
Four ways to encode position.
Where each one injects position, whether it is absolute or relative, and whether a max-sequence limit applies.
| Method | Where | Type | Max-seq limit | Notes |
|---|---|---|---|---|
Fixed (sinusoidal) | Added to input embedding | Absolute | None (formula covers any position) | Original Transformer. Not trained. |
Learned | Added to input embedding | Absolute | Yes (matrix row count) | Trained like token embeddings. Simple, but two limits. |
RoPEtoday's standard | Inside attention, on Q and K | Relative | None (context extension) | Today's standard: Llama, Qwen, Mistral, DeepSeek. |
NoPE | Nowhere (no explicit encoding) | Implicit (causal mask) | None | Research finding. Works under some conditions. |
the takeaway
Attention is not all you need.
Attention decides who looks at whom. Positional encoding adds where. RoPE rotates Query and Key so relative position falls straight out of the dot product, with no max-sequence wall in sight.