Attention Is All You Need — working notes

Working copy of Attention Is All You Need. Not a summary of the PDF — a translation I can reread without opening arXiv.

The move that matters: drop recurrence. Sequence structure is injected with positional encodings; mixing happens only through attention.

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$$

$\sqrt{d_k}$ is not decoration. Without it, dot products grow with dimension and the softmax saturates — gradients die. That single line is why Attention is a routing function rather than a memory.

Block	Job
Encoder self-attention	Mix the source with itself
Decoder masked attention	Mix the target so far, no peeking
Cross-attention	Route decoder queries into encoder values
FFN	Think, per position, after the mix

Two things I still do not believe I understand:

1. Why eight heads, not two or thirty-two, beyond “it worked.”
2. How much of “attention is all you need” is actually “residuals plus a wide FFN.” [^1]

Related: Permanent notes must stand alone, Maps of content.

See the distilled claim: Attention is a routing function.