Tag
#claim
- #essay 4
- #books 3
- #method 3
- #reading 3
- #transformers 3
- #attention 2
- #claim 2
- #garden 2
- #history 2
- #literature 2
- #architecture 1
- #devonthink 1
- #moc 1
- #publishing 1
- #retrieval 1
- #structure 1
- #trails 1
- #zettelkasten 1
- Attention is a routing function
Self-attention does not store. It routes: each position writes a query, every position offers a key, and values are fetched in proportion to the match. That is
- The residual stream is the state
In a Transformer, the residual stream — not the attention head, not the FFN — is the actual state of the model. Each block reads from the stream, computes a del