The residual stream is the state

In a Transformer, the residual stream — not the attention head, not the FFN — is the actual state of the model.

Each block reads from the stream, computes a delta, and writes it back. Attention is a routing delta. The FFN is a thinking delta. Layer norm is bookkeeping.

This is the claim I wanted when I captured Inbox: a thought from this morning. It stands without the paper; the paper is cited from Attention Is All You Need — working notes.

Links: Attention is a routing function, Maps of content.