Transformers Turn Attention Into an Architecture

A vision transformer trades convolution's built-in locality for learned routing, and the training recipe pays the difference.

The attention note left routing as one operation inside a recurrent model. A transformer promotes routing to the backbone. Recurrence goes away, and the model reads every token in parallel. In language the tokens are words or subwords. In vision they are image patches.

Self-attention makes the path between two positions short

Inside self-attention every token reads from other tokens. A patch gathers evidence from a distant patch in one step. A word gathers evidence from earlier or later words, depending on the setup.

A recurrent model carries one compressed summary forward, and that compression is the ceiling. Attention removes the carry.

Stack the blocks and the effect compounds. Each layer builds a table of relationships between the visible positions. Then it updates every patch with the evidence attention routes to it. Local patch evidence becomes global object evidence.

Patch size is a modeling decision you make before training

Patchification is the entry point. The model cuts the image into patches, embeds each patch, adds positional information, and feeds the sequence into transformer blocks. After that step the architecture looks language-like.

The patch is a rough unit. Small patches keep fine structure and lengthen the sequence, and large patches cut compute and lose detail. Tokenization is part of the model, so set the patch size against the smallest structure your label depends on.

The patch grid also changes how the pressure list from the overview note reaches the model. Translation now moves evidence between patches instead of sliding it under one shared filter. Scale decides how many patches an object covers.

Attention needs position because it does not have one

Attention is mostly permutation-friendly. Shuffle the tokens and the operation barely notices. The model needs positional information to know where a token came from.

In vision that information carries weight. A patch at the top of the image and the same patch at the bottom are not the same evidence. Position is the only thing that separates them.

Multi-head attention gives the model several routing channels

Different heads learn different routing patterns. One head stays local while another connects object parts, and a third reaches for global context. The gain is capacity for several relation types at once.

Do not over-romanticize a single head. A tidy caption for one head is something a person writes after training.

Less built-in bias means the data must teach what the architecture no longer assumes

The convolution note named the two assumptions a ConvNet ships with: locality and weight sharing. Nearby pixels belong together, and the model reuses one filter across every position. Patch tokens keep much less of both.

A vanilla transformer must learn that structure from data and pretraining. Vision transformers became convincing alongside scale, stronger augmentation, large datasets, and self-supervised pretraining. Architecture details hand back some of the prior too, and the training recipe carries a lot of the win.

The same recipe teaches whatever else the data contains. The visualization note's shortcut hunt transfers to attention maps without a change of tools. A route can land on the background, a repeated texture, or a dataset artifact and still produce the right label. A clean attention map is confidence under the learned representation, and nothing more.

Small data, tight latency, and dominant local texture still favor ConvNets

I still reach for a convolutional backbone when the data is small and the latency budget is tight. Local texture and spatial hierarchy can dominate a task, and convolution already assumes both.

A transformer is a trade. Take it when distant parts of the image decide the label and you can pay the data bill.

The Builder Test

Name the smallest structure your label depends on. Measure it against your patch size. If that structure fits inside one patch, the grid is fine. If it straddles several patches, attention must reassemble it, and that reassembly is what you test first.

Then name what supplies the locality prior you removed. If you cannot name one, you dropped a prior and put nothing in its place.

Inspect what a patch attends to. If the route does not match the evidence a person uses, the model solves a different task.

What Carries

Carry the routing view. A transformer learns which pieces of the image must talk to each other. The architecture says almost nothing about which pieces those are.

Price the prior before you remove it. When nothing in the recipe teaches locality, the patch grid buys flexibility you cannot use.

Routing and recurrence both end at a label. The next family must produce the image itself.