Vision · 2020 · attention meets vision
Vision Transformer
Images treated as sequences of patches processed by a transformer: no convolutions, spatial relationships learned through attention and position embeddings.
ModLensVision
Interactive Diagram
Focus the lens
Click any component to read what it does. Signal direction follows the edges.
inputImage. No spatial assumptions retained beyond the patch grid.
Core idea
Drop the spatial inductive bias and let attention discover it. With enough data, a generic sequence model matches and then exceeds convolutional vision.
Why it exists
Convolutions limit receptive fields; self-attention gives every patch direct access to every other, and scales well with data and compute.
Mathematics
\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V
Data Flow
What moves through the system
- 01Image is split into fixed-size patches.
- 02Patches are linearly projected into tokens.
- 03Transformer encoder mixes information globally via attention.
- 04A class token or pooled tokens feed the head.
Strengths
- + Global context from layer one
- + Scales predictably with data/compute
- + Unified backbone across modalities
Limitations
- − Weak inductive bias: data-hungry
- − Quadratic attention cost at high resolution
Applications
- · Large-scale recognition
- · Multimodal encoders
- · Dense prediction with adapted variants