Foundation · 2017 · the default substrate
Transformer
The architecture that replaced recurrence with attention: parallelizable sequence modeling whose scaled descendants power modern foundation models.
ModLensFoundation
Interactive Diagram
Focus the lens
Click any component to read what it does. Signal direction follows the edges.
inputTokens + Pos. Discrete inputs embedded into a continuous space with order restored by positions.
Core idea
Let every element query every other element directly. Attention routes information by content, not by distance, and the whole sequence trains in parallel.
Why it exists
Recurrent models bottleneck information through time steps; attention removes the bottleneck and the sequential training constraint at once.
Mathematics
\mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V
\mathrm{MultiHead} = \mathrm{Concat}(h_1,\dots,h_h)W^O
Data Flow
What moves through the system
- 01Tokens are embedded with positional information.
- 02Attention layers mix information across the sequence.
- 03Feed-forward layers transform each position independently.
- 04Residuals and normalization stack these into deep networks.
Strengths
- + Parallel training
- + Direct long-range connections
- + Scales with data and compute
Limitations
- − Quadratic attention cost
- − Weak locality prior without structure
Applications
- · Language
- · Vision
- · Speech
- · Multimodal foundations