Foundation · 2017/2020– · sparse scaling
Mixture of Experts
Scaling capacity without scaling compute per token: feed-forward layers are replaced by many expert subnetworks, with a router selecting a few per token.
ModLensFoundation
Interactive Diagram
Focus the lens
Click any component to read what it does. Signal direction follows the edges.
inputToken. One representation, about to choose its specialists.
Core idea
Not every token needs every parameter. Route each token to the specialists best suited to it, and total capacity grows while per-token cost stays flat.
Why it exists
Dense scaling pays for every parameter on every token; sparsity decouples total knowledge from per-token compute.
Mathematics
y = \sum_{i \in \mathrm{TopK}} g_i(x)\, E_i(x)
Data Flow
What moves through the system
- 01Token representations reach a routing layer.
- 02A gate scores experts and selects the top-k.
- 03Selected experts process the token; outputs are weighted-summed.
- 04Training balances load across experts.
Strengths
- + Large capacity at fixed inference cost
- + Specialization emerges across experts
Limitations
- − Routing is a learned single point of failure
- − Memory and communication overhead
- − Load-balancing complexity
Applications
- · Frontier LLMs
- · Cost-efficient serving at scale