Selective Architectures Move The Hard Problem Into Routing

Mixture-of-experts wakes a few experts per token. Retrieval moves knowledge out of the weights and into an index. Both buy capacity, and both make selection the new failure point.

Total parameters stop describing the bill. A router decides which parameters wake for this token, and that decision sets the cost.

Conditional Compute Asks Which Parts Must Wake

Dense scaling spends every parameter on every token. Every layer runs and every weight multiplies, whatever the token contains. A selective system asks which parts must wake for this input.

That question buys capacity and specialization. The model holds more parameters than any one token pays for, and different inputs wake different parts. The cost lands on whatever makes the choice.

Reach for a selective architecture when the bottleneck is capacity or knowledge access. A vague product spec does not improve because a router picks the expert. You keep the problem and add a router.

A Router Buys Capacity And Sells You A Routing Problem

A sparse mixture-of-experts model puts a router in front of a set of expert networks. Each token goes to a selected subset, and only that subset runs. Total parameters rise while active compute per token tracks the subset that ran.

Switch Transformer sends each token to one expert. Routing gets simpler and the scaling argument gets simpler with it. One expert per token leaves nothing to dilute a wrong choice.

Load-balancing losses exist because routers collapse. Without pressure to spread load, a router learns to send every token to the same expert. Real traffic pushes the same way, because one domain can dominate a batch. The rest of the capacity sits idle while you pay to hold it in memory.

Training also gets less stable, because a small change in router scores moves a token to a different expert. Specialization is a training outcome, and the balancing term pushes against it. Balance and specialization pull in opposite directions, so you measure both.

Retrieval Moves Knowledge Out Of The Weights

Retrieval attacks a different bottleneck. RAG conditions generation on passages pulled from an index, and RETRO retrieves chunks during generation itself. Both lower the pressure to memorize everything in parameters.

A secret trained into the weights resists deletion, as the security note showed. A row in an index leaves when you delete the row. The legality note ended at a takedown, which is cheaper to serve against an index than against a checkpoint.

Attribution turns from a promise into a pointer to the passage that produced the sentence. Weights are an expensive place to keep a fact that changes, and the index updates without a training run. The retriever now sits on the critical path, and a wrong passage becomes a confident wrong answer.

A retriever can also return nothing useful and leave the model to answer from the weights anyway. That failure reads as a normal answer, and the log is the only place it shows.

Selection Must Be Inspectable

The overview note set one rule. When a model surprises you, name the layer that did the work. A selective system adds a layer that chooses, so the router and the retriever join the weights as suspects.

Log the identity of the chosen path rather than a score for it. Which expert handled this token? Which passage did the retriever return? A system that cannot answer those two questions has added complexity nobody can debug.

Treat the path log as a shipped feature. Without it, a regression in one domain arrives as a support ticket instead of a metric.

Sparse Compute Saves Work Only When Routing Cooperates

Routing overhead has a unit, and the parallelism note named it. Communication competes with compute, and a cluster that waits is not learning. A router adds communication, because tokens must travel to whichever device holds their expert.

Routing overhead, expert imbalance, and brittle specialization each spend part of the saving before it arrives. Memory holds every expert whether it fires or not. Batching gets harder when the tokens in one batch route to different experts.

Fewer active parameters describes one line of the bill. Wall-clock time for a batch covers the rest. Routing adds to that number before conditional compute takes anything off it. A saving you cannot see in the clock is arithmetic about parameters.

The Builder Test

Take one batch of real traffic and log the selected path for every token. Keep the expert identity and the retrieved passage ids next to the output. Then read three numbers off that log.

• Expert load across the batch, which shows whether the router collapsed onto a few experts.
• Quality by domain against a dense baseline, because an average gain hides the slice that got worse.
• Active compute and the communication time the router added, counted together.

Keep the dense baseline alive long enough to compare. A selective system without a baseline reports only its own numbers.

If expert load is lopsided, fix the balancing term before you buy more experts. If one domain lost quality, send that domain to the dense path until the router earns it back.

What Carries

When a model chooses what to use, the choice is part of the model. I treat a router and a retriever as components with owners and dashboards, the same as a service. Without that, the capacity you bought stays theoretical.

Routing decides what wakes. Adaptation decides what changes.