Switching Off AI Experts Can Break More Than Routing

Mixture-of-experts language models divide part of their computation among specialist neural networks called experts. For each token, a router selects a small subset and assigns weights to their outputs. This limits computation, but the full expert bank still creates storage, loading and scheduling costs. Disabling experts changes which computations remain available—and potentially how strongly their outputs influence the model.

In an arXiv preprint, Hao Cheng, Zhiyu Yang, Bo Ma and Hanzhang Jia investigate that second effect. Their central finding is that identical disabled sets can produce sharply different outcomes depending on routing and weighting choices. The results support using the model’s training-time weighting rule as a reference, rather than treating expert deactivation as a single operation.

Four ways to disable the same experts

The researchers compare four implementations. Removal masks disabled experts before calculating router probabilities, selects replacements and computes their weights using the model’s original rule. Freezing preserves the original routing decisions and surviving weights, but deletes the disabled experts’ contributions. Despite its name, freezing here does not mean leaving those experts active with unchanged parameters.

Remaining-expert renormalization keeps the same survivors as freezing, without filling empty slots, but rescales their weights to sum to one. Unnormalized re-routing fills the slots with replacements and uses their original router probabilities; disabled experts remain included in the probability calculation’s denominator.

The distinction matters because the studied models were not trained with one universal weighting convention. OLMoE and DeepSeek-V2 use raw router probabilities, whose sum across selected experts is below one. Granite and Qwen3 normalize selected weights to sum to one.

The authors develop a weight-sum model describing these differences. For raw-probability models, forcing surviving weights to sum to one introduces amplification relative to the residual pathway—the part of the layer output retained alongside the expert contribution. For normalized-weight models, switching to raw probabilities attenuates that contribution. These are changes in how expert outputs are combined, not merely changes in which experts participate.

A controlled comparison with large differences

The experiments cover five instruction-tuned checkpoints from four model families, plus a base OLMoE checkpoint. Model parameters remain fixed. Experts are ranked using a C4 calibration corpus, with the lowest-scoring experts disabled separately in each layer. Additional experiments use selection-frequency rankings and random sets. Within each compared model, proportion and selection criterion, all four implementations receive the same disabled set.

Evaluation includes WikiText-2 perplexity, multiple-choice benchmarks and GSM8K mathematical word problems. Perplexity measures how well a model predicts the evaluation text; lower values indicate better prediction.

With half the experts disabled using activation-based rankings, the worst perplexity follows the predicted mismatch in every instruction-tuned checkpoint: remaining-expert renormalization for OLMoE and DeepSeek-V2-Lite, and unnormalized re-routing for Granite and Qwen3.

DeepSeek-V2-Lite offers a particularly clear comparison. Freezing produces perplexity of 17.21, while renormalizing the same surviving experts produces 258,621.64. Neither method fills the vacated slots. Their difference therefore isolates the effect of rescaling the surviving contributions under this experimental configuration.

Preserving the training rule is not a guarantee of mild damage. Across the central half-bank comparison, implementations retaining that rule still produce perplexities ranging from roughly 1.4 to 73 times the intact baseline. Which experts are disabled also substantially affects the outcome.

Better perplexity does not guarantee useful answers

Task results complicate any simple ranking. On instruction-tuned OLMoE, GSM8K strict final-answer accuracy falls from 36.2% to zero under all four implementations after half the experts are disabled. Some two-option benchmark scores remain above their uniform-guessing reference, while MMLU approaches chance. Different metrics therefore reveal different aspects of the remaining performance.

A separate OLMoE experiment changes how many experts are activated per token. Additional renormalization causes a larger perplexity penalty at smaller tested expert counts, consistent with the authors’ prediction that normalizing a smaller selected probability sum introduces greater amplification.

The scope remains limited: six checkpoints, four families, a fixed calibration corpus and GSM8K as the free-form generation evaluation. The study does not establish whether recovery training would repair the damage, and it does not demonstrate deployment speedups. Its practical message is narrower: expert selection and weight handling must be evaluated together, against the actual target tasks. Matching the training-time rule can avoid a severe additional failure mode without preserving the intact model’s capabilities.