Turning a mixture of experts into a dense language model

The conversion idea

A mixture-of-experts model contains multiple expert networks, with a router determining which experts contribute to processing an input. The paper focuses on a different compression goal from simply reducing the number of experts while retaining an MoE architecture: converting the trained model into a standard dense one.

The authors describe a pipeline that scores experts, selects and groups them, and concatenates them into a dense feed-forward network. They then refine that network using knowledge distillation: training a model, called the student, to match outputs from a teacher. In this study, the original MoE serves as the teacher.

Comparing design choices

The researchers report testing seven expert-scoring methods, five grouping methods and two magnitude-scaling methods across different selected-expert counts. This produced 350 configurations on Qwen3-30B-A3B. The abstract also reports evaluation on DeepSeek-V2-Lite and GPT-OSS-20B.

The authors identify expert scoring as the most influential design choice. Their new diversity-aware scoring method, they report, consistently outperformed prior methods on all three named models. The abstract does not give the sizes of those performance differences, so it is not possible to tell from this summary how large or practically important the advantage was in each case.

A reported head-to-head result

The abstract describes a controlled comparison at matched parameter count: the MoE-to-dense approach was compared with pruning a dense model directly. After about four billion training tokens of distillation, the converted model achieved an average downstream accuracy 6.3 percentage points higher, according to the authors. They also report 1.6 times faster training wall-clock speed for this comparison.

These figures refer to the reported comparison, not a general guarantee that MoE-to-dense conversion will outperform dense pruning in every setting. The abstract does not provide enough detail to independently assess the comparison or determine how its results vary across tasks and configurations. Training wall-clock speed is also distinct from inference speed and deployment memory use; the abstract’s reported comparison is about training speed and downstream accuracy.

What remains uncertain

The findings suggest that the way experts are chosen and combined can matter when transforming an MoE into a dense model. But the abstract alone cannot show how broadly the reported results apply, or what practical deployment trade-offs the method offers. In particular, it does not summarize memory or inference measurements. Those outcomes should not be inferred from the architectural conversion alone.

This account is based on the paper’s abstract, not a review of its full methods and results. The abstract is enough to describe the approach and the authors’ headline findings, but not to evaluate benchmark design, task-by-task outcomes, or the robustness of the comparisons in detail. Readers should treat the numerical results as reported by the authors and consult the full paper for experimental context. The available evidence does not establish the paper’s peer-review or publication status.