Combining two AI specialists does not guarantee keeping both strengths

What counts as a successful combination?

Model merging combines changes to a neural network’s numerical weights into one model. The question is not simply whether its average score rises. A mathematics improvement could conceal a programming decline.

In this preprint, submitted to arXiv and listed as under review for ICLR 2027, Suxin Ji and colleagues require something stricter: the merged model must outperform the shared starting model on both tasks. They call this “strict joint success.” It does not require beating each selected specialist at its own task.

The researchers also measure worst-task gain: calculate the accuracy change on each task, take the smaller change, then average that quantity across evaluations. This exposes losses that an overall average can hide.

Why nearby changes can still clash

The specialists here are not separately trained from scratch. They are selected from random perturbations—changes to the weights—around a shared checkpoint. Different candidates can perform well on different tasks.

Sharing that origin makes their weight changes straightforward to combine, but says little about whether their benefits survive. The paper separates a combined change’s effect into the effects of the individual changes and an interaction term.

Its mathematical counterexample makes an important distinction. Two arbitrarily small, perpendicular weight changes can each improve their own task while harming the other task enough that their combination worsens both. This can happen even without an interaction term.

That is a theoretical demonstration using simplified scores, not an experiment showing how actual language-model accuracy behaves. It establishes why proximity and perpendicularity alone cannot guarantee success; it does not prove that the proposed selection method will work.

Selecting pairs before merging

The experiments use six instruction-tuned checkpoints from the Qwen2.5, Llama 3.1 and OLMo 3 families, ranging from 0.5 billion to 8 billion parameters. Seven benchmark tasks include mathematics, programming, story ordering and chemical reaction prediction, giving 21 task pairs. Each checkpoint has five independent perturbation pools, each containing 5,000 candidates.

Every candidate is scored on every task. For each task, the procedure retains up to 50 candidates that at least match the starting model on that task’s selection examples. Pair ranking then considers both candidates’ performance on both tasks, helping penalize cross-task damage.

The additional “spectral” signal compares dominant directions within the candidates’ weight-change matrices, layer by layer. At comparable measured quality, the ranking favors pairs with less overlap. This is a prioritization signal, not a direct measurement of task compatibility.

The procedure tests up to ten pairs with five weighting choices each: at most 50 proposed merges per task pair. Separate validation examples determine whether a proposal improves both tasks and which qualifying proposal to keep. If none qualifies, it returns the starting model—an abstention. Final results use held-out test data, separate from selection and validation.

The output is one fixed model, not an ensemble consulting multiple specialists for each answer. The procedure uses no gradient updates, although candidate scoring and merge validation still require substantial evaluation.

More joint gains, with important exceptions

The primary aggregate covers the five checkpoints with at least 1.5 billion parameters: 525 model–task-pair–pool evaluations per method. Keeping screening, validation and weighting choices fixed, strict joint success rose from 49.1% with quality-only pair ranking to 50.5% with a global geometric comparison and 53.1% with spectral ranking.

These percentages describe successful evaluation configurations, not accuracy on individual questions. The spectral procedure’s mean worst-task gain was +0.61 percentage points. Across all configurations, 17.1% regressed on at least one task and 13.0% returned the starting model. Abstentions remained in the denominator and contributed zero gain.

The full procedure beat simply averaging the strongest individual specialists on every checkpoint, but led the four-method comparison on only four of six. Selecting a single multitask candidate did better on Qwen 7B; the TIES merging method did better on OLMo 3. On the smallest Qwen checkpoint, mean worst-task gain remained slightly negative.

A useful filter, not a safety guarantee

Positive validation results did not prevent held-out regressions. Nor do the aggregate improvements establish statistical significance: evaluations share tasks and assessment items, and variation across five pools captures search randomness rather than all uncertainty.

The findings concern this particular candidate-generation, selection and two-expert merging protocol. They do not establish success rates throughout weight space or show that compatible pairs remain compatible when more experts are added. The practical lesson is narrower: evaluate both tasks, test actual combinations, and retain the option not to merge.