AI Experts Can Use Different LoRA Ranks—But the Merge Method Matters

A shared setting that may not be necessary

Model merging combines separately fine-tuned specialists into one model without retraining it. In this study, those specialists use LoRA adapters: low-rank updates to a shared base model. An adapter’s rank sets its capacity, and the authors ask whether every specialist really needs the same setting.

Their preprint, Model Merging Does Not Require Uniform LoRA Ranks, argues that the answer depends on how the updates are combined. Some tested methods accommodated different ranks with little change in average performance. Others suffered substantial losses, particularly on instruction following. This is evidence from a limited experimental setting, not a guarantee for arbitrary models or adapters.

What the researchers tested

The team trained 36 experts using Qwen3.5 models with 0.8 billion, 2 billion and 4 billion parameters. At each size, they trained mathematics, instruction-following and multilingual adapters at ranks 4, 8, 16 and 32, using QLoRA with a four-bit quantized base.

Evaluation used GSM8K for mathematics and IFEval for instruction following. The multilingual score averaged MMLU, ARC and MMLU-zh results. The reported multi-domain average gave equal weight to the three domain scores.

The best individual rank varied by domain and model size. At 4B, for example, the selected ranks were 4 for mathematics, 32 for instruction following and 8 for multilingual performance. However, the authors report that single-domain differences across ranks were typically no more than about two percentage points. Their larger concern was what happened during merging.

Why keeping the biggest updates can lose useful information

The paper examines a compression budget that retains as many spectral components as the largest input rank. Spectral decomposition expresses updates as component directions with associated singular values, which measure their size.

With three equally ranked experts, the pooled updates contain approximately three times that rank in components, but the budget keeps only about one-third. Increasing every expert’s rank therefore does not increase the fraction retained. Unequal ranks can loosen this bottleneck: one measured configuration retained about 73% of its pooled components.

More room does not necessarily mean better choices. The authors’ analysis suggests that selection by size discards components predicted to reduce task loss. In one magnitude-truncated configuration, mathematics accounted for about 93% of retained energy, while instruction following accounted for about 1%. That measures representation in the retained update, not a direct percentage of capability preserved.

Selecting by predicted usefulness

The researchers compare six methods, including simple averaging, greedy soup, TIES, KnOTS, plain singular-value decomposition truncation, and gradient-guided selection, or GGSM.

GGSM keeps the same component budget but selects directions by their predicted effect on loss rather than size. It computes gradients at an initial merged model using 64 validation examples per domain, normalizes those gradients, and combines them equally. Unlike the data-free baselines, it therefore requires validation data and backward computation.

Its usefulness estimates are approximate. Testing 50 sampled components in a 0.8B model produced a moderate correlation with measured loss changes. The authors also report overestimation for some components, especially those with extreme predicted scores.

The results—and an important exception

Comparing each method’s best tested unequal-rank configuration with its best uniform-rank configuration, GGSM, plain SVD truncation and simple averaging changed by between −0.3 and +1.8 percentage points across model sizes. TIES and KnOTS were inconsistent. At 4B, TIES fell from 75.1 to 71.1 on the multi-domain average.

Plain SVD is an important exception to a simple “magnitude selection is bad” interpretation: it selects by magnitude yet tolerated unequal ranks here. The authors argue that additional conflict-resolution operations in TIES and KnOTS interact poorly with heterogeneous ranks. Greedy soup also declined at the two smaller scales, so avoiding component selection alone did not guarantee stability.

Instruction following exposed particularly large differences. At uniform rank 4 in the 4B model, IFEval scores were 66.0 for TIES and 56.6 for KnOTS, against 81.7 for the best standalone instruction expert.

What the evidence supports

Across the 4B uniform-rank sweep, the authors report a 0.7-point range for GGSM versus 6.7 points for TIES. Stability is a stronger finding than a clear peak-accuracy advantage for GGSM.

The experiments used a single merge seed, three domains, LoRA fine-tuning and one model family. The authors estimate a roughly 1.4-point resolution floor for paired comparisons and call for more seeds and larger evaluations to establish small benefits. Their practical conclusion is conditional: independent rank choices worked with several tested operators, but were not reliably safe with every merging method.