A layer-by-layer merging method aims to shorten AI reasoning without retraining

Reasoning models often generate lengthy intermediate steps before answering. Shortening those responses without losing useful problem-solving ability is the goal of Low-Rank Hierarchical Merging, or LHM. In a preprint posted on November 2, 2026, Zeqiu Yu, Xuesheng Zhang, Wenxiao Zhao and Shixiao Wang describe a method that changes model weights without additional training.

Their experiments report a promising accuracy–length tradeoff on mathematical benchmarks. However, the most dramatic compression result is not the setting that delivers their highest accuracy, and inconsistencies in the manuscript warrant caution when interpreting its headline comparisons.

Keeping different amounts of each layer

LHM starts with a base model and a corresponding reasoning model. Subtracting their weights produces a “task vector”: the parameter changes separating the two. A merged model adds a scaled portion of those changes back to the base model.

A single scaling factor treats all layers alike. LHM instead assigns different factors to different layers. The intuition is that some parameter changes may be more important to preserve than others, allowing selective merging to retain reasoning performance while reducing verbosity.

To guide that selection, the method estimates information from the Hessian, a mathematical description of how loss curves as parameters change. Rather than constructing this enormous matrix, it uses random probes to estimate its trace, a compact summary of curvature. The authors use Hutchinson’s estimator with randomly generated token sequences, avoiding task-specific examples for this estimation stage.

The resulting Layer Second-Order Sensitivity score multiplies each layer’s estimated Hessian trace by the squared magnitude of its task-vector component. This combines two considerations: how sensitive the loss appears to be, and how much the reasoning model changed that layer.

A final mapping gives higher-scoring layers larger merge coefficients. A global target coefficient controls the intended accuracy–conciseness tradeoff. The coefficients are incorporated into the weights offline, rather than adding a separate mechanism during generation.

“Training-free” does not mean computation-free: estimating curvature requires backward passes. Nor is the complete experimental procedure entirely task-data-free. Although curvature estimation uses random inputs, the authors choose the global target coefficient using a 64-example GSM8K validation split.

What the experiments report

The study evaluates Qwen2.5-Math base models paired with DeepSeek-R1-Distill-Qwen reasoning models at 1.5-billion and 7-billion parameters. Its six mathematical benchmarks are GSM8K, MATH500, Minerva Math, OlympiadBench, CollegeMath and AIME24. Evaluations use zero-shot prompting and greedy decoding.

At the smaller scale, Table 1 reports 48.8% average accuracy for LHM-AIM, compared with 45.6% for the ORCA-AIM merging baseline and 47.3% for the unmerged reasoning model. Average output length falls from the reasoning model’s 5,276 tokens to 2,126, approximately a 59.7% reduction.

Those averages do not imply improvement on every benchmark. On AIME24, for example, LHM-AIM scores 26.7%, below the unmerged reasoning model’s 30.0%.

At 7 billion parameters, the paper reports 76.9% average accuracy and 1,247 output tokens for LHM-AIM, versus 75.1% and 1,319 tokens for ORCA-AIM. These are results from the evaluated model pairs, not evidence of equivalent gains across other architectures or tasks.

The authors also report output-length reductions reaching 93.1% at a more aggressive setting. Crucially, they say accuracy then approaches the base model’s level. That compression figure should not be combined with the accuracy of the balanced operating point.

Assumptions and unresolved details

The sensitivity score is an approximation. Its derivation assumes the base model is near a stationary point of the loss and that parameter changes are not strongly aligned with dominant curvature directions. The paper’s checklist acknowledges that these assumptions may fail. Results also lack confidence intervals or a characterization of variability from random curvature probes.

There are internal reporting problems. The prose claims a 4.2-percentage-point gain over ACM-TIES, but the displayed averages, 48.8% and 43.6%, differ by 5.2 points. The coefficient mapping also does not exactly preserve its stated target mean; the appendix explicitly derives a residual offset.

LHM therefore offers an interesting reported approach to selective weight merging, rather than a demonstrated universal guarantee of shorter, equally capable reasoning. Its strongest supported takeaway is the measured tradeoff within the tested mathematical settings, with the manuscript’s inconsistencies kept visible.