无需数据，用信息量自适应合并大模型，提升推理精度并大幅缩短输出。
Data-Free Layer-Adaptive Merging via Fisher Information for Long-to-Short Reasoning LLMs
- 基于费雪信息矩阵计算每层合并权重，实现无数据自适应融合。
- 7B模型上比现有方法高6.2分，1.5B模型准确率提升3.9点，输出长度减少91.9%。
- 理论证明可解释为何分层合并优于统一合并，适合高效推理系统构建者。
模型合并已成为无需额外训练即可整合专用大语言模型能力的实用方法。在长到短（L2S）场景中，将基础模型与长链思维推理模型合并，旨在保留推理准确性的同时减少输出长度。现有方法依赖任务算术及其变体，隐含假设模型输出随合并系数呈线性变化——我们证明该假设在L2S设置中系统性失效。本文首次提供分层自适应合并的理论依据：证明合并误差受每层海森矩阵范数约束（命题1），并建立费雪信息矩阵（FIM）作为该边界的合理可计算代理，基于局部最优处的费雪-海森等价性。基于此，提出FIM-Merging：仅用随机令牌输入计算对角FIM，并据此分配每层合并系数。在7B L2S基准上，FIM-TIES在六项评测中的五项达领先水平，于MATH500上较ACM-TIES提升6.2分（90.2 vs. 84.0），且无需校准数据。在1.5B基准上，平均准确率达47.3，超过此前最优的ACM-TIES（43.3），提升3.9分，同时相比长CoT模型平均响应长度减少91.9%。本框架也为现有分层自适应方法（如ACM）为何在实践中优于均匀合并提供了统一理论解释。
原文摘要 · Abstract (English)
Model merging has emerged as a practical approach to combine capabilities of specialized large language models (LLMs) without additional training. In the Long-to-Short (L2S) scenario, merging a base model with a long-chain-of-thought reasoning model aims to preserve reasoning accuracy while reducing output length. Existing methods rely on Task Arithmetic and its variants, which implicitly assume that model outputs vary linearly with the merging coefficient -- an assumption we show is systematically violated in L2S settings. We provide the first theoretical justification for layer-adaptive merging: we prove that merging error is bounded by a term proportional to the per-layer Hessian norm (Proposition~1), and establish that the Fisher Information Matrix (FIM) is a principled, computable proxy for this bound via the Fisher-Hessian equivalence at local optima. Building on this theory, we propose \textbf{FIM-Merging}, which computes diagonal FIM using only random token inputs (no domain-specific calibration data required) and uses it to assign per-layer merging coefficients. On the 7B L2S benchmark, FIM-TIES achieves state-of-the-art performance on five out of six evaluation benchmarks, including a \textbf{+6.2} point gain on MATH500 over ACM-TIES (90.2 vs.\ 84.0), while requiring no calibration data. On the 1.5B benchmark, FIM-TIES achieves an average accuracy of \textbf{47.3}, surpassing the previous best ACM-TIES (43.3) by \textbf{+3.9} points, while reducing average response length by \textbf{91.9\%} relative to the long-CoT model. Our framework also provides a unified theoretical explanation for why existing layer-adaptive methods such as ACM empirically outperform uniform merging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。