发现模型合并存在稳定的一维子空间，可无训练外推提升性能。
Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
- 通过分析预训练轨迹，发现合并模型收敛到一维线性流形。
- 提出Extra-Merge方法，在不更新梯度下实现损失最小化。
- 在多个模型和任务上表现优于传统合并方法，适合高效微调场景。
模型合并已成为增强大语言模型的轻量级范式，但其内在机制仍不清晰。本文分析了预训练后期轨迹，发现一个 extbf{秩一子空间}现象：尽管原始优化步骤剧烈振荡，连续的 extit{合并}检查点却坍缩至一个稳定、近似一维的线性流形。我们从 extit{河谷景观}分析出发，理论证明平均操作如同几何低通滤波器，抑制高曲率噪声，揭示最优下降方向。基于此洞察，提出 extbf{Extra-Merge}，一种无需训练的策略，沿该子空间外推以最小化损失。在GPT-2与LLaMA系列（124M至2B）上的大量实验表明，Extra-Merge持续优于标准合并基线。尤其在Pythia-12B下游任务中实现一致的零样本准确率提升，并有效泛化至Muon优化器 extcite{jordan2024muon}。
原文摘要 · Abstract (English)
Model merging has emerged as a lightweight paradigm for enhancing Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. In this work, we analyze late-stage pre-training trajectories and uncover a \textbf{Rank-1 Subspace} phenomenon: while raw optimization steps oscillate violently, consecutive \emph{merged} checkpoints collapse onto a stable, approximately one-dimensional linear manifold. We theoretically ground this observation in a \emph{river-valley} landscape analysis: averaging acts as a geometric low-pass filter that dampens high-curvature noise to reveal the optimal descent direction. Capitalizing on this insight, we propose \textbf{Extra-Merge}, a training-free strategy that extrapolates along this subspace to minimize loss without additional gradient updates. Extensive experiments across GPT-2 and LLaMA families (124M to 2B) demonstrate that Extra-Merge consistently outperforms standard merging baselines. Notably, it yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon optimizer \citep{jordan2024muon}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。