通过优化轨迹感知的融合方法,提升大模型能力合并效果。
Harnessing Optimization Dynamics for Curvature-Informed Model Merging
- 利用优化器二阶统计量作为曲率代理,动态重加权参数更新。
- 在多任务微调检查点上,合并模型性能优于主流基线,减少负迁移。
- 适合需要高效融合多个能力模块的大模型应用开发者。
模型合并是一种无需联合训练即可整合大语言模型能力的有效后训练策略。本文研究在监督微调(SFT)阶段,将涵盖数学、代码、精准指令遵循、通用指令遵循和知识回忆等不同能力的多个SFT检查点合并为单一模型的问题。提出优化轨迹感知(OTA)合并方法,利用优化器二阶矩统计量作为对角曲率代理,重新加权参数更新以缓解干扰。同时提出快速费雪嫁接(FFG),一种基于曲率的任务定位步骤,通过稀疏化冲突或低重要性更新来增强鲁棒性。FFG生成极低秩掩码,集中于早期注意力查询/键投影和词元嵌入,利用不同能力间的共享曲率特性。进一步开发了内存轻量化的二阶矩压缩方案,保留了完整版OTA的效果。在多种能力导向的SFT检查点上,OTA+FFG显著优于强基线权重空间方法,降低负迁移并保持在不同稀疏度下的稳定性。分析揭示检查点间存在显著曲率重叠,为线性合并在实践中有效提供了新视角。消融实验表明FFG对减少任务干扰至关重要,且压缩后的二阶矩仍能维持完整方法的优势。为促进复现,所有代码、训练与评估脚本、可视化结果及特定能力的SFT检查点均已开源:https://github.com/pmahdavi/ota-merge。
原文摘要 · Abstract (English)
Model merging is an effective post-training strategy for composing capabilities in large language models without joint retraining. We study this in the supervised fine-tuning (SFT) stage, where multiple capability-based SFT checkpoints -- spanning math, code, precise instruction following, general instruction following, and knowledge recall -- must be consolidated into a single model. We introduce Optimization Trajectory Aware (OTA) Merging, a curvature-aware aggregation that leverages optimizer second-moment statistics as a diagonal curvature proxy to reweight parameter edits and mitigate interference. Complementing OTA, we propose Fast Fisher Grafting (FFG), a curvature-driven task-localization step that sparsifies conflicting or low-importance edits. FFG induces extremely low-rank masks concentrated in early attention query/key projections and token embeddings, exploiting shared curvature across capabilities. We further develop a memory-light compression of the second moments that preserves OTA's effect. Across diverse capability-based SFT checkpoints, OTA+FFG improves merged-model quality over strong weight-space baselines, reduces negative transfer, and remains robust across sparsity levels. Analyses reveal substantial curvature overlap between checkpoints, offering a novel lens on why simple linear merging can be effective in practice. Ablations confirm that FFG is critical for reducing task interference and that the compressed second moments retain the gains of the full formulation. To facilitate reproducibility, we open-source all code, training and evaluation scripts, visualization artifacts, and capability-specific SFT checkpoints at https://github.com/pmahdavi/ota-merge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。