无需训练，用参数范数反比估计模型融合权重
NAN: A Training-Free Solution to Coefficient Estimation in Model Merging
- 基于最小二乘优化，用参数范数倒数估计融合系数
- 在多个数据集上显著提升基线方法性能
- 通用性强，可无缝集成到各类模型融合策略中
模型融合通过将独立微调的模型合并为统一模型，提供了一种无需训练的多任务学习替代方案，且无需访问原始数据。然而，现有方法常依赖启发式规则确定融合系数，限制了其可扩展性和通用性。本文从最小二乘优化视角重新审视模型融合，发现最优融合权重应与各模型编码的任务特异性信息量成正比。基于此，提出NAN方法，通过参数范数的倒数来估计融合系数。该方法无需训练、即插即用，适用于多种融合策略。大量实验表明，NAN能持续提升基线方法的性能。
原文摘要 · Abstract (English)
Model merging offers a training-free alternative to multi-task learning by combining independently fine-tuned models into a unified one without access to raw data. However, existing approaches often rely on heuristics to determine the merging coefficients, limiting their scalability and generality. In this work, we revisit model merging through the lens of least-squares optimization and show that the optimal merging weights should scale with the amount of task-specific information encoded in each model. Based on this insight, we propose NAN, a simple yet effective method that estimates model merging coefficients via the inverse of parameter norm. NAN is training-free, plug-and-play, and applicable to a wide range of merging strategies. Extensive experiments on show that NAN consistently improves performance of baseline methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。