无需数据或重训,通过参数差异估算协方差,实现高效模型合并。
ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation
- 利用微调模型的参数差异隐式估计任务输入协方差。
- 在GPT-2上7个任务平均提升4%,超越现有数据无关方法。
- 闭式解法计算高效,适合实际部署,理论基础扎实。
模型合并旨在将多个任务专用专家模型融合为单一模型,同时保持跨任务的泛化能力。然而,当专家模型在不同目标下训练时,相互干扰常导致性能显著下降。尽管已有进展,但在无数据访问、无需重训或修改架构的前提下解决此干扰仍是根本挑战。本文提供理论分析,表明每个任务的输入协方差——决定最优合并的关键因素——可从其微调模型的参数差异中隐式估计,即使在完全无数据设置下亦可实现。基于此洞察,我们提出 exttt{ACE-Merging}(Adaptive Covariance Estimation),一种有效缓解跨任务干扰的方法。该方法具备原理清晰的闭式解法,区别于以往迭代或启发式策略。在视觉与语言基准上的大量实验表明, exttt{ACE-Merging} 在数据无关方法中达到新标杆。例如,在GPT-2的七个任务上,其平均绝对提升达4%。得益于高效的闭式公式,该方法以较小计算开销实现卓越性能,为模型合并提供了实用且理论严谨的解决方案。
原文摘要 · Abstract (English)
Model merging aims to combine multiple task-specific expert models into a single model while preserving generalization across diverse tasks. However, interference among experts, especially when they are trained on different objectives, often leads to significant performance degradation. Despite recent progress, resolving this interference without data access, retraining, or architectural modification remains a fundamental challenge. This paper provides a theoretical analysis demonstrating that the input covariance of each task, which is a key factor for optimal merging, can be implicitly estimated from the parameter differences of its fine-tuned model, even in a fully data-free setting. Building on this insight, we introduce \acem, an Adaptive Covariance Estimation framework that effectively mitigates inter-task interference. Our approach features a principled, closed-form solution that contrasts with prior iterative or heuristic methods. Extensive experiments on both vision and language benchmarks demonstrate that \acem sets a new state-of-the-art among data-free methods. It consistently outperforms existing baselines; for example, \acem achieves an average absolute improvement of 4\% over the previous methods across seven tasks on GPT-2. Owing to its efficient closed-form formulation, \acem delivers superior performance with a modest computational cost, providing a practical and theoretically grounded solution for model merging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。