利用模块线性提升大模型任务算术性能
Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs
- 按子模块分别线性合并，而非整体合并
- 在多任务下优于标准方法，且跨模型规模稳定有效
- 适合需要高效多任务融合的场景
任务算术是一种简单但高效的模型合并策略，可使合并后模型具备多任务能力。近期研究发现，具有线性的模型能提升任务算术表现。与依赖全局线性化的现有方法不同，本文指出线性特性已存在于模型的子模块中（如层、自注意力、MLP）。通过统计分析，我们发现子模块的线性程度显著高于整体模型。基于此，提出一种创新的模型合并策略：独立合并各子模块，并推导出基于子模块线性性质的最优合并权重闭式解。实验表明，该方法在不同模型规模和多种任务上均持续优于标准任务算术及其他基线方法。结果凸显了利用子模块线性的优势，为高效实用的多任务模型合并提供了新视角。
原文摘要 · Abstract (English)
Task arithmetic is a straightforward yet highly effective strategy for model merging, enabling the resultant model to exhibit multi-task capabilities. Recent research indicates that models demonstrating linearity enhance the performance of task arithmetic. In contrast to existing methods that rely on the global linearization of the model, we argue that this linearity already exists within the model's submodules. In particular, we present a statistical analysis and show that submodules (e.g., layers, self-attentions, and MLPs) exhibit significantly higher linearity than the overall model. Based on these findings, we propose an innovative model merging strategy that independently merges these submodules. Especially, we derive a closed-form solution for optimal merging weights grounded in the linear properties of these submodules. Experimental results demonstrate that our method consistently outperforms the standard task arithmetic approach and other established baselines across different model scales and various tasks. This result highlights the benefits of leveraging the linearity of submodules and provides a new perspective for exploring solutions for effective and practical multi-task model merging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。