提出核心空间合并方法，在保持低秩高效的同时提升模型融合精度。
Accurate and Efficient Low-Rank Model Merging in Core Space
- 在共同对齐基下直接合并低秩适配参数，避免全尺寸矩阵运算。
- 跨视觉与语言任务均达当前最优性能，计算开销仅为现有方法的极小部分。
- 理论证明合并不损失信息，且复杂度分析显示显著效率优势，适合高效部署场景。
本文针对大规模神经网络低秩适配模型的合并挑战提出核心空间合并框架（Core Space merging）。随着低秩适配技术（如LoRA）的兴起，模型微调变得更为高效，但现有合并方法常因合并全尺寸权重矩阵而牺牲效率。所提方法在共同对齐基中实现LoRA适配模型的合并，既保留了低秩适配的高效性，又显著提升多任务准确性。我们提供了投影至核心空间无信息损失的严格证明，并进行了复杂度分析，证实其计算优势。大量实验证明，该方法在视觉与语言任务上均达到当前最优表现，同时仅需极少计算资源。代码已开源：https://github.com/apanariello4/core-space-merging。
原文摘要 · Abstract (English)
In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。