通过自适应子空间选择,将多个微调模型合并为一个高效模型,接近独立模型精度。
MASS: MoErging through Adaptive Subspace Selection
- 基于任务更新的低秩分解,只保留关键子空间成分进行合并。
- 在8~20个任务上恢复98%的独立微调模型平均精度。
- 无需训练,推理时仅多一次前向传播,存储成本仅为单模型2倍。
模型合并近期成为轻量级集成替代方案,将多个微调模型合并为单一参数集,无需额外训练开销。然而,现有方法难以达到独立微调模型的完整准确率。本文提出MASS(MoErging through Adaptive Subspace Selection),通过统一多个微调模型,实现跨任务近乎顶尖的性能表现。基于每个任务更新的低秩分解,MASS仅存储各任务最显著的奇异分量,并将其合并至共享模型中。推理时,一个无参数、数据无关的路由器根据输入中间特征,识别最佳解释其特性的子空间(或组合),并激活相应任务特定模块。该过程完全无需训练,仅增加两次前向传播开销及约2倍存储成本,且与任务数量无关。我们在基于CLIP的图像分类任务上,使用ViT-B-16、ViT-B-32和ViT-L-14分别评估了8、14和20个任务的基准,建立了新基准。最显著的是,MASS可恢复高达约98%的独立微调模型平均准确率,是存储成本极低的集成实用替代方案。
原文摘要 · Abstract (English)
Model merging has recently emerged as a lightweight alternative to ensembling, combining multiple fine-tuned models into a single set of parameters with no additional training overhead. Yet, existing merging methods fall short of matching the full accuracy of separately fine-tuned endpoints. We present MASS (MoErging through Adaptive Subspace Selection), a new approach that closes this gap by unifying multiple fine-tuned models while retaining near state-of-the-art performance across tasks. Building on the low-rank decomposition of per-task updates, MASS stores only the most salient singular components for each task and merges them into a shared model. At inference time, a non-parametric, data-free router identifies which subspace (or combination thereof) best explains an input's intermediate features and activates the corresponding task-specific block. This procedure is fully training-free and introduces only a two-pass inference overhead plus a ~2 storage factor compared to a single pretrained model, irrespective of the number of tasks. We evaluate MASS on CLIP-based image classification using ViT-B-16, ViT-B-32 and ViT-L-14 for benchmarks of 8, 14 and 20 tasks respectively, establishing a new state-of-the-art. Most notably, MASS recovers up to ~98% of the average accuracy of individual fine-tuned models, making it a practical alternative to ensembling at a fraction of the storage cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。