提出DTS框架,在仅增1%存储下保留多任务模型个性。
Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging
- 通过奇异值分解与分组缩放,高效压缩任务专属参数。
- 在多个任务上性能超越现有方法,且仅需1%额外存储。
- 无需数据即可融合新任务,适合追求效率的部署场景。
模型合并已成为无需额外训练即可实现多任务能力的有前景范式。然而,传统基础合并方法常因参数冲突导致性能下降,即使面对相似任务亦然。尽管近期个性化合并框架能有效保留任务特异性信息以维持性能,但通常带来存储开销。本文提出基于近似的个性化合并框架DTS(分解、阈值化、缩放),极大提升任务特异性存储效率。DTS首先对任务特异性信息进行奇异值分解,仅保留少量奇异值与向量;随后引入新颖的阈值策略,将奇异向量元素分组并为每组分配缩放因子。为进一步支持未见任务的泛化,我们扩展DTS为一种无需数据的融合方式,基于任务特征的语义相似性进行合并。大量实验表明,DTS始终优于现有最优基线,且每任务仅增加1%存储开销。在未见任务上的实验也显示该变体具有显著更优的泛化性能。代码已开源:https://github.com/krumpguo/DTS。
原文摘要 · Abstract (English)
Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training. However, traditional basic merging methods often experience performance degradation due to parameter conflicts, even when applied to similar tasks. While recent personalized merging frameworks successfully preserve task-specific information to maintain performance, they typically incur storage overhead. In this paper, we propose Decomposition, Thresholding, and Scaling (DTS), an approximation-based personalized merging framework that pushes task-specific storage efficiency. DTS first applies singular value decomposition to the task-specific information and retains only a small subset of singular values and vectors. It then introduces a novel thresholding strategy that partitions singular vector elements into groups and assigns a scaling factor to each group. To enable generalization to unseen tasks, we further extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics. Extensive experiments demonstrate that DTS consistently outperforms state-of-the-art baselines while requiring only 1\% extra storage per task. Furthermore, experiments on unseen tasks show that the DTS variant achieves significantly better generalization performance. Our code is available at https://github.com/krumpguo/DTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。