通过重要性度量实现精准任务参数融合,提升多任务模型性能与稳定性。
Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics
- 基于一阶泰勒展开计算参数重要性,实现任务特异性参数筛选。
- 在多个基准上超越传统方法,任务遗忘控制更精确,性能提升显著。
- 无需大量调参,适合资源受限场景下的高效多任务部署。
预训练模型通过大规模先验知识表示,显著提升了各类任务的性能。然而,在真实多任务学习(MTL)场景中,高计算成本和推理效率低下仍是主要挑战。传统方法如剪枝、量化和知识蒸馏难以充分应对多任务环境的复杂性。本文提出一种无训练框架——选择性任务算术(STA),通过任务特定参数融合提升多任务性能。针对三大问题:(i) 参数重要性差异——采用基于一阶泰勒展开的损失敏感重要性度量,精准评估各任务参数重要性;(ii) 超参数依赖过重——通过重要性度量增强任务向量稀疏性,减少调参需求,提升泛化能力;(iii) 忽视任务遗忘控制——利用重要性度量实现更精细的任务遗忘,抑制噪声干扰。实验表明,STA在多个基准上表现优异,兼具卓越的多任务性能与可控的任务遗忘能力。
原文摘要 · Abstract (English)
Pretrained models have revolutionized deep learning by enabling significant performance improvements across a wide range of tasks, leveraging large-scale, pre-learned knowledge representations. However, deploying these models in real-world multi-task learning (MTL) scenarios poses substantial challenges, primarily due to high computational costs and inefficiencies in inference. Traditional approaches such as pruning, quantization, and knowledge distillation have been explored to mitigate these issues, but they often fall short in fully addressing the complexities of multi-task environments. This paper introduces \textbf{\underline{S}}elective \textbf{\underline{T}}ask \textbf{\underline{A}}rithmetic \underline{\textbf{(STA)}}, a training-free framework designed to enhance multi-task performance through task-specific parameter fusion. STA addresses three key challenges: (i) \textbf{Parameter importance diversity: } Recognizing that different tasks relie on distinct parameters, STA employs a loss-sensitive parameter importance metric derived from a first-order Taylor expansion to accurately measure the importance of parameters for each task. (ii) \textbf{Over-reliance on hyperparameter tuning: }By enhancing the sparsity of task vectors through parameter importance metrics, STA reduces the need for extensive hyperparameter tuning, thereby improving the generalization and robustness of the model. (iii) \textbf{Neglect of other abilities in task arithmetic: } Previous works have largely overlooked the potential for more precise task forgetting. STA leverages its parameter importance metric to achieve more controlled and effective task forgetting, minimizing the impact of noisy elements that can degrade model performance. Experimental results demonstrate that STA achieves superior multi-task performance across benchmarks and excellent performance in task forgetting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。