通过任务特定干预减少表示偏差，高效合并多任务模型
Parameter-Efficient Interventions for Enhanced Model Merging
- 用任务特异性干预缓解模型表示偏差
- 引入微型干预仅修改部分表征，参数更少性能不降
- 在更少参数下超越现有最优方法，适合资源受限场景
模型合并将特定任务模型的知识整合为统一的多任务模型，避免对所有任务数据进行联合训练。然而，当前方法因存在表示偏差而面临性能干扰问题。为此，我们提出IntervMerge，一种新型多任务模型合并方法，通过任务特定干预有效缓解模型中的表示偏差。为进一步提升效率，我们引入迷你干预机制，仅修改部分表征，从而在不损失性能的前提下显著减少额外参数。实验结果表明，IntervMerge在参数更少的情况下持续优于当前最优方法。
原文摘要 · Abstract (English)
Model merging combines knowledge from task-specific models into a unified multi-task model to avoid joint training on all task data. However, current methods face challenges due to representation bias, which can interfere with tasks performance. As a remedy, we propose IntervMerge, a novel approach to multi-task model merging that effectively mitigates representation bias across the model using taskspecific interventions. To further enhance its efficiency, we introduce mini-interventions, which modify only part of the representation, thereby reducing the additional parameters without compromising performance. Experimental results demonstrate that IntervMerge consistently outperforms the state-of-the-art approaches using fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。