动态调整视觉语言模型前缀权重，提升持续学习效果
Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting
- 根据输入标记重要性动态分配前缀权重
- 适配器仅在必要时启用，减少冗余计算
- 在跨域分类增量学习中达当前最优性能
我们研究了近期提出的视觉语言模型（VLMs）的领域-类别增量学习场景。现有方法采用参数高效技术如前缀调优或适配器，通过添加向量将任务特定信息融入输入标记以实现模型适应。然而，先前方法常对这些向量权重进行归一化，忽略了不同输入标记所需调整程度的差异。为此，我们提出动态前缀加权（DPW）框架，通过门控模块依据输入标记重要性动态调整前缀权重，并设计加权机制使适配器输出权重作为前缀调优权重的残差，确保适配器仅在必要时使用。实验表明，该方法在视觉语言模型的领域-类别增量学习场景中达到当前最优性能。代码已公开：https://github.com/YonseiML/dpw。
原文摘要 · Abstract (English)
We investigate recently introduced domain-class incremental learning scenarios for vision-language models (VLMs). Recent works address this challenge using parameter-efficient methods, such as prefix-tuning or adapters, which facilitate model adaptation to downstream tasks by incorporating task-specific information into input tokens through additive vectors. However, previous approaches often normalize the weights of these vectors, disregarding the fact that different input tokens require different degrees of adjustment. To overcome this issue, we propose Dynamic Prefix Weighting (DPW), a framework that dynamically assigns weights to prefixes, complemented by adapters. DPW consists of 1) a gating module that adjusts the weights of each prefix based on the importance of the corresponding input token, and 2) a weighting mechanism that derives adapter output weights as a residual of prefix-tuning weights, ensuring that adapters are utilized only when necessary. Experimental results demonstrate that our method achieves state-of-the-art performance in domain-class incremental learning scenarios for VLMs. The code is available at: https://github.com/YonseiML/dpw.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。