arXiv:2503.20612cs.CV2025-03被引 11

针对视觉语言模型持续学习中提示设计难题,提出实例感知提示框架。

IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting

  • 根据实例特征自适应分配提示参数,跨层动态调整
  • 在11个数据集上显著降低前后向遗忘,提升任务适应性
  • 适合资源受限场景下持续学习的视觉语言模型优化

近期预训练视觉语言模型(PT-VLMs)常面临多领域任务增量学习(MTIL)场景,即多个类别和领域的多模态任务逐步到达。由于无法访问过往与未见任务,内存受限的MTIL存在前向与后向遗忘问题。为此,参数高效微调技术(PEFT)如提示调优被用于适配PT-VLM应对多样增量任务。现有方法仅关注PEFT策略选择,忽略提示参数设置的影响。本文提出实例感知提示(IAP)框架,解决不同任务下提示设计优化问题。具体地,实例感知门控提示(IA-GP)在实例层面自适应分配提示参数于Transformer各层,增强新任务适应并缓解遗忘;实例感知类别分布驱动提示(IA-CDDP)通过为每个实例生成任务标签相关置信度分数,提升任务适配准确性。在11个数据集上,使用三个评估指标进行实验,验证了所提方法的有效性。代码已开源于https://github.com/FerdinandZJU/IAP。

原文摘要 · Abstract (English)

Recent pre-trained vision-language models (PT-VLMs) often face a Multi-Domain Task Incremental Learning (MTIL) scenario in practice, where several classes and domains of multi-modal tasks are incrementally arrived. Without access to previously seen tasks and unseen tasks, memory-constrained MTIL suffers from forward and backward forgetting. To alleviate the above challenges, parameter-efficient fine-tuning techniques (PEFT), such as prompt tuning, are employed to adapt the PT-VLM to the diverse incrementally learned tasks. To achieve effective new task adaptation, existing methods only consider the effect of PEFT strategy selection, but neglect the influence of PEFT parameter setting (e.g., prompting). In this paper, we tackle the challenge of optimizing prompt designs for diverse tasks in MTIL and propose an Instance-Aware Prompting (IAP) framework. Specifically, our Instance-Aware Gated Prompting (IA-GP) strategy enhances adaptation to new tasks while mitigating forgetting by adaptively assigning prompts across transformer layers at the instance level. Our Instance-Aware Class-Distribution-Driven Prompting (IA-CDDP) improves the task adaptation process by determining an accurate task-label-related confidence score for each instance. Experimental evaluations across 11 datasets, using three performance metrics, demonstrate the effectiveness of our proposed method. The source codes are available at https://github.com/FerdinandZJU/IAP.

持续学习提示调优视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。