解决视觉语言模型微调时前景注意力偏移问题
FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models
- 通过可学习的前景可靠性门增强前景视图质量
- 在多个数据集上提升准确率，最高达+3.2%
- 适合需要稳定微调效果的视觉语言模型应用
基于CLIP的提示调优使预训练视觉语言模型能高效适应下游任务。尽管已有研究取得显著进展，但对调优过程中视觉编码器内部注意力表征变化的关注仍有限。本文将提示调优预测失败归因于视觉编码器前景注意力的偏移，提出前景视图引导提示调优（FVG-PT），一种自适应的即插即用前景注意力引导模块。具体而言，FVG-PT引入可学习的前景可靠性门以自动提升前景视图质量，采用前景蒸馏补偿模块引导视觉注意力聚焦前景，并进一步设计先验校准模块以缓解过度关注前景导致的泛化性能下降。在多个骨干模型和数据集上的实验表明FVG-PT具有有效性与兼容性。代码已开源：https://github.com/JREion/FVG-PT
原文摘要 · Abstract (English)
CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they pay limited attention to changes in the internal attention representations of VLMs during the tuning process. In this paper, we attribute the failure modes of prompt tuning predictions to shifts in foreground attention of the visual encoder, and propose Foreground View-Guided Prompt Tuning (FVG-PT), an adaptive plug-and-play foreground attention guidance module, to alleviate the shifts. Concretely, FVG-PT introduces a learnable Foreground Reliability Gate to automatically enhance the foreground view quality, applies a Foreground Distillation Compensation module to guide visual attention toward the foreground, and further introduces a Prior Calibration module to mitigate generalization degradation caused by excessive focus on the foreground. Experiments on multiple backbone models and datasets show the effectiveness and compatibility of FVG-PT. Codes are available at: https://github.com/JREion/FVG-PT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。