arXiv:2507.15480cs.CV2025-07ICCV被引 1

通过动态调整跨模态融合矩阵,提升视觉语言模型的微调效果。

One Last Attention for Your Vision-Language Model

  • 在模型末尾加轻量注意力层,生成可学习掩码来调节最终融合表示。
  • 在多种微调场景下均显著提升性能,甚至接近当前最佳方法。
  • 无需修改中间特征,代码改动极小,适合快速部署与实验。

预训练的视觉语言模型(如CLIP)虽具备出色的零样本能力,但其下游性能依赖于有效的微调。现有方法多聚焦于单模态表示(文本或视觉)的优化,却忽略了决策过程中关键的融合表示(即理性矩阵)的作用。为此,本文提出一种简单而有效的理性适配方法(RAda),通过在视觉语言模型末尾添加轻量级注意力层,生成可学习掩码,动态校准理性矩阵中每个元素的贡献,从而实现对最终跨模态交互的精准调整,而无需修改中间特征。实验表明,在不同设置下(如冻结或更新预训练编码器、仅能访问无标签测试数据的测试时训练),RAda均表现出良好的泛化能力,性能优于基线且接近当前最优方法。代码已开源。

原文摘要 · Abstract (English)

Pretrained vision-language models (VLMs), such as CLIP, achieve remarkable zero-shot performance, yet their downstream potential hinges on effective fine-tuning. Most adaptation methods typically focus on refining representation from separate modalities (text or vision) but neglect the critical role of their fused representations in the decision-making process, \emph{\ie} rational matrix that drives the final prediction. To bridge the gap, we propose a simple yet effective \textbf{R}ational \textbf{Ada}ptaion ({RAda}) to explicitly exploit the final fused representation during fine-tuning. RAda employs a learned mask, obtained from a lightweight attention layer attached at the end of a VLM, to dynamically calibrate the contribution of each element in the rational matrix, enabling targeted adjustments to the final cross-modal interactions without incurring costly modifications to intermediate features. Experiments in different settings (i.e., updating, or freezing pretrained encoders in adaptation, and test-time training that can only access the unlabeled test data) show that RAda serves as a versatile fine-tuning technique, improving the baseline with minimal code and performing comparably against current arts in most settings. Code is available at \href{https://github.com/khufia/RAda/tree/main}{github.com/khufia/RAda}.

视觉语言微调注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。