arXiv:2609.06967cs.CVcs.AI2026-09

通过感知变换的提示机制,提升视觉语言模型迁移能力。

Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models

论文配图:Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models
图 1 · 摘自论文原文
  • 用固定文本描述图像变换,实现跨模态一致性。
  • 新损失函数缓解多正例时梯度稀释问题。
  • 适合需要强泛化与少样本迁移的场景。

在不损害视觉语言模型泛化能力的前提下,实现有效的迁移学习至关重要。然而,现有方法常忽视数据特性,简单沿用预训练阶段的训练策略,将同类别样本视为独立实例,并忽略图像与其对应文本提示之间的关联变换,导致模型学习难度增加。本文提出基于变换感知的提示条件化与重校准对比损失。固定文本描述符用于识别配对图像所施加的变换,从而在变换层级上保持一致,而不改变类别语义。该设计使图像与文本分支在变换层面对齐,生成更丰富的表征,同时保留模型泛化能力。此外,所提损失函数有效缓解了软目标交叉熵中每个锚点存在多个正例时的正梯度稀释问题。在迁移学习中,本方法将同类别样本视为正例而非独立实例,促进模型更有效地学习领域特定特征。在分布偏移、迁移学习和少样本设置下的实验均显示优于现有方法。源代码已公开于 https://github.com/SoongE/ReCalCon。

原文摘要 · Abstract (English)

Ensuring effective transfer learning for vision-language models without compromising their generalization performance is crucial. However, many existing methods overlook data characteristics and simply reuse the training strategies adopted during pre-training. Specifically, they treat same-class samples as distinct instances and transform images independently of their paired text prompts, which makes model learning more difficult. We address these limitations through transformation-aware prompt conditioning and a re-calibrated contrastive loss. Fixed text descriptors identify the transformations applied to paired images, providing transformation-level consistency without altering class semantics. This design aligns the image and text branches at the transformation level, enabling richer representations while preserving the models' ability to generalize. In addition, our loss function mitigates positive-gradient dilution in soft-target cross-entropy when each anchor has multiple valid positives. During transfer, our approach treats same-class samples as positives rather than distinct instances, enabling the model to learn domain-specific features more effectively. Experiments across distribution shift, transfer learning, and few-shot settings demonstrate consistent improvements over existing approaches. Source code for our method is available at https://github.com/SoongE/ReCalCon.

视觉语言模型迁移学习对比学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。