动态调节对比学习温度,让多模态模型更精准区分相似与不相似样本。
TeMo: Temperature Modulation for Multimodal Contrastive Learning

- 根据样本相似度自适应调整温度,实现细粒度对比学习。
- 在零样本检索与分类任务中均刷新了性能纪录。
- 适合需要高精度多模态理解的场景,如跨模态搜索。
对比学习通过拉近相似样本、推远不相似样本实现优异性能。其关键超参数温度τ控制负样本的惩罚强度。现有方法通常固定或全局学习该参数。本文提出TeMo(Temperature Modulation)框架,一种基于相似度的温度调制方法,可针对每个正负样本对动态调整温度,实现更精细的多模态对比学习。该方法通过逐步过渡,将温度调制的多模态与单模态损失与标准多模态对比损失无缝结合,使模型在不同训练阶段捕捉粗粒度与细粒度语义。大量实验表明,TeMo各组件在多种零样本检索和分类任务中持续提升性能,达成新的最先进结果。
原文摘要 · Abstract (English)
Contrastive learning approaches achieve strong performance by training models to bring similar samples closer while pushing dissimilar samples apart. A crucial component of contrastive learning is the temperature hyperparameter $\tau$, which controls the penalty strength applied to negative samples. However, most existing methods either fix this hyperparameter or learn a global value during training. In this paper, we introduce TeMo, Temperature Modulation framework, a similarity-based modulation approach that adaptively adjusts the temperature for each positive-negative pair according to their similarity, enabling more fine-grained multimodal contrastive learning. Our approach seamlessly integrates temperature-modulated multimodal and unimodal losses with the standard multimodal contrastive loss by gradually transitioning between them. This design allows the model to capture both coarse- and fine-grained semantics at different training stages. Extensive experiments demonstrate that each component of TeMo consistently enhances performance across diverse zero-shot retrieval and classification tasks, establishing new state-of-the-art results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。