提出xNCE解决多模态对比学习中的模态鸿沟问题，提升零样本性能。
On the modality gap and the contrastive loss in multi-modal representation learning
- 引入跨模态与同模态负例的混合对比损失xNCE
- 在低温度下显著缩小图像与文本嵌入差距，保持检索性能
- 零样本分类全面超越基线，适合追求鲁棒性的模型设计者
我们研究了CLIP类双编码器对比学习中的模态鸿沟问题，即尽管图像与文本嵌入在共享空间中训练，仍存在对齐偏差。我们指出该现象源于独立编码器下InfoNCE公式的缺陷。通过单模态实验发现，在低温条件下InfoNCE会主动制造模态鸿沟。理论分析表明，此为InfoNCE在低温下的模式失效。为此提出xNCE：使用跨模态及同模态负例进行对比学习。xNCE在MS-COCO上达到与InfoNCE相当的检索性能，且在所有温度下均有效缩小鸿沟。值得注意的是，xNCE在所有基准测试中均优于InfoNCE的零样本分类表现，而高温度InfoNCE与正则化InfoNCE均未实现提升，证明xNCE在不损害迁移所需判别几何的前提下降低了模态鸿沟。
原文摘要 · Abstract (English)
We study the modality gap in CLIP-style dual-encoder contrastive learning, where image and text embeddings remain misaligned despite being trained in a shared space. We argue that the gap is induced by a failure of the InfoNCE formulation with independent encoders. We conduct a uni-modal experiment with two independent encoders and identical initialization conditions and find that InfoNCE actively generates a gap at low temperatures. We provide a theoretical analysis of this phenomenon and show that the modality gap is indeed a mode-failure of InfoNCE, but only at low temperatures. We propose a simple modification called xNCE, which uses intermodal as well as intra-modality negative contrastive pairs. xNCE matches retrieval performance on MS-COCO while consistently reducing the gap even at low temperatures. Notably, xNCE improves zero-shot classification over the InfoNCE baseline across all benchmarks, whereas high-temperature InfoNCE and regularized InfoNCE both fail to do so, demonstrating that xNCE reduces the modality gap without sacrificing the discriminative geometry needed for transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。