arXiv:2506.09781cs.LGstat.ML2025-06ICML被引 3

揭示对比学习中嵌入相似性的本质,提出优化小批量训练的新方法。

On the Similarities of Embeddings in Contrastive Learning

  • 用余弦相似度统一分析对比学习机制
  • 小批量训练时负样本相似性方差大,影响表示质量
  • 设计辅助损失降低方差,提升小批量性能

对比学习基于简单而有效的原则:正样本对的嵌入被拉近,负样本对的嵌入被推开。本文从余弦相似度视角提出统一框架,得出两个关键理论发现:在全批量设置下,当负样本对相似度低于阈值时,正样本对无法实现完全对齐,可通过引入视图内负样本缓解;在小批量设置下,较小的批量大小会增强负样本间的分离度,即其相似性方差更高,从而导致表示质量下降。为此,我们提出一种辅助损失函数,用于降低小批量下负样本相似性的方差。实验表明,该损失能有效提升小批量场景下的性能。

原文摘要 · Abstract (English)

Contrastive learning operates on a simple yet effective principle: Embeddings of positive pairs are pulled together, while those of negative pairs are pushed apart. In this paper, we propose a unified framework for understanding contrastive learning through the lens of cosine similarity, and present two key theoretical insights derived from this framework. First, in full-batch settings, we show that perfect alignment of positive pairs is unattainable when negative-pair similarities fall below a threshold, and this misalignment can be mitigated by incorporating within-view negative pairs into the objective. Second, in mini-batch settings, smaller batch sizes induce stronger separation among negative pairs in the embedding space, i.e., higher variance in their similarities, which in turn degrades the quality of learned representations compared to full-batch settings. To address this, we propose an auxiliary loss that reduces the variance of negative-pair similarities in mini-batch settings. Empirical results show that incorporating the proposed loss improves performance in small-batch settings.

对比学习嵌入对齐小批量训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。