arXiv:2506.04411cs.LG2025-06NeurIPS被引 5

自监督对比学习本质上近似于有监督的对比学习,理论解释其成功原理。

Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning

  • 发现自监督对比学习隐含逼近一种只保留不同类样本的有监督损失
  • 证明当类别数增加时,两者损失差距趋近于零,且不依赖标签和网络结构
  • 揭示了模型表示的几何特性,适用于少样本线性探测任务

尽管自监督对比学习(CL)在实践中表现优异,但其理论基础尚未完全建立。本文证明,标准的CL目标函数在本质上近似于我们提出的仅含负样本的有监督对比损失(NSCL),该损失排除了同类别样本间的对比。在类别数增加的条件下,我们证明了CL与NSCL损失之间的差距趋于零,且该界具有标签无关性和架构无关性。我们刻画了NSCL全局最小值的几何结构:学习到的表示存在增强坍缩、类内坍缩,且类中心构成等角紧框架(simplex equiangular tight frame)。我们还提出一个新的少样本线性探测误差上界,其依赖于两个特征可变性度量——类内离散度和类中心连线方向上的变化。结果表明,方向变化主导上界,而类内离散度的影响随标注样本数增加而减弱。这些性质使CL与NSCL训练的表示能通过简单线性探测实现高精度少样本标签恢复。我们进一步通过实验证实:CL与NSCL损失差距以$/mathcal{O}(1/ ext{类别数})$速率衰减;两者高度相关;最小化CL损失会将NSCL损失拉近到直接优化所得的水平;所提误差上界能紧密预测实际探测性能。

原文摘要 · Abstract (English)

Despite its empirical success, the theoretical foundations of self-supervised contrastive learning (CL) are not yet fully established. In this work, we address this gap by showing that standard CL objectives implicitly approximate a supervised variant we call the negatives-only supervised contrastive loss (NSCL), which excludes same-class contrasts. We prove that the gap between the CL and NSCL losses vanishes as the number of semantic classes increases, under a bound that is both label-agnostic and architecture-independent. We characterize the geometric structure of the global minimizers of the NSCL loss: the learned representations exhibit augmentation collapse, within-class collapse, and class centers that form a simplex equiangular tight frame. We further introduce a new bound on the few-shot error of linear-probing. This bound depends on two measures of feature variability--within-class dispersion and variation along the line between class centers. We show that directional variation dominates the bound and that the within-class dispersion's effect diminishes as the number of labeled samples increases. These properties enable CL and NSCL-trained representations to support accurate few-shot label recovery using simple linear probes. Finally, we empirically validate our theoretical findings: the gap between CL and NSCL losses decays at a rate of $\mathcal{O}(\frac{1}{\#\text{classes}})$; the two losses are highly correlated; minimizing the CL loss implicitly brings the NSCL loss close to the value achieved by direct minimization; and the proposed few-shot error bound provides a tight estimate of probing performance in practice. The code and project page of the paper are available at [\href{https://github.com/DLFundamentals/understanding-ssl}{code}, \href{https://dlfundamentals.github.io/ssl-is-approximately-sl/}{project page}].

自监督学习对比学习理论分析少样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。