用高斯混合模型解析自监督学习原理，揭示其为何能有效降维。
Understanding Self-Supervised Learning via Gaussian Mixture Models
- 将数据增强视为同成分的独立采样，构建理论分析框架。
- 对比学习在非各向同性高斯下仍能找最优低维子空间。
- 适用于多模态模型（如CLIP），可过滤噪声提升表征质量。
自监督学习通过损失函数促使数据点与其增强版本的嵌入相近，从而从无标签数据中学习表征。尽管效果显著，其理论机制尚不清晰。本文在高斯混合模型的自然设定下分析自监督学习，定义数据增强为来自同一混合成分的独立采样。结果表明，基础对比学习（即InfoNCE损失）即使在非各向同性高斯情况下也能找到最优低维子空间，而传统谱方法无法实现此目标。类似结论也适用于非对比式自监督学习（如SimSiam损失）。进一步扩展至多模态对比学习（如CLIP），发现其能学习到Fisher最优子空间的子集，有效滤除表示中的噪声。实验基于合成数据验证了上述理论发现。
原文摘要 · Abstract (English)
Self-supervised learning attempts to learn representations from un-labeled data; it does so via a loss function that encourages the embedding of a point to be close to that of its augmentations. This simple idea performs remarkably well, yet it is not precisely theoretically understood why this is the case. In this paper we analyze self-supervised learning in a natural context: dimensionality reduction in Gaussian Mixture Models. Crucially, we define an augmentation of a data point as being another independent draw from the same underlying mixture component. We show that vanilla contrastive learning (specifically, the InfoNCE loss) is able to find the optimal lower-dimensional subspace even when the Gaussians are not isotropic -- something that vanilla spectral techniques cannot do. We also prove a similar result for "non-contrastive" self-supervised learning (i.e., SimSiam loss). We further extend our analyses to multi-modal contrastive learning algorithms (e.g., CLIP). In this setting we show that contrastive learning learns the subset of fisher-optimal subspace, effectively filtering out all the noise from the learnt representations. Finally, we corroborate our theoretical finding through synthetic data experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。