揭示对比学习在自然图像中为何有效，提出最优网络结构与学习机制
A Theory of Contrastive Learning with Natural Images
- 最优表示由正弦滤波器构成的CNN实现，首层为正弦基函数
- 训练后网络自动学习正弦滤波器并完成部分去相关化
- 适用于理解对比学习本质，适合研究表征学习的学者
为何使用简单图像和数据增强的对比学习能产生对下游任务有用的表征？我们通过解析计算，在一系列基础增强方式和具有平稳统计特性的任意图像数据集上，求解了对比损失下的最优表示。结果表明，对于某些增强方式，最优表示可通过一个首层滤波器为正弦函数的卷积神经网络实现，后续包含逐点非线性、全局平均池化及执行部分去相关的线性层。我们还发现，对于更复杂的增强方式，此类CNN中的最优权重仍为正弦形式。正弦函数的频率和权重可通过给定数据集期望功率谱的简单水填算法计算得出。在不同图像数据集和增强方式上的实验表明，使用SGD训练的此类CNN会实际学习到首层的正弦滤波器，并完成部分去相关化。
原文摘要 · Abstract (English)
Why does contrastive learning with simple images and augmentations yield useful representations for downstream tasks? We address this question by analytically computing the optimal representation in terms of a contrastive loss for a range of basic augmentations and any image dataset with stationary statistics. We show that for certain augmentations the optimum can be attained by a CNN whose first layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer that performs partial whitening. We also show that the optimal weights in such CNNs for more complicated augmentations are still sinusoids. The frequencies of the sinusoids and their weights can be computed using a simple waterfilling algorithm given the dataset's expected power spectrum. Experiments with different image datasets and augmentations show that such CNNs trained with SGD empirically learn sinusoids in their first layer and to perform partial whitening
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。