用少量标签数据训练两层卷积网络，SimCLR预训练可逼近最优性能。
Understanding the Benefits of SimCLR Pre-Training in Two-Layer Convolutional Neural Networks
- 在玩具图像模型上分析两层CNN的SimCLR预训练机制
- 标签量有限时，联合预训练+微调接近最优测试误差
- 相比直接监督训练，所需标签数显著更少
SimCLR是视觉任务中最具影响力的对比学习方法之一。它通过让模型区分增强图像的正负样本对，利用大量无标签数据预训练深层神经网络。人们普遍认为，SimCLR能学习到高效的表示，从而提升后续监督微调的性能。尽管效果显著，但其内在机理仍缺乏理论理解。本文从理论角度对SimCLR方法进行案例研究，聚焦于训练一个两层卷积神经网络（CNN）来学习一个玩具图像数据模型。我们证明，在特定标签数量条件下，结合SimCLR预训练与监督微调可实现几乎最优的测试损失。值得注意的是，SimCLR预训练所需的标签复杂度远低于直接在监督数据上训练。该分析揭示了SimCLR在减少标签依赖方面的优势。
原文摘要 · Abstract (English)
SimCLR is one of the most popular contrastive learning methods for vision tasks. It pre-trains deep neural networks based on a large amount of unlabeled data by teaching the model to distinguish between positive and negative pairs of augmented images. It is believed that SimCLR can pre-train a deep neural network to learn efficient representations that can lead to a better performance of future supervised fine-tuning. Despite its effectiveness, our theoretical understanding of the underlying mechanisms of SimCLR is still limited. In this paper, we theoretically introduce a case study of the SimCLR method. Specifically, we consider training a two-layer convolutional neural network (CNN) to learn a toy image data model. We show that, under certain conditions on the number of labeled data, SimCLR pre-training combined with supervised fine-tuning achieves almost optimal test loss. Notably, the label complexity for SimCLR pre-training is far less demanding compared to direct training on supervised data. Our analysis sheds light on the benefits of SimCLR in learning with fewer labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。