通过引导梯度停止机制，实现无需负样本的对比学习，提升自监督模型稳定性。
Implicit Contrastive Representation Learning with Guided Stop-gradient
- 利用异构网络结构与梯度截断，隐式引入对比学习思想
- 在小批量下训练更稳定，且无需预测器也能防止特征坍塌
- 适用于SimSiam、BYOL等主流自监督算法，性能显著提升
在自监督表示学习中，孪生网络通过拉近正样本对的表示来学习变换不变性，但易陷入退化解。对比学习通过负样本对损失避免坍塌，但对负样本数量减少不鲁棒。为此，部分算法采用仅正样本方法，依赖源-目标编码器的异构架构应对坍塌问题。本文基于此架构，提出一种隐式引入对比学习思想的方法——引导梯度停止（Guided Stop-gradient）。将该方法应用于SimSiam和BYOL等基准算法，结果显示训练更稳定，性能提升明显。此外，使用该方法的算法在小批量设置下仍表现良好，即使无预测器也未发生坍塌。代码已开源：https://github.com/bych-lee/gsg。
原文摘要 · Abstract (English)
In self-supervised representation learning, Siamese networks are a natural architecture for learning transformation-invariance by bringing representations of positive pairs closer together. But it is prone to collapse into a degenerate solution. To address the issue, in contrastive learning, a contrastive loss is used to prevent collapse by moving representations of negative pairs away from each other. But it is known that algorithms with negative sampling are not robust to a reduction in the number of negative samples. So, on the other hand, there are algorithms that do not use negative pairs. Many positive-only algorithms adopt asymmetric network architecture consisting of source and target encoders as a key factor in coping with collapse. By exploiting the asymmetric architecture, we introduce a methodology to implicitly incorporate the idea of contrastive learning. As its implementation, we present a novel method guided stop-gradient. We apply our method to benchmark algorithms SimSiam and BYOL and show that our method stabilizes training and boosts performance. We also show that the algorithms with our method work well with small batch sizes and do not collapse even when there is no predictor. The code is available at https://github.com/bych-lee/gsg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。