用统一对比损失提升自训练,少标注数据下表现更优
A Unified Contrastive Loss for Self-Training
- 用可学习类别原型替代交叉熵,统一为对比损失框架
- 在三个数据集上少标注时性能显著提升,收敛更快
- 适合标签稀缺场景,对超参数不敏感,迁移能力更强
自训练方法在半监督学习中有效利用大量无标签数据,尤其在标签数据稀少时表现突出。尽管许多方法依赖交叉熵损失(CE),但近期研究显示监督对比损失(SupCon)更具优势。此外,无监督对比学习也能生成高质量表征。为此,本文提出一个通用框架,将自训练中的所有交叉熵损失替换为统一的对比损失,并引入类别原型(类别的可学习参数),恢复了与交叉熵设置的理论等价性。该框架应用于主流自训练方法后,在三个不同数据集上均实现显著性能提升,且具备更快收敛速度、更强迁移能力及更高的超参数稳定性。代码已开源。
原文摘要 · Abstract (English)
Self-training methods have proven to be effective in exploiting abundant unlabeled data in semi-supervised learning, particularly when labeled data is scarce. While many of these approaches rely on a cross-entropy loss function (CE), recent advances have shown that the supervised contrastive loss function (SupCon) can be more effective. Additionally, unsupervised contrastive learning approaches have also been shown to capture high quality data representations in the unsupervised setting. To benefit from these advantages in a semi-supervised setting, we propose a general framework to enhance self-training methods, which replaces all instances of CE losses with a unique contrastive loss. By using class prototypes, which are a set of class-wise trainable parameters, we recover the probability distributions of the CE setting and show a theoretical equivalence with it. Our framework, when applied to popular self-training methods, results in significant performance improvements across three different datasets with a limited number of labeled data. Additionally, we demonstrate further improvements in convergence speed, transfer ability, and hyperparameter stability. The code is available at \url{https://github.com/AurelienGauffre/semisupcon/}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。