通过自适应调节温度参数,提升对比学习在分布外数据上的泛化能力。
Improving Domain Generalization in Contrastive Learning using Adaptive Temperature Control
- 根据负样本与锚点同域概率动态调整温度,强化域不变特征学习
- 在分布外测试上性能优于基线方法,且在分布内任务上表现更优
- 适合需要强泛化能力的少标注场景,如跨域图像识别
使用对比学习进行自监督预训练是处理稀疏标注数据的强大方法,但在训练与测试数据分布发生偏移时性能可能显著下降。本文研究了训练数据来自多个域,而测试数据来自未见域且存在显著协变量偏移的情况。提出一种新方法,在对比学习中引入域标签,通过调整InfoNCE损失中的温度参数(控制负样本权重)来增强表示的域不变性。该温度根据负样本与锚点来自同一域的概率自适应调整,使来自相似域的样本对权重更高,促使模型基于域不变属性进行区分。在一种变体MNIST数据集上的实验表明,该方法在分布外性能上优于域泛化基线;同时在分布内任务上也显著优于基线,保持了优异的性能。
原文摘要 · Abstract (English)
Self-supervised pre-training with contrastive learning is a powerful method for learning from sparsely labeled data. However, performance can drop considerably when there is a shift in the distribution of data from training to test time. We study this phenomenon in a setting in which the training data come from multiple domains, and the test data come from a domain not seen at training that is subject to significant covariate shift. We present a new method for contrastive learning that incorporates domain labels to increase the domain invariance of learned representations, leading to improved out-of-distribution generalization. Our method adjusts the temperature parameter in the InfoNCE loss -- which controls the relative weighting of negative pairs -- using the probability that a negative sample comes from the same domain as the anchor. This upweights pairs from more similar domains, encouraging the model to discriminate samples based on domain-invariant attributes. Through experiments on a variant of the MNIST dataset, we demonstrate that our method yields better out-of-distribution performance than domain generalization baselines. Furthermore, our method maintains strong in-distribution task performance, substantially outperforming baselines on this measure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。