提出无需调参的对比学习损失函数,提升性能并避免温度调节难题。
Temperature-Free Loss Function for Contrastive Learning
- 用反双曲正切函数替代温度缩放,实现无温度对比学习。
- 在5个基准上表现稳定,无需调参即达满意效果。
- 理论分析揭示原方法梯度问题,新方法优化训练稳定性。
对比学习作为自监督学习中最有前景的方法之一,在多个领域取得突破。主流实现方式为使用InfoNCE损失函数:通过捕捉样本对间的相似性来学习数据表示。尽管成功,但采用InfoNCE损失需调节核心超参数——温度,用于校准相似度得分。多项研究强调其重要性与对性能的敏感性,然而寻找有效温度需大量试错实验,增加了应用难度。为此,本文提出一种无需温度的InfoNCE损失部署方法:将温度缩放替换为反双曲正切函数,得到改进型InfoNCE损失。该方法不仅免去超参数调优,且在对比学习中表现出性能提升。详细理论分析表明,现有温度缩放在梯度下降中导致严重问题,而新方法具备更优的梯度特性。该方法在5个对比学习基准上验证,无需温度调节即可获得满意结果。
原文摘要 · Abstract (English)
As one of the most promising methods in self-supervised learning, contrastive learning has achieved a series of breakthroughs across numerous fields. A predominant approach to implementing contrastive learning is applying InfoNCE loss: By capturing the similarities between pairs, InfoNCE loss enables learning the representation of data. Albeit its success, adopting InfoNCE loss requires tuning a temperature, which is a core hyperparameter for calibrating similarity scores. Despite its significance and sensitivity to performance being emphasized by several studies, searching for a valid temperature requires extensive trial-and-error-based experiments, which increases the difficulty of adopting InfoNCE loss. To address this difficulty, we propose a novel method to deploy InfoNCE loss without temperature. Specifically, we replace temperature scaling with the inverse hyperbolic tangent function, resulting in a modified InfoNCE loss. In addition to hyperparameter-free deployment, we observed that the proposed method even yielded a performance gain in contrastive learning. Our detailed theoretical analysis discovers that the current practice of temperature scaling in InfoNCE loss causes serious problems in gradient descent, whereas our method provides desirable gradient properties. The proposed method was validated on five benchmarks on contrastive learning, yielding satisfactory results without temperature tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。