解释可训练温度与偏置如何让Sigmoid对比损失全局最优
Global Minimizers of Sigmoid Contrastive Loss
- 通过可训练温度和偏置实现损失函数全局最小化
- 在特定配置下损失可降至零，适用于高维表示空间
- 为SigLIP等模型提供理论支持，适合研究对比学习者
通过对比预训练获取并对齐表示的元任务自CLIP和ALIGN提出以来日益重要。本文从理论上解释了在Sigmoid损失中使用可训练逆温度和偏置的优势，这正是Google DeepMind近期提出的SigLIP和SigLIP2模型所采用的方法。温度和偏置能使损失函数在一类称为$(\mathsf{m}, \mathsf{b}_{\mathsf{rel}})$-Constellations的丰富配置下趋近于零。$(\mathsf{m}, \mathsf{b}_{\mathsf{rel}})$-Constellations是与球面码相关的新型组合对象，由边界$\mathsf{m}$和相对偏置$\mathsf{b}_{\mathsf{rel}}$参数化。我们利用对这类构型的刻画，从理论上解释了SigLIP在检索任务中的成功，揭示了SigLIP与CLIP中存在的模态差距，并确定了生成高质量表示所需的最低维度。最后，我们提出了一个显式包含相对偏置的Sigmoid损失重参数化形式，在合成数据实验中改善了训练动态。
原文摘要 · Abstract (English)
The meta-task of obtaining and aligning representations through contrastive pretraining is steadily gaining importance since its introduction in CLIP and ALIGN. In this paper we theoretically explain the advantages of synchronizing with trainable inverse temperature and bias under the sigmoid loss, as implemented in the recent SigLIP and SigLIP2 models of Google DeepMind. Temperature and bias can drive the loss function to zero for a rich class of configurations that we call $(\mathsf{m}, \mathsf{b}_{\mathsf{rel}})$-Constellations. $(\mathsf{m}, \mathsf{b}_{\mathsf{rel}})$-Constellations are a novel combinatorial object related to spherical codes and are parametrized by a margin $\mathsf{m}$ and relative bias $\mathsf{b}_{\mathsf{rel}}$. We use our characterization of constellations to theoretically justify the success of SigLIP on retrieval, to explain the modality gap present in SigLIP and CLIP, and to identify the necessary dimension for producing high-quality representations. Finally, we propose a reparameterization of the sigmoid loss with explicit relative bias, which improves training dynamics in experiments with synthetic data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。