提出统一几何框架,揭示加权对比学习的可实现性与失效根源。
A Unified Geometric Framework for Weighted Contrastive Learning

- 将加权InfoNCE视为距离几何问题,用权重定义目标几何结构。
- 类别不平衡下,标准对比学习会破坏类间对称性,而软版保持规则单纯形结构。
- 标签非球面分布时,传统方法无法达最优,需采用几何一致的权重设计。
对比学习旨在通过学习反映相似性图的表示来保留样本间的关联结构,但其嵌入的几何特性仍不清晰。本文表明,加权InfoNCE目标可被解释为距离几何问题,其中权重方案指定了需由表示实现的目标几何。该视角精确刻画了若干有监督和弱监督目标下的最优嵌入。在有监督分类中,SupCon与Soft SupCon(其类间对为小非零相似)均使同类别样本坍缩至单一原型。然而,平衡的SupCon恢复经典正单纯形几何,类别不平衡则打破对称性:SupCon导致类间相似度不均,取决于类大小;而Soft SupCon无论是否平衡,均维持规则单纯形。在连续标签场景中,我们的框架揭示了不同失效模式:除非标签位于超球面上,y-Aware CL一般无法达到其熵最优,暴露欧氏标签权重与球面隐空间相似性的不匹配。相比之下,几何一致的设定如欧氏-欧氏权重或X-CLR可获得唯一最优嵌入。结果表明,权重选择决定了对比学习是否几何可实现、退化或不一致,为设计对比目标提供原则性框架。
原文摘要 · Abstract (English)
Contrastive learning (CL) aims to preserve relational structure between samples by learning representations that reflect a similarity graph. Yet, the geometry of the resulting embeddings remains poorly understood. Here we show that weighted InfoNCE objectives can be interpreted as Distance Geometry Problems, where the weighting scheme specifies the target geometry to be realized by the representation. This viewpoint yields exact characterizations of the optimal embeddings for several supervised and weakly supervised objectives. In supervised classification, both SupCon and Soft SupCon (a dense relaxation of it where pairs from distinct classes have small non-zero similarity) collapse samples within each class to a single prototype. However, while balanced SupCon recovers the classical regular simplex geometry, class imbalance breaks this symmetry: SupCon induces non-uniform inter-class similarities depending on class sizes, whereas Soft SupCon preserves a regular simplex geometry regardless of class imbalance. In continuous-label settings, our framework reveals a different failure mode: y-Aware CL generally cannot attain its entropic optimum unless the labels lie on a hypersphere, exposing a mismatch between Euclidean label weights and spherical latent similarity. By contrast, geometrically consistent choices such as Euclidean-Euclidean weighting or X-CLR admit unique optimal embeddings. Our results show that the choice of weighting scheme determines whether contrastive learning is geometrically realizable, degenerate, or inconsistent, providing a principled framework for designing contrastive objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。