用欧氏几何替代余弦相似度，提升图文预训练模型性能
Embedding Geometries of Contrastive Language-Image Pre-Training
- 改用欧氏距离和欧氏软最大函数，优化图文对齐机制
- 在多个数据集上达到或超过CLIP的准确率，最高提升1.2%
- 适合关注模型设计原理与跨模态对齐的研究者
自CLIP发布以来，基于InfoNCE损失的对比学习成为多模态预训练的主流方法。尽管广泛应用，其原始设计中的L2归一化与余弦相似度得分仍少被重新审视。我们系统性地测试了语言-图像预训练中不同的几何结构与软最大函数，并发现采用直观欧氏几何的变体——欧氏CLIP（EuCLIP）——在性能上可媲美甚至超越原始CLIP。同时，它在支持层次关系方面至少与更复杂的双曲几何方法相当。
原文摘要 · Abstract (English)
Since the publication of CLIP, the approach of using InfoNCE loss for contrastive pre-training has become widely popular for bridging two or more modalities. Despite its wide adoption, CLIP's original design choices of L2 normalization and cosine similarity logit have rarely been revisited. We have systematically experimented with alternative geometries and softmax logits for language-image pre-training and identified that variants with intuitive Euclidean geometry, Euclidean CLIP (EuCLIP), match or exceed the performance of CLIP and support hierarchical relationships at least as well as more complicated hyperbolic alternative.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。