用双曲空间优化CLIP,让模型更懂长文本图文对齐。
HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding

- 将CLIP的欧式对齐转为双曲空间,利用其层级结构建模长文本语义
- 在文本重排、摘要等扰动下,长文本检索性能提升19.5%
- 适用于CLIP和Stable Diffusion XL等多类模型,提升鲁棒性
CLIP已成为图像-文本对齐的主流范式,但在长文本描述(>77个词元)上表现不佳,原因在于其绝对位置编码及预训练时使用短标题。长文本常被重排、摘要或部分省略,现有方法虽扩展了位置编码,但面对文本扰动时图像-文本对齐能力仍下降。本文归因于欧氏对比目标强制一对一匹配,缺乏对全局上下文与局部元素间层次关系的显式建模。为此,提出HyFL-CLIP,通过跨流形相似性蒸馏,将欧氏空间中已学习的图文对齐知识迁移到双曲空间,利用其几何特性捕捉层级与蕴含关系。该方法通过爱因斯坦中点聚合,关联词粒度特征、长文本描述、短文本成分与图像,显式建模部分-整体关系。在多样基准测试中,包括长文本跨模态检索、带标题扰动的跨模态检索、同模态检索及短文本跨模态检索,结果显示HyFL-CLIP显著提升长文本理解鲁棒性,尤其在文本扰动下长文本检索性能较最优基线最高提升19.5%。此外,该方法可无缝集成至Stable Diffusion XL(SDXL)等框架中。
原文摘要 · Abstract (English)
CLIP (Contrastive Language-Image Pre-training) has become a de facto paradigm for image-text alignment, but it struggles with long-context descriptions (>77 tokens) due to absolute positional encoding and pretraining on short captions. In long contexts, sentences are often reordered, summarized, or partially omitted. Although prior works extend CLIP with longer positional encodings, they often suffer from degraded image-text alignment under such text perturbations. We attribute this limitation to the Euclidean contrastive objective, which enforces strict one-to-one matching and lacks explicit mechanisms for modeling hierarchical relationships between global context and its constituent elements. To address this issue, we propose HyFL-CLIP, a hyperbolic fine-tuning framework that distills the well-established text-image alignment learned in Euclidean CLIP into hyperbolic space via cross-manifold similarity distillation, leveraging its geometry to capture hierarchical and entailment relations. Our method models hierarchical semantics by linking summarized token-wise features, long-context descriptions, constituent short textual components, and images, capturing part-whole relationships via hyperbolic entailment with Einstein midpoint aggregation. Experiments on diverse benchmarks, including long-context cross-modal retrieval, cross-modal retrieval with caption perturbations, intra-modality retrieval, and short-text cross-modal retrieval, show that HyFL-CLIP achieves more robust long-context understanding. In particular, it yields up to 19.5% improvement in long-text cross-modal retrieval under textual perturbations over the best prior method. We also show HyFL-CLIP can be seamlessly integrated into other model frameworks by applying it to Stable Diffusion XL (SDXL).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。