让视觉语言模型学会层次关系,提升图文匹配精度。
Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training

- 用层级查询条件控制视觉特征聚合,融合全局与局部信息
- 在COCO和Flickr上提升R@5和R@10检索准确率,多标签分类性能更优
- 适合关注细粒度图文对齐与结构化表示的研究者
基于对比学习的视觉语言模型(VLM)虽能学习强健的全局图文表征,但其欧氏嵌入和全局池化难以表达部件-整体、父-子等关系结构。超球面VLM通过蕴含目标缓解此问题,而文本条件化方法则通过句级和短语级查询实现细粒度对齐。然而两者尚未结合:超球面VLM使用静态图像与区域特征,而查询条件化方法缺乏层级几何结构。本文提出Hyper3-CLIP,一种层次条件化的超球面VLM,融合全局、局部及全局-局部对比学习,并引入查询条件化视觉池化。训练时,从文本构建轻量级查询层次,包括完整标题、句子片段、局部部件描述和提取短语。每个查询引导视觉块的池化,生成支持图像-文本、整体-部分和父-子蕴含损失的表示。查询条件化池化仅在训练阶段激活。实验表明,Hyper3-CLIP在COCO和Flickr上提升R@5和R@10检索性能,在VOC和COCO上改善多标签分类表现,同时在层次指标上保持竞争力。我们还评估了固定提示下的零样本提示敏感性,并研究了训练中使用的局部GRIT部件预算的影响。代码已开源。
原文摘要 · Abstract (English)
CLIP-like vision-language models (VLMs) trained with contrastive objectives learn strong global image-text representations, but their Euclidean embeddings and global pooling fail to encode relational structure such as part-whole and parent-child relations. Hyperbolic VLMs address this gap with entailment-based objectives, and text-conditioned variants improve fine-grained alignment through sentence- and phrase-level queries. However, these two lines of work remain separate: hyperbolic VLMs use static image and region features, while query-conditioned methods lack hierarchical geometric structure. We present Hyper3-CLIP, a hierarchy-conditioned hyperbolic VLM that combines global, local, and global-local contrastive learning with query-conditioned visual pooling. To train the model, we construct lightweight query hierarchies from text, comprising full captions, sentence fragments, localized part descriptions, and extracted phrases. Each query conditions the pooling of visual patches, and the resulting representations support image-text, whole-part, and parent-child entailment losses. Query-conditioned pooling is active only during training. Hyper3-CLIP improves R@5 and R@10 retrieval on COCO and Flickr, as well as multi-label classification on VOC and COCO, while remaining competitive on hierarchy metrics. We also audit zero-shot prompt sensitivity under fixed prompt regimes and study the effect of the localized GRIT part budget used during training. Code is available at https://github.com/Hyper3Labs/hyper3-clip.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。