让图文对齐更精细,从整图到局部区域逐级匹配。
$β$-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignment
- 用跨注意力动态聚合图像块,实现多粒度图文对齐。
- 在Urban1K上图文检索准确率达91.8%和92.3%,FG-OVD硬集达30.9%。
- 适合需要细粒度图文理解的任务,如视觉定位与密集检索。
CLIP通过对齐全局视觉与文本表征实现了强大的零样本图像-文本检索,但在细粒度任务上表现仍不足,即使使用长而详细的描述进行微调也是如此。本文提出$β$-CLIP,一种多粒度文本条件对比学习框架,旨在实现从完整描述到句子、短语等不同粒度文本与其对应视觉区域的分层对齐。针对每一粒度层级,$β$-CLIP利用跨注意力机制动态聚合图像块,生成上下文感知的视觉嵌入。为解决该层级结构中的语义重叠问题,引入$β$-上下文对比对齐损失($β$-CAL),该目标参数化了严格查询特定匹配与宽松图像内上下文化之间的权衡,支持软交叉熵与硬二值交叉熵两种形式。实验表明,两种损失在分层监督下表现各异:交叉熵的softmax增强细粒度区分能力,二值交叉熵的sigmoid更利于长文本检索,两者均受益于层次结构。大量实验证明,$β$-CLIP显著提升密集对齐性能,在Urban1K上达到T2I 91.8%、I2T 92.3%的R@1准确率,在FG-OVD(Hard)上达30.9%,超越无需硬负例训练的方法。$β$-CLIP为密集视觉-语言对应关系建立了鲁棒且自适应的基准。代码与模型已开源。
原文摘要 · Abstract (English)
CLIP achieves strong zero-shot image-text retrieval by aligning global vision and text representations, yet it falls behind on fine-grained tasks even when fine-tuned on long, detailed captions. In this work, we propose $β$-CLIP, a multi-granular text-conditioned contrastive learning framework designed to achieve hierarchical alignment between multiple textual granularities-from full captions to sentences and phrases-and their corresponding visual regions. For each level of granularity, $β$-CLIP utilizes cross-attention to dynamically pool image patches, producing contextualized visual embeddings. To address the semantic overlap inherent in this hierarchy, we introduce the $β$-Contextualized Contrastive Alignment Loss ($β$-CAL). This objective parameterizes the trade-off between strict query-specific matching and relaxed intra-image contextualization, supporting both soft Cross-Entropy and hard Binary Cross-Entropy formulations. We find that each loss interacts differently with hierarchical supervision: CE's softmax sharpens fine-grained discrimination, while BCE's sigmoid favors long-text retrieval while both benefit from hierarchy. Through extensive experiments, we demonstrate that $β$-CLIP significantly improves dense alignment: achieving 91.8% T2I 92.3% I2T at R@1 on Urban1K and 30.9% on FG-OVD (Hard), setting state-of-the-art among methods trained without hard negatives. $β$-CLIP establishes a robust, adaptive baseline for dense vision-language correspondence. The code and models are released at https://github.com/fzohra/B-CLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。