通过分层对齐提升弱监督视觉定位的精度
AlignCAT: Visual-Linguistic Alignment of Category and Attribute for Weakly Supervised Visual Grounding
- 分两阶段对齐：先用类别和全局上下文粗对齐，再用描述信息细对齐
- 在RefCOCO等三个数据集上均优于现有方法
- 适合需要精准文本-图像匹配的研究者
弱监督视觉定位旨在根据文本描述定位图像中的物体。尽管已有显著进展，但现有方法因类别与属性层面的语义模糊，缺乏足够的跨模态推理能力。为此，我们提出AlignCAT，一种基于查询的语义匹配框架。首先，粗粒度对齐模块利用类别信息和全局上下文，有效缓解类别不一致物体的干扰；其次，细粒度对齐模块挖掘描述性信息并捕捉词级文本特征，实现属性一致性。通过充分挖掘语言线索，AlignCAT逐步过滤误对齐的视觉查询，提升对比学习效率。在RefCOCO、RefCOCO+和RefCOCOg三个基准测试上的实验证明，该方法在两项视觉定位任务中均优于现有弱监督方法。代码已开源。
原文摘要 · Abstract (English)
Weakly supervised visual grounding (VG) aims to locate objects in images based on text descriptions. Despite significant progress, existing methods lack strong cross-modal reasoning to distinguish subtle semantic differences in text expressions due to category-based and attribute-based ambiguity. To address these challenges, we introduce AlignCAT, a novel query-based semantic matching framework for weakly supervised VG. To enhance visual-linguistic alignment, we propose a coarse-grained alignment module that utilizes category information and global context, effectively mitigating interference from category-inconsistent objects. Subsequently, a fine-grained alignment module leverages descriptive information and captures word-level text features to achieve attribute consistency. By exploiting linguistic cues to their fullest extent, our proposed AlignCAT progressively filters out misaligned visual queries and enhances contrastive learning efficiency. Extensive experiments on three VG benchmarks, namely RefCOCO, RefCOCO+, and RefCOCOg, verify the superiority of AlignCAT against existing weakly supervised methods on two VG tasks. Our code is available at: https://github.com/I2-Multimedia-Lab/AlignCAT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。