arXiv:2505.22850cs.CV2025-05被引 3

将对比学习从图文空间转到视觉嵌入空间,提升细粒度计数精度。

What is the Right Embedding Space for Contrastive Learning in Referring Expression Counting?

  • 在图像内部视觉令牌间进行对比学习,无需依赖文本对齐。
  • 负样本数量大幅提升,使模型在复杂场景下误差降低28%(MAE)。
  • 可无缝接入现有模型,适用于多种计数任务,适合视觉细粒度识别研究者。

指代表达计数(REC)需要区分由细微文本线索描述的视觉上相似的对象。现有方法通过图像-文本对比学习视觉特征来区分正负指代表达,但该策略严重依赖精确的图文对齐,且可用负样本数量有限。我们提出在视觉嵌入空间中完全进行对比学习,以避免上述限制。为此,我们引入C-REX——一种简单有效的监督对比学习框架,通过对比同一图像内的视觉标记来学习细粒度视觉表示。通过将负样本从错误的指代表达改为图像中的其他视觉标记,C-REX极大增加了潜在负样本数量,提供更丰富稳定的监督信号。该设计增强了细粒度视觉区分能力,并在复杂开放世界计数设置下表现更好。C-REX作为即插即用模块,可无架构改动集成至任意现有REC模型。我们在三种不同REC架构中应用,实现最先进性能,MAE最高提升28%,RMSE提升24.5%。此外,我们的框架具有通用性,也可应用于无类别计数等任务,显著提升现有模型性能。

原文摘要 · Abstract (English)

Referring Expression Counting (REC) requires distinguishing visually similar objects described by fine-grained text cues. Existing methods tackle this via image-text contrastive learning of visual features which aims to distinguish visual features corresponding to positive and negative referring expressions. However, this strategy relies heavily on accurate image-text alignment and is constrained by the limited number of available negative expressions. We argue that these limitations can be avoided by performing contrastive learning entirely in the visual embedding space. To this end, we introduce C-REX, a simple but effective supervised contrastive learning framework that learns fine-grained visual representations by contrasting visual tokens within the same image. By shifting the negative samples from incorrect Referring Expressions to visual tokens from the image, C-REX vastly increases the number of potential negatives, providing richer and more stable supervision. This design leads to stronger fine-grained visual discrimination and better generalization across complex open-world counting settings. C-REX serves as a plug-in module that can be seamlessly applied to any existing REC model without architectural changes. We apply it in three different REC architectures and achieve state-of-the-art results improving by up to 28% in MAE and 24.5% in RMSE. Moreover, we show that our framework is versatile and general, and can be applied to other counting tasks like class-agnostic counting, improving the performance of existing models.

视觉计数对比学习细粒度识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。