arXiv:2505.18686cs.CV2025-05CVPR被引 12

弱监督下联合学习指代表达理解与分割,提升模型泛化能力。

WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation

  • 双分支架构融合指代理解和分割任务,通过对比学习实现协同优化。
  • 在RefCOCO上分别提升3.91%和13.11%,半监督设置下提升超8%。
  • 适合弱监督视觉语言理解场景,尤其对标注稀缺任务有显著优势。

弱监督指代表达理解(WREC)与分割(WRES)旨在利用图像-文本对等弱监督信号学习对象定位。传统方法将两者独立建模,本文提出WeakMCN,一种多任务协作网络,采用双分支结构联合建模。其中WREC分支基于锚点对比学习,并作为教师指导WRES分支。创新性地引入动态视觉特征增强(DVFE)与协作一致性模块(CCM),分别从特征融合与优化角度促进跨任务协同。在RefCOCO、RefCOCO+和RefCOCOg三个主流基准上,实验表明WeakMCN性能优于现有单任务方法,如在RefCOCO上分别提升3.91%(WREC)和13.11%(WRES)。此外,在半监督设置下,1%数据训练时,半监督REC和RES分别提升8.94%和7.71%。代码已开源。

原文摘要 · Abstract (English)

Weakly supervised referring expression comprehension(WREC) and segmentation(WRES) aim to learn object grounding based on a given expression using weak supervision signals like image-text pairs. While these tasks have traditionally been modeled separately, we argue that they can benefit from joint learning in a multi-task framework. To this end, we propose WeakMCN, a novel multi-task collaborative network that effectively combines WREC and WRES with a dual-branch architecture. Specifically, the WREC branch is formulated as anchor-based contrastive learning, which also acts as a teacher to supervise the WRES branch. In WeakMCN, we propose two innovative designs to facilitate multi-task collaboration, namely Dynamic Visual Feature Enhancement(DVFE) and Collaborative Consistency Module(CCM). DVFE dynamically combines various pre-trained visual knowledge to meet different task requirements, while CCM promotes cross-task consistency from the perspective of optimization. Extensive experimental results on three popular REC and RES benchmarks, i.e., RefCOCO, RefCOCO+, and RefCOCOg, consistently demonstrate performance gains of WeakMCN over state-of-the-art single-task alternatives, e.g., up to 3.91% and 13.11% on RefCOCO for WREC and WRES tasks, respectively. Furthermore, experiments also validate the strong generalization ability of WeakMCN in both semi-supervised REC and RES settings against existing methods, e.g., +8.94% for semi-REC and +7.71% for semi-RES on 1% RefCOCO. The code is publicly available at https://github.com/MRUIL/WeakMCN.

弱监督指代理解多任务学习图像分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。