提升图像中基于动作描述的物体分割准确率
Towards Motion-aware Referring Image Segmentation
- 通过提取运动关键词增强数据，让模型学习更多动作表达
- 在跨模态融合嵌入上使用对比学习，显著提升运动类查询效果
- 新构建M-Bench基准，专测动作驱动的语义分割能力
指代图像分割（RIS）需根据文本描述定位图像中的对象。我们发现现有方法在运动相关查询上的表现远低于外观类查询。为此，我们提出一种高效的图像增强方案，从原始描述中提取以运动为中心的短语，使模型在无需额外标注的情况下接触更多运动表达。其次，由于同一对象在不同语境下描述方式不同，我们提出多模态径向对比学习（MRaCL），在图像-文本融合嵌入空间而非单模态表示上进行。为全面评估，我们引入一个聚焦运动中心查询的新测试集，并构建新基准M-Bench，其中对象主要通过动作区分。大量实验表明，该方法在多个RIS模型上显著提升运动类查询性能，同时保持对外观类描述的竞争力。代码已开源。
原文摘要 · Abstract (English)
Referring Image Segmentation (RIS) requires identifying objects from images based on textual descriptions. We observe that existing methods significantly underperform on motion-related queries compared to appearance-based ones. To address this, we first introduce an efficient data augmentation scheme that extracts motion-centric phrases from original captions, exposing models to more motion expressions without additional annotations. Second, since the same object can be described differently depending on the context, we propose Multimodal Radial Contrastive Learning (MRaCL), performed on fused image-text embeddings rather than unimodal representations. For comprehensive evaluation, we introduce a new test split focusing on motion-centric queries, and introduce a new benchmark called M-Bench, where objects are distinguished primarily by actions. Extensive experiments show our method substantially improves performance on motion-centric queries across multiple RIS models, maintaining competitive results on appearance-based descriptions. Codes are available at https://github.com/snuviplab/MRaCL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。