提出遮蔽增强方法MaskRIS，提升图像描述分割的鲁棒性
MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image Segmentation
- 联合图像与文本遮蔽，结合语义畸变感知学习
- 在三个基准数据集上达到新最优性能，显著提升抗遮挡能力
- 适用于弱监督与全监督场景，通用性强
指代图像分割（RIS）是一项视觉-语言任务，需根据自然语言描述定位并分割图像中的目标。尽管先前研究聚焦于视觉与语言特征对齐，但训练技术如数据增强仍被忽视。本文提出一种新型训练框架MaskRIS，通过图像与文本遮蔽策略，并引入畸变感知上下文学习（DCL），有效利用遮蔽带来的信息冗余。实验表明，该方法可显著提升模型对遮挡、信息缺失及语言复杂性的鲁棒性，在完全监督与弱监督设置下均优于现有方法，在RefCOCO、RefCOCO+和RefCOCOg数据集上实现新最佳性能。代码已开源。
原文摘要 · Abstract (English)
Referring Image Segmentation (RIS) is an advanced vision-language task that involves identifying and segmenting objects within an image as described by free-form text descriptions. While previous studies focused on aligning visual and language features, exploring training techniques, such as data augmentation, remains underexplored. In this work, we explore effective data augmentation for RIS and propose a novel training framework called Masked Referring Image Segmentation (MaskRIS). We observe that the conventional image augmentations fall short of RIS, leading to performance degradation, while simple random masking significantly enhances the performance of RIS. MaskRIS uses both image and text masking, followed by Distortion-aware Contextual Learning (DCL) to fully exploit the benefits of the masking strategy. This approach can improve the model's robustness to occlusions, incomplete information, and various linguistic complexities, resulting in a significant performance improvement. Experiments demonstrate that MaskRIS can easily be applied to various RIS models, outperforming existing methods in both fully supervised and weakly supervised settings. Finally, MaskRIS achieves new state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg datasets. Code is available at https://github.com/naver-ai/maskris.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。