通过负样本拼接增强训练数据，提升图像指代分割的准确率
Finding NeMo: Negative-mined Mosaic Augmentation for Referring Image Segmentation
- 用CLIP筛选负样本拼成马赛克图，让模型学更难的区分任务
- 在多个数据集上显著提升难例场景下的分割准确率
- 适合研究视觉语言理解与数据增强的学者参考
指代图像分割（RIS）旨在根据文本描述从图像中分割出对应目标。该任务的难度受相似物体存在及表达复杂性影响，现有模型在简单与困难场景间表现差距明显。我们发现瓶颈在于训练数据，提出一种简单但有效的数据增强方法——负样本挖掘马赛克增强（NeMo）。该方法利用预训练多模态对齐模型（如CLIP）筛选三个负样本，与原图拼成马赛克图像，使样本更具挑战性。实验表明，适当调整难度至关重要，既不能过于模糊也不能太简单。增强后的数据促使模型更好识别相似视觉实体间的细微差异，并更准确理解完整表达以定位目标。在多个数据集和模型上均取得一致性能提升。
原文摘要 · Abstract (English)
Referring Image Segmentation is a comprehensive task to segment an object referred by a textual query from an image. In nature, the level of difficulty in this task is affected by the existence of similar objects and the complexity of the referring expression. Recent RIS models still show a significant performance gap between easy and hard scenarios. We pose that the bottleneck exists in the data, and propose a simple but powerful data augmentation method, Negative-mined Mosaic Augmentation (NeMo). This method augments a training image into a mosaic with three other negative images carefully curated by a pretrained multimodal alignment model, e.g., CLIP, to make the sample more challenging. We discover that it is critical to properly adjust the difficulty level, neither too ambiguous nor too trivial. The augmented training data encourages the RIS model to recognize subtle differences and relationships between similar visual entities and to concretely understand the whole expression to locate the right target better. Our approach shows consistent improvements on various datasets and models, verified by extensive experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。