让模型根据语言指令判断该画可见部分还是完整物体，实现意图驱动的分割。
R2SM: Referring and Reasoning for Selective Masks
- 根据自然语言提示判断应生成可见遮挡部分或完整形状的掩码。
- 构建了包含可见与完整掩码标注的R2SM数据集，支持细粒度训练与评估。
- 适合关注多模态推理与用户意图理解的研究者使用。
我们提出新任务Referring and Reasoning for Selective Masks（R2SM），在文本引导分割基础上引入由用户意图驱动的掩码类型选择。该任务要求视觉语言模型仅根据自然语言提示判断应生成可见掩码（modal）还是完整掩码（amodal）。为此，我们构建了R2SM数据集，通过扩充COCOA-cls、D2SA和MUVA的标注获得。数据集包含可见与完整形状的文本查询，每条对应真实掩码，支持模型微调与评估。例如，若提示明确要求部分遮挡物体的完整形状，则模型需输出包含被遮挡区域的amodal掩码；若未提及隐藏区域，则生成标准modal掩码。R2SM基准为推进多模态推理与意图感知分割提供了具有挑战性且富有洞察力的测试平台。
原文摘要 · Abstract (English)
We introduce a new task, Referring and Reasoning for Selective Masks (R2SM), which extends text-guided segmentation by incorporating mask-type selection driven by user intent. This task challenges vision-language models to determine whether to generate a modal (visible) or amodal (complete) segmentation mask based solely on natural language prompts. To support the R2SM task, we present the R2SM dataset, constructed by augmenting annotations of COCOA-cls, D2SA, and MUVA. The R2SM dataset consists of both modal and amodal text queries, each paired with the corresponding ground-truth mask, enabling model finetuning and evaluation for the ability to segment images as per user intent. Specifically, the task requires the model to interpret whether a given prompt refers to only the visible part of an object or to its complete shape, including occluded regions, and then produce the appropriate segmentation. For example, if a prompt explicitly requests the whole shape of a partially hidden object, the model is expected to output an amodal mask that completes the occluded parts. In contrast, prompts without explicit mention of hidden regions should generate standard modal masks. The R2SM benchmark provides a challenging and insightful testbed for advancing research in multimodal reasoning and intent-aware segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。