让模型同时分割互动中的动作者和对象,更懂视频里的因果关系。
InterRVOS: Interaction-aware Referring Video Object Segmentation
- 提出交互感知的分割任务,区分动作者与目标对象角色
- 构建127K规模数据集,含演员与目标独立标注掩码
- 基于多模态大模型设计新架构,显著提升角色识别准确率
指代视频对象分割(RVOS)旨在根据自然语言描述分割视频中的目标对象。然而,现有方法通常只关注被指称的对象(通常是动作执行者),即使描述明确涉及多个具有不同角色的物体之间的互动。例如,“A抛B”暗示方向性交互,但传统方法仅分割动作者(A),忽略参与的其他目标对象(B)。本文提出交互感知指代视频对象分割(InterRVOS),一项新任务,聚焦于对交互关系的建模,要求模型分别分割动作者和目标对象,反映其在交互中的不对称角色。该任务形式化促进了对对象关系的细粒度理解,因为许多视频事件由这些关系定义而非单一对象。为此,我们提出了新的评估协议,分别评估动作者与目标的分割性能,实现更精确的模型能力评估。我们还构建了包含超过127,000条自动标注表达式的大规模数据集InterRVOS-127K,其中交互表达式带有演员与目标的独立掩码标注。此外,我们开发了基于多模态大模型的ReVIOSa架构,引入交互感知特殊标记并采用注意力掩码损失,增强角色特异性分割。大量实验表明,ReVIOSa不仅在所提出的InterRVOS-127K评估集上优于现有基线,还在标准RVOS基准上表现强劲。
原文摘要 · Abstract (English)
Referring video object segmentation (RVOS) aims to segment objects in a video described by a natural language expression. However, most existing approaches focus on segmenting only the referred object (typically the actor), even when the expression clearly describes an interaction involving multiple objects with distinct roles. For instance, "A throwing B" implies a directional interaction, but standard RVOS segments only the actor (A), neglecting other involved target objects (B). In this paper, we introduce Interaction-aware Referring Video Object Segmentation (InterRVOS), a novel task that focuses on the modeling of interactions. It requires the model to segment the actor and target objects separately, reflecting their asymmetric roles in an interaction. This task formulation enables fine-grained understanding of object relationships, as many video events are defined by such relationships rather than individual objects. To support this task, we propose a new evaluation protocol that separately evaluates actor and target segmentation, enabling more accurate assessment of the model's ability to distinguish and segment actor and target roles. We also present InterRVOS-127K, a large-scale dataset with over 127K automatically annotated expressions, including interaction expressions annotated with distinct masks for actor and target objects. Furthermore, we develop ReVIOSa, an MLLM-based architecture that introduces interaction-aware special tokens and leverages an attention mask loss to enhance role-specific segmentation. Extensive experiments show that ReVIOSa not only outperforms existing baselines on our proposed InterRVOS-127K evaluation set, but also achieves strong performance on standard RVOS benchmarks. Our project page is available at: https://cvlab-kaist.github.io/InterRVOS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。