提出事件推理框架，让模型更懂视频中物体的动态指代关系。
EventRR: Event Referential Reasoning for Referring Video Object Segmentation
- 将指代推理拆解为对象摘要与事件结构化推理两部分。
- 在四个数据集上显著超越当前最佳方法，mAP 提升超过 5%。
- 适合研究视频理解、跨模态指代消解的开发者与学者。
指代视频目标分割（RVOS）旨在根据语言表达分割视频中的目标对象。现有方法将指代表达视为无结构序列，忽视其对指代推理至关重要的语义结构。与仅关注物体属性和物体间关系的图像指代表达不同，视频指代表达还包含事件属性和事件间的时序关系，这一复杂性挑战了传统图像结构化推理方法。本文提出事件指代推理（EventRR）框架，将 RVOS 分解为对象摘要与指代推理两个阶段。摘要阶段将每帧压缩为一组瓶颈标记，并在视频级汇总步骤中高效交换全局跨模态时序上下文。推理阶段将视频指代表达的语义事件结构提取为高度表达性的指代事件图（REG），一种单根有向无环图。通过引导 REG 的拓扑遍历，提出时序概念-角色推理（TCRR），从叶节点到根节点逐步累积每个时序查询的指代得分。每次推理可解释为基于 REG 中概念-角色关系的问题-答案对。在四个主流基准数据集上的大量实验表明，EventRR 在定量和定性上均优于现有最先进方法。代码已开源。
原文摘要 · Abstract (English)
Referring Video Object Segmentation (RVOS) aims to segment out the object in a video referred by an expression. Current RVOS methods view referring expressions as unstructured sequences, neglecting their crucial semantic structure essential for referent reasoning. Besides, in contrast to image-referring expressions whose semantics focus only on object attributes and object-object relations, video-referring expressions also encompass event attributes and event-event temporal relations. This complexity challenges traditional structured reasoning image approaches. In this paper, we propose the Event Referential Reasoning (EventRR) framework. EventRR decouples RVOS into object summarization part and referent reasoning part. The summarization phase begins by summarizing each frame into a set of bottleneck tokens, which are then efficiently aggregated in the video-level summarization step to exchange the global cross-modal temporal context. For reasoning part, EventRR extracts semantic eventful structure of a video-referring expression into highly expressive Referential Event Graph (REG), which is a single-rooted directed acyclic graph. Guided by topological traversal of REG, we propose Temporal Concept-Role Reasoning (TCRR) to accumulate the referring score of each temporal query from REG leaf nodes to root node. Each reasoning step can be interpreted as a question-answer pair derived from the concept-role relations in REG. Extensive experiments across four widely recognized benchmark datasets, show that EventRR quantitatively and qualitatively outperforms state-of-the-art RVOS methods. Code is available at https://github.com/bio-mlhui/EventRR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。