用动态时间标记提升视频分割精度，解决物体复杂运动与空间变化难题
The Devil is in Temporal Token: High Quality Video Reasoning Segmentation
- 引入帧级和时序级标记，结合大模型自回归能力捕捉时空特征
- 在ReVOS数据集上三项指标领先VISA 5.9%至12.5%，显著提升分割性能
- 适合需要高精度视频理解与推理的场景，如智能监控与自动驾驶
现有视频推理分割方法依赖单一关键帧或全视频的特殊标记，难以充分捕捉空间复杂性和帧间运动。为此，我们提出VRS-HQ，一种端到端视频推理分割方法，利用多模态大模型（MLLM）向分层标记注入丰富的时空特征。核心创新包括时间动态聚合（TDA）和标记驱动的关键帧选择（TKS）。设计帧级<SEG>与时序级<TAK>标记，借助MLLM的自回归学习有效捕获局部与全局信息。随后采用基于相似性的加权融合与帧选择策略，并使用SAM2进行关键帧分割与传播。为提升关键帧定位精度，TKS在推理阶段根据SAM2的遮挡得分过滤关键帧。VRS-HQ在ReVOS数据集上达到最优性能，相比VISA在三个子集上的J&F得分分别提升5.9%/12.5%/9.1%。结果表明该方法具备强大的时序推理与分割能力。代码与模型权重将公开发布。
原文摘要 · Abstract (English)
Existing methods for Video Reasoning Segmentation rely heavily on a single special token to represent the object in the keyframe or the entire video, inadequately capturing spatial complexity and inter-frame motion. To overcome these challenges, we propose VRS-HQ, an end-to-end video reasoning segmentation approach that leverages Multimodal Large Language Models (MLLMs) to inject rich spatiotemporal features into hierarchical tokens.Our key innovations include a Temporal Dynamic Aggregation (TDA) and a Token-driven Keyframe Selection (TKS). Specifically, we design frame-level <SEG> and temporal-level <TAK> tokens that utilize MLLM's autoregressive learning to effectively capture both local and global information. Subsequently, we apply a similarity-based weighted fusion and frame selection strategy, then utilize SAM2 to perform keyframe segmentation and propagation. To enhance keyframe localization accuracy, the TKS filters keyframes based on SAM2's occlusion scores during inference. VRS-HQ achieves state-of-the-art performance on ReVOS, surpassing VISA by 5.9%/12.5%/9.1% in J&F scores across the three subsets. These results highlight the strong temporal reasoning and segmentation capabilities of our method. Code and model weights will be released at VRS-HQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。