arXiv:2501.14607cs.CV2025-01ICCV被引 38

用视觉定位基础模型提升视频目标分割精度,支持实时推理。

ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations

  • 基于位置预测的可变形掩码解码器,逐步引导像素级分割。
  • 在Ref-YouTube-VOS上实现3.9%的性能提升,达到51帧/秒实时速度。
  • 适合需要高精度视频语义理解与快速响应的应用场景。

指代式视频对象分割(RVOS)旨在根据文本描述对视频中目标对象进行全时序分割。该任务需融合深层视觉-语言理解、像素级密集预测及时空推理能力,挑战巨大。尽管近年进展显著,现有方法在综合考量上述三方面时仍存在明显差距。本文提出ReferDINO,一种强性能的RVOS模型:继承基础视觉定位模型的区域级跨模态对齐能力,并进一步引入像素级密集感知与跨模态时空推理机制。其核心包含两个组件:1)基于定位引导的可变形掩码解码器,通过可微形变机制利用位置预测逐步指导掩码生成;2)物体一致性的时序增强模块,将预训练的时间变化文本特征注入帧间交互,捕捉对象感知的动态变化。此外,设计置信度感知查询剪枝策略,在不牺牲性能前提下加速对象解码。在五个基准数据集上的大量实验表明,ReferDINO显著优于先前方法(如在Ref-YouTube-VOS上提升3.9%( J&F )),并实现51帧/秒的实时推理速度。

原文摘要 · Abstract (English)

Referring video object segmentation (RVOS) aims to segment target objects throughout a video based on a text description. This is challenging as it involves deep vision-language understanding, pixel-level dense prediction and spatiotemporal reasoning. Despite notable progress in recent years, existing methods still exhibit a noticeable gap when considering all these aspects. In this work, we propose \textbf{ReferDINO}, a strong RVOS model that inherits region-level vision-language alignment from foundational visual grounding models, and is further endowed with pixel-level dense perception and cross-modal spatiotemporal reasoning. In detail, ReferDINO integrates two key components: 1) a grounding-guided deformable mask decoder that utilizes location prediction to progressively guide mask prediction through differentiable deformation mechanisms; 2) an object-consistent temporal enhancer that injects pretrained time-varying text features into inter-frame interaction to capture object-aware dynamic changes. Moreover, a confidence-aware query pruning strategy is designed to accelerate object decoding without compromising model performance. Extensive experimental results on five benchmarks demonstrate that our ReferDINO significantly outperforms previous methods (e.g., +3.9% (\mathcal{J}&\mathcal{F}) on Ref-YouTube-VOS) with real-time inference speed (51 FPS).

视频分割视觉定位实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。