提出视频中像素级目标定位新任务,支持全时段精准分割。
Towards Visual Query Segmentation in the Wild
- 设计多阶段框架,利用目标与背景线索动态更新记忆
- 在4111段视频上实现超越现有方法的定位精度
- 适合需要完整目标追踪的智能监控与视频分析场景
本文提出视觉查询分割(VQS),一种全新的视觉查询定位范式,旨在给定外部视觉查询时,对未剪辑视频中目标对象的所有像素级出现进行分割。相比仅定位目标最后一次出现的边界框方式,VQS实现更全面(所有出现)和更精确(像素级掩码)的定位,更具实际应用价值。为此,我们构建了首个专为VQS设计的大规模基准VQS-4K,包含4,111段视频、超过130万帧,覆盖222个物体类别。每段视频配以外部帧定义的视觉查询及其目标掩码,并标注对应时空掩码片段。所有数据均经人工精细标注与迭代修正。此外,我们提出VQ-SAM方法,通过扩展SAM 2,融合视频中的目标特异性与背景干扰线索,采用自适应记忆生成模块,实现记忆的渐进演化,在VQS-4K上表现优异,显著优于现有方法。本工作推动了视觉查询定位从局部到全局、从粗略到精细的演进,相关代码与数据将公开。
原文摘要 · Abstract (English)
In this paper, we introduce visual query segmentation (VQS), a new paradigm of visual query localization (VQL) that aims to segment all pixel-level occurrences of an object of interest in an untrimmed video, given an external visual query. Compared to existing VQL locating only the last appearance of a target using bounding boxes, VQS enables more comprehensive (i.e., all object occurrences) and precise (i.e., pixel-level masks) localization, making it more practical for real-world scenarios. To foster research on this task, we present VQS-4K, a large-scale benchmark dedicated to VQS. Specifically, VQS-4K contains 4,111 videos with more than 1.3 million frames and covers a diverse set of 222 object categories. Each video is paired with a visual query defined by a frame outside the search video and its target mask, and annotated with spatial-temporal masklets corresponding to the queried target. To ensure high quality, all videos in VQS-4K are manually labeled with meticulous inspection and iterative refinement. To the best of our knowledge, VQS-4K is the first benchmark specifically designed for VQS. Furthermore, to stimulate future research, we present a simple yet effective method, named VQ-SAM, which extends SAM 2 by leveraging target-specific and background distractor cues from the video to progressively evolve the memory through a novel multi-stage framework with an adaptive memory generation (AMG) module for VQS, significantly improving the performance. In our extensive experiments on VQS-4K, VQ-SAM achieves promising results and surpasses all existing approaches, demonstrating its effectiveness. With the proposed VQS-4K and VQ-SAM, we expect to go beyond the current VQL paradigm and inspire more future research and practical applications on VQS. Our benchmark, code, and results will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。