通过关键时刻采样提升视频目标分割精度
MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understanding
- 用专用标记定位关键时刻，无需外部时间编码
- 密集采样重要时刻，稀疏处理无关帧，兼顾运动与上下文
- 双向锚点更新提升追踪稳定性，适合长视频分割任务
指代视频目标分割（RefVOS）旨在根据自然语言描述分割视频中的目标对象，需具备时间推理与细粒度视觉理解能力。现有基于大模型的方法通常依赖手工启发式或外部关键帧模型，前者忽略重要时间线索，后者增加系统复杂性。为此，我们提出统一框架，联合优化时间句段定位（TSG）与RefVOS，天然融入关键时刻定位能力。训练时引入新TSG范式，使用专用["FIND"]标记通过时间标记相似性匹配识别关键时刻，避免外部时间戳编码。推理时设计关键时刻采样（MCS）策略，密集采样信息丰富时刻，稀疏采样非必要帧，保留运动细节与全局上下文。为增强跟踪稳定性，提出双向锚点更新传播（BAP），以最相关时刻为起点进行高质量掩码初始化，并在采样点动态更新，缓解误差累积。代码与模型将公开于：https://github.com/Dmmm1997/MomentSeg
原文摘要 · Abstract (English)
Referring Video Object Segmentation (RefVOS) seeks to segment target objects in videos guided by natural language descriptions, demanding both temporal reasoning and fine-grained visual comprehension. Existing sampling strategies for LLM-based approaches typically rely on either handcrafted heuristics or external keyframe models. The former often overlooks essential temporal cues, while the latter increases system complexity. To address this, we propose a unified framework that jointly optimizes Temporal Sentence Grounding (TSG) and RefVOS, naturally incorporating key moment grounding capability. During training, we introduce a novel TSG paradigm that employs a dedicated \texttt{[FIND]} token for key moment identification through temporal token similarity matching, thereby avoiding the need for external timestamp encodings. For inference, we design a Moment-Centric Sampling (MCS) strategy that densely samples informative moments while sparsely sampling non-essential frames, preserving both motion details and global context. To further enhance tracking stability, we develop Bidirectional Anchor-updated Propagation (BAP), which leverages the most relevant moment as start point for high-quality mask initialization and dynamically updates at sampled points to mitigate accumulated errors. Code and model will be available at: https://github.com/Dmmm1997/MomentSeg
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。