用语言对齐选轨迹，提升视频目标分割精度
Referring Video Object Segmentation via Language-aligned Track Selection
- 利用SAM2对象令牌构建视频级紧凑表示
- 通过交并比伪标签实现视觉与语言对齐
- 在MeViS数据集上达到当前最优效果
指代式视频目标分割（RVOS）需根据自然语言描述追踪并分割视频中的目标，要求同时具备复杂运动理解能力及视觉表征与语言描述的对齐。近期提出的Segment Anything Model 2（SAM2）因其能生成跨帧连贯的分割掩码轨迹，并在对象令牌中蕴含固有的时空对象性，成为潜在解决方案。本文提出SOLA（Selection by Object Language Alignment）框架，利用SAM2对象令牌作为紧凑的视频级对象表示，并通过轻量级轨迹选择模块将其与语言特征对齐。为有效促进该对齐，我们设计了一种基于交并比（IoU）的伪标签策略，弥合了SAM2表征与语言特征之间的模态差距。大量实验表明，SOLA在MeViS数据集上取得当前最优性能，证明其是解决RVOS的有效方案。
原文摘要 · Abstract (English)
Referring video object segmentation (RVOS) requires tracking and segmenting an object throughout a video according to a given natural language expression, demanding both complex motion understanding and the alignment of visual representations with language descriptions. Given these challenges, the recently proposed Segment Anything Model 2 (SAM2) emerges as a potential candidate due to its ability to generate coherent segmentation mask tracks across video frames, and provide an inherent spatio-temporal objectness in its object token representations. In this paper, we introduce SOLA (Selection by Object Language Alignment), a novel framework that leverages SAM2 object tokens as compact video-level object representations, which are aligned with language features through a lightweight track selection module. To effectively facilitate this alignment, we propose an IoU-based pseudo-labeling strategy, which bridges the modality gap between SAM2 representations with language features. Extensive experiments show that SOLA achieves state-of-the-art performance on the MeViS dataset and demonstrate that SOLA offers an effective solution for RVOS. Our project page is available at: https://cvlab-kaist.github.io/SOLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。