仅用文本表达训练视频目标分割模型,省去标注成本。
Weakly-Supervised Referring Video Object Segmentation through Text Supervision

- 用大模型生成正负文本表达,实现纯文本监督。
- 通过伪掩码和时序排序约束提升分割精度。
- 适合无标注或弱标注场景,推动低成本视频理解。
指代式视频目标分割(RVOS)旨在根据文本描述定位视频中的目标实例。传统方法依赖昂贵的像素级掩码标注,而现有弱监督方法仍需边界框或点标注,仍较耗时。本文提出一种新方法WSRVOS,仅使用文本表达进行训练。首先设计对比性文本增强策略,利用多模态大语言模型生成正负文本表达;随后提取视频与文本的视觉和语言特征,通过双向跨模态选择与交互实现细粒度对齐。进一步提出实例感知的表达分类机制,优化模型区分正负表达的能力;引入正向预测融合策略生成高质量伪掩码,作为额外监督信号。最后设计时序片段排序约束,要求相邻帧掩码重叠度满足特定顺序。在四个公开数据集(A2D Sentences、J-HMDB Sentences、Ref-YouTube-VOS、Ref-DAVIS17)上实验表明,该方法性能优越。代码已开源。
原文摘要 · Abstract (English)
Referring video object segmentation (RVOS) aims to segment the target instance in a video, referred by a text expression. Conventional approaches are mostly supervised learning, requiring expensive pixel-level mask annotations. To tackle it, weakly-supervised RVOS has recently been proposed to replace mask annotations with bounding boxes or points, which are however still costly and labor-intensive. In this paper, we design a novel weakly-supervised RVOS method, namely WSRVOS, to train the model with only text expressions. Given an input video and the referring expression, we first design a contrastive referring expression augmentation scheme that leverages the captioning capabilities of a multimodal large language model to generate both positive and negative expressions. We extract visual and linguistic features from the input video and generated expressions, then perform bi-directional vision-language feature selection and interaction to enable fine-grained multimodal alignment. Next, we propose an instance-aware expression classification scheme to optimize the model in distinguishing positive from negative expressions. Also, we introduce a positive-prediction fusion strategy to generate high-quality pseudo-masks, which serve as additional supervision to the model. Last, we design a temporal segment ranking constraint such that the overlaps between mask predictions of temporally neighboring frames are required to conform to specific orders. Extensive experiments on four publicly available RVOS datasets, including A2D Sentences, J-HMDB Sentences, Ref-YouTube-VOS, and Ref-DAVIS17, demonstrate the superiority of our method. Code is available at https://github.com/viscom-tongji/WSRVOS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。