arXiv:2506.22880cs.CVcs.AI2025-06被引 2

分离视觉与语义特征,提升视频分割与定位精度

Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Grounder

  • 将文本与视觉特征通过线性投影解耦,避免信息混淆
  • 在多个任务上达到当前最佳性能,包括视频问答
  • 适合需要精准语义理解的视频分析研究者

现有视频分割与定位方法(如Sa2VA)直接融合特征,导致动态视觉信息与静态语义纠缠,降低分割精度。为此,我们提出DeSa2VA,一种结合文本预训练与线性解耦模块的增强提示方案,以解决SAM-2中的信息处理局限。首先,设计一种预训练范式,将文本真值标签转换为点级提示,并生成对应文本掩码,通过混合损失函数优化,强化模型语义定位能力。其次,利用线性投影将大语言模型生成的隐藏状态解耦至独立的文本与视觉特征子空间。最后,采用动态掩码融合策略,通过预测文本/视觉掩码与真实标注的三重监督,协同整合解耦特征。大量实验表明,该方法在图像分割、图像问答、视频分割和视频问答等多个任务上均达到领先水平。代码已公开于https://github.com/longmalongma/DeSa2VA。

原文摘要 · Abstract (English)

Existing video segmenter and grounder approaches, exemplified by Sa2VA, directly fuse features within segmentation models. This often results in an undesirable entanglement of dynamic visual information and static semantics, thereby degrading segmentation accuracy. To systematically mitigate this issue, we propose DeSa2VA, a decoupling-enhanced prompting scheme integrating text pre-training and a linear decoupling module to address the information processing limitations inherent in SAM-2. Specifically, first, we devise a pre-training paradigm that converts textual ground-truth labels into point-level prompts while generating corresponding text masks. These masks are refined through a hybrid loss function to strengthen the model's semantic grounding capabilities. Next, we employ linear projection to disentangle hidden states that generated by a large language model into distinct textual and visual feature subspaces. Finally, a dynamic mask fusion strategy synergistically combines these decoupled features through triple supervision from predicted text/visual masks and ground-truth annotations. Extensive experiments demonstrate state-of-the-art performance across diverse tasks, including image segmentation, image question answering, video segmentation, and video question answering. Our codes are available at https://github.com/longmalongma/DeSa2VA.

视频分割特征解耦多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。