让大模型同时搞定视频对象定位与追踪,性能刷新纪录。
GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation
- 用关键帧提供全局信息,连续帧实现局部追踪,统一建模
- 在MeViS和Ref-Youtube-VOS上达到新最好结果,超越现有方法
- 适合需要高效视频分割的多模态大模型研究者使用
本文提出一种新型框架GLUS,利用多模态大语言模型(MLLM)进行指代式视频目标分割(RefVOS)。以往基于MLLM的方法常面临“指代”与“分割”的矛盾:要么专注少数关键帧的全局理解,要么依赖外部模块处理连续帧的局部追踪。GLUS创新性地将全局与局部一致性统一于单一模型中:通过稀疏的“上下文帧”提供全局信息,连续的“查询帧”实现局部追踪,并联合训练预训练的视频分割记忆库,同步捕获短时与长时时间信息。为提升有限上下文窗口内的信息效率,引入物体对比学习以区分难分假阳性目标,并设计自精炼框架识别关键帧并完成传播。综合上述方法,GLUS构建了一个简洁高效的基准,在MeViS与Ref-Youtube-VOS数据集上达到新最佳性能。
原文摘要 · Abstract (English)
This paper proposes a novel framework utilizing multi-modal large language models (MLLMs) for referring video object segmentation (RefVOS). Previous MLLM-based methods commonly struggle with the dilemma between "Ref" and "VOS": they either specialize in understanding a few key frames (global reasoning) or tracking objects on continuous frames (local reasoning), and rely on external VOS or frame selectors to mitigate the other end of the challenge. However, our framework GLUS shows that global and local consistency can be unified into a single video segmentation MLLM: a set of sparse "context frames" provides global information, while a stream of continuous "query frames" conducts local object tracking. This is further supported by jointly training the MLLM with a pre-trained VOS memory bank to simultaneously digest short-range and long-range temporal information. To improve the information efficiency within the limited context window of MLLMs, we introduce object contrastive learning to distinguish hard false-positive objects and a self-refined framework to identify crucial frames and perform propagation. By collectively integrating these insights, our GLUS delivers a simple yet effective baseline, achieving new state-of-the-art for MLLMs on the MeViS and Ref-Youtube-VOS benchmark. Our project page is at https://glus-video.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。