arXiv:2409.19603cs.CVcs.AI2024-09NeurIPS被引 143

用一句话指令,自动分割并追踪视频中的物体。

One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos

论文配图:One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos
图 1 · 摘自论文原文
  • 结合大模型推理与SAM,实现跨帧语义一致性分割。
  • 提出<TRK>标记和稀疏密集采样,提升时序理解与效率。
  • 在复杂推理与追踪任务上表现优异,适合多场景通用分割。

我们提出VideoLISA,一种基于视频的多模态大语言模型,用于解决语言指令驱动的视频推理分割问题。借助大语言模型的推理能力与世界知识,并融合Segment Anything Model(SAM),VideoLISA 能根据语言指令生成时序一致的视频分割掩码。现有图像类方法如LISA在视频任务中受限于额外的时间维度,难以实现帧间动态理解与一致分割。VideoLISA通过将稀疏密集采样策略融入视频-大语言模型,在计算约束下平衡时序上下文与空间细节。此外,我们提出一种一令牌全帧分割方法(One-Token-Seg-All),利用特制的<TRK>标记,实现跨多帧的对象分割与追踪。在多个基准测试上,包括我们新提出的ReasonVOS基准,VideoLISA 在涉及复杂推理、时间理解与对象追踪的视频分割任务中展现出卓越性能。尽管专为视频优化,VideoLISA 在图像分割任务上也表现出良好泛化能力,揭示其作为语言指令驱动对象分割统一基础模型的潜力。代码与模型将开源。

原文摘要 · Abstract (English)

We introduce VideoLISA, a video-based multimodal large language model designed to tackle the problem of language-instructed reasoning segmentation in videos. Leveraging the reasoning capabilities and world knowledge of large language models, and augmented by the Segment Anything Model, VideoLISA generates temporally consistent segmentation masks in videos based on language instructions. Existing image-based methods, such as LISA, struggle with video tasks due to the additional temporal dimension, which requires temporal dynamic understanding and consistent segmentation across frames. VideoLISA addresses these challenges by integrating a Sparse Dense Sampling strategy into the video-LLM, which balances temporal context and spatial detail within computational constraints. Additionally, we propose a One-Token-Seg-All approach using a specially designed <TRK> token, enabling the model to segment and track objects across multiple frames. Extensive evaluations on diverse benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate VideoLISA's superior performance in video object segmentation tasks involving complex reasoning, temporal understanding, and object tracking. While optimized for videos, VideoLISA also shows promising generalization to image segmentation, revealing its potential as a unified foundation model for language-instructed object segmentation. Code and model will be available at: https://github.com/showlab/VideoLISA.

视频分割大模型语言指令对象追踪

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。