用视觉语言模型实现视频场景分割,更懂剧情且能解释判断依据。
Scene-VLM: Multimodal Video Scene Segmentation via Vision-Language Models
- 联合处理图像、字幕和元数据,利用多模态信息进行序列推理。
- 在MovieNet上比之前最佳方法提升6点AP和13.7点F1,精度显著提高。
- 可生成自然语言理由解释分割边界,适合需要可解释性的应用。
将长视频分割为语义连贯场景是大规模视频理解的基础任务。现有基于编码器的方法受限于视觉中心偏差,孤立分类每个镜头,忽略时序依赖,缺乏叙事理解与可解释性。本文提出Scene-VLM,首个针对视频场景分割微调的视觉语言模型框架。该模型联合处理帧、字幕及可选元数据,实现跨连续镜头的多模态推理。预测按顺序生成,引入上下文聚焦窗口机制,确保每帧决策有足够时间上下文。我们还提出从VLM的标记级逻辑值中提取置信度分数的方法,实现可控的精确率-召回率权衡,此前仅限于编码器方法。此外,通过少量定向监督,模型可对边界决策生成连贯自然语言理由。该方法在标准场景分割基准上达到领先性能,在MovieNet上相比前驱方法提升6点AP和13.7点F1。
原文摘要 · Abstract (English)
Segmenting long-form videos into semantically coherent scenes is a fundamental task in large-scale video understanding. Existing encoder-based methods are limited by visual-centric biases, classify each shot in isolation without leveraging sequential dependencies, and lack both narrative understanding and explainability. In this paper, we present Scene-VLM, the first fine-tuned vision-language model (VLM) framework for video scene segmentation. Scene-VLM jointly processes visual and textual cues including frames, transcriptions, and optional metadata to enable multimodal reasoning across consecutive shots. The model generates predictions sequentially with causal dependencies among shots and introduces a context-focus window mechanism to ensure sufficient temporal context for each shot-level decision. In addition, we propose a scheme to extract confidence scores from the token-level logits of the VLM, enabling controllable precision-recall trade-offs that were previously limited to encoder-based methods. Furthermore, we demonstrate that our model can be aligned to generate coherent natural-language rationales for its boundary decisions through minimal targeted supervision. Our approach achieves state-of-the-art performance on standard scene segmentation benchmarks. On MovieNet, for example, Scene-VLM yields significant improvements of +6 AP and +13.7 F1 over the previous leading method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。