无需参数设定，自动划分长视频场景边界。
Parameter-free Video Segmentation for Vision and Language Understanding
- 基于最小描述长度原则与动态规划，自动确定视频分块位置。
- 在长视频中检测场景边界更准确，优于现有方法。
- 适用于视频摘要与问答任务，提升下游性能。
创意视频内容的激增推动了语言模型处理视频输入以实现多模态理解的需求。然而，端到端模型因规模和复杂性难以处理长视频。一种有效替代方案是将视频分割为较小片段分别处理，这促使我们研究如何确定分块边界。本文提出一种基于最小描述长度原理并结合动态规划搜索的视频分割算法，完全无需参数设定，仅需特征向量输入，不需预设阈值或片段数量与大小。实验表明，该算法生成的断点能更准确地逼近长视频中的真实场景边界，即使对比那些已知真实场景数的方法也表现更优。随后，我们在长视频摘要和检索增强型视频问答两个任务中验证该算法，结果表明，使用本算法生成的场景分段显著提升了下游任务性能。
原文摘要 · Abstract (English)
The proliferation of creative video content has driven demand for adapting language models to handle video input and enable multimodal understanding. However, end-to-end models struggle to process long videos due to their size and complexity. An effective alternative is to divide them into smaller chunks to be processed separately, and this motivates a method for choosing where the chunk boundaries should be. In this paper, we propose an algorithm for segmenting videos into contiguous chunks, based on the minimum description length principle, coupled with a dynamic programming search. The algorithm is entirely parameter-free, given feature vectors, not requiring a set threshold or the number or size of chunks to be specified. We show empirically that the breakpoints it produces more accurately approximate scene boundaries in long videos, compared with existing methods for scene detection, even when such methods have access to the true number of scenes. We then showcase this algorithm in two tasks: long video summarization, and retrieval-augmented video question answering. In both cases, scene breaks produced by our algorithm lead to better downstream performance than existing methods for video segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。