用工具增强生成电影梗概,让角色和剧情更一致可信。
MovieTeller: Tool-augmented Movie Synopsis with ID Consistent Progressive Abstraction
- 用现成人脸识别工具锚定角色身份和位置,确保事实准确
- 分阶段抽象生成,突破模型上下文长度限制
- 无需微调,直接调用现有模型,适合快速部署
随着数字娱乐内容爆炸式增长,自动化视频摘要在内容索引、个性化推荐和高效媒体存档中变得不可或缺。长视频(如电影、剧集)的自动梗概生成对现有视觉语言模型(VLMs)仍是重大挑战,因其在长时序上下文中常出现角色身份不一致、叙事断裂等问题。为此,我们提出 MovieTeller,一种通过工具增强的渐进式抽象框架,用于生成电影梗概。核心在于无需训练的、基于工具的、事实驱动的生成流程:首先调用专用人脸检测模型作为外部“工具”,建立角色身份与对应边界框的事实锚点;这些锚点被注入提示词,引导 VLM 推理,确保场景描述基于可验证的事实。此外,渐进式抽象管道将整部电影的摘要分解为多阶段处理,有效缓解当前 VLM 的上下文长度限制。实验表明,相比端到端基线,本方法在事实准确性、角色一致性及整体叙事连贯性上均有显著提升。
原文摘要 · Abstract (English)
With the explosive growth of digital entertainment, automated video summarization has become indispensable for applications such as content indexing, personalized recommendation, and efficient media archiving. Automatic synopsis generation for long-form videos, such as movies and TV series, presents a significant challenge for existing Vision-Language Models (VLMs). While proficient at single-image captioning, these general-purpose models often exhibit critical failures in long-duration contexts, primarily a lack of ID-consistent character identification and a fractured narrative coherence. To overcome these limitations, we propose MovieTeller, a novel framework for generating movie synopses via tool-augmented progressive abstraction. Our core contribution is a training-free, tool-augmented, fact-grounded generation process. Instead of requiring costly model fine-tuning, our framework directly leverages off-the-shelf models in a plug-and-play manner. We first invoke a specialized face recognition model as an external "tool" to establish Factual Groundings--precise character identities and their corresponding bounding boxes. These groundings are then injected into the prompt to steer the VLM's reasoning, ensuring the generated scene descriptions are anchored to verifiable facts. Furthermore, our progressive abstraction pipeline decomposes the summarization of a full-length movie into a multi-stage process, effectively mitigating the context length limitations of current VLMs. Experiments demonstrate that our approach yields significant improvements in factual accuracy, character consistency, and overall narrative coherence compared to end-to-end baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。