用80亿参数模型实现长篇影视视频的音画脚本自动生成
OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video
- 构建分场景层次化脚本生成框架，融合音画信息
- 在时序定位和语义准确性上超越大模型，接近闭源顶尖水平
- 适合影视自动化创作、内容分析与智能编剧研究者
当前多模态大模型在短视频理解方面表现卓越，但将长篇电影视频转化为详细、时序精准的脚本仍是重大挑战。本文提出视频转脚本（V2S）新任务，旨在生成包含角色动作、对白、表情和音频提示的分场景层次化脚本。为此，我们构建首个由人工标注的基准数据集，并提出基于时间分段的层次化评估框架。此外，我们提出80亿参数的全模态（音画）语言模型OmniScript，采用渐进式训练流程：先通过思维链监督微调进行情节与角色推理，再使用时序分割奖励进行强化学习。大量实验表明，尽管参数量小，OmniScript显著优于更大规模开源模型，在时序定位和多领域语义准确率上达到与顶级闭源模型Gemini 3-Pro相当的水平。
原文摘要 · Abstract (English)
Current multimodal large language models (MLLMs) have demonstrated remarkable capabilities in short-form video understanding, yet translating long-form cinematic videos into detailed, temporally grounded scripts remains a significant challenge. This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts encompassing character actions, dialogues, expressions, and audio cues. To facilitate this, we construct a first-of-its-kind human-annotated benchmark and propose a temporally-aware hierarchical evaluation framework. Furthermore, we present OmniScript, an 8B-parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension. OmniScript is trained via a progressive pipeline that leverages chain-of-thought supervised fine-tuning for plot and character reasoning, followed by reinforcement learning using temporally segmented rewards. Extensive experiments demonstrate that despite its parameter efficiency, OmniScript significantly outperforms larger open-source models and achieves performance comparable to state-of-the-art proprietary models, including Gemini 3-Pro, in both temporal localization and multi-field semantic accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。