端到端视频字幕提取框架,一步生成字幕与时间戳。
EVE: Towards End-to-End Video Subtitle Extraction with Vision-Language Models
- 设计双分支时空字幕显著模块,融合跨帧关联与空间语义。
- 在250万条视频数据上训练,实现字幕与时间戳同步输出。
- 首个专用字幕提取数据集ViSa,支持多语言评估。
视频字幕在短视频和电影中至关重要,不仅辅助模型理解内容,还支撑翻译与检索等应用。现有方法多采用多阶段框架,错误逐级累积,且因按帧处理而忽略时间依赖。尽管大型视觉语言模型具备强大光学字符识别能力,但准确预测字幕时间戳仍具挑战。为此,我们提出基于大视觉语言模型的端到端视频字幕提取框架EVE,可同时输出字幕文本及其时间戳。核心是引入双分支时空字幕显著(S³)模块,作为视觉语言模型的适配器,仅用少量标记即可表征字幕相关内容并建模帧间相关性。其中,空间语义上下文聚合分支捕捉全局空间语义信息,时序字幕标记查询分支显式查询字幕相关标记并考虑帧间时序关联。该模块保留的少量标记输入语言模型,直接生成字幕及对应时间戳。此外,我们构建了首个面向视频字幕提取的大规模数据集ViSa,包含超过250万条带时间戳与双语标注的视频,为社区提供结构化训练与评估基准。
原文摘要 · Abstract (English)
Video subtitles play a crucial role in short videos and movies, as they not only help models better understand video content but also support applications such as video translation and content retrieval. Existing video subtitle extraction methods typically rely on multi-stage frameworks, where errors accumulate across stages and temporal dependencies are underutilized due to frame-wise processing. Moreover, although some Large Vision-Language Models (LVLMs) possess strong OCR capabilities, predicting accurate timestamps for subtitle texts remains challenging. To this end, we propose an End-to-end Video subtitle Extraction framework based on LVLMs, named EVE, which can output subtitles and their timestamps simultaneously. Specifically, we introduce a dual-branch Spatiotemporal Subtitle-Salient (S\textsuperscript{3}) Module that serves as an adapter for LVLMs, capable of representing subtitle-related content and considering inter-frame correlations using only a small number of tokens. Within this module, the Spatial Semantic Context Aggregate branch aggregates high-level global semantics to provide spatial visual contextual information, while the Temporal Subtitle Token Query branch explicitly queries subtitle-relevant tokens while considering temporal correlation across frames. The small number of tokens retained by the S\textsuperscript{3} module are fed to the language model, which then directly outputs the subtitle text along with its timestamps. Furthermore, we construct the first large-scale dataset dedicated to video subtitle extraction, ViSa, containing over 2.5M videos with timestamped and bilingual annotation, thereby providing the community with a well-organized training and evaluation benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。