让视频字幕生成并行化,提速同时更准地定位事件
Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning

- 重构因果依赖图,弱相关事件间可并行生成
- 在多个数据集上实现更快生成且准确率提升
- 适合需要高效高精度视频理解的场景
密集视频字幕旨在生成时间对齐的视频事件描述,对事件级视频理解与生成均有帮助。当前自回归视频大模型虽具强大生成与跨模态建模能力,但逐令牌生成严重限制推理效率,随视频长度和事件密度增加尤为明显。本文提出一种并行化自回归框架,不仅提升生成效率,还增强时间对齐字幕性能。核心思路是利用时空分离事件间的弱局部依赖,重构因果依赖图,实现无损并行生成:跨事件弱相关令牌可并行解码,而事件内强耦合令牌保持顺序解码以保证语义连贯性。为此引入两个关键组件:(1) 隐式全局规划机制,自动学习事件结构,生成编码全局事件间因果关系的紧凑令牌,并自适应聚合音视频语义;(2) 事件因子化并行解码机制,有效平衡局部关注与全局事件感知。在多个基准测试中,该方法在效率与性能上均显著优于现有方案。
原文摘要 · Abstract (English)
Dense video captioning aims to generate temporally grounded descriptions of video events, benefiting both event-level video understanding and generation. In this domain, autoregressive video large language models have emerged as a prevalent paradigm due to their strong generative and cross-modal modeling capacity. However, generating dense captions under the token-by-token paradigm severely limits inference efficiency and hinders scalability as video length and event density increase. In this work, we propose a parallelized autoregressive framework that not only improves generation efficiency but also enhances temporally grounded captioning performance. Our key insight is to exploit the weak local dependencies across temporally distinct events to restructure the causal dependency graph, thereby enabling lossless parallel generation. Specifically, tokens with weak cross-event dependencies can be decoded in parallel, while tightly coupled tokens within each event retain sequential decoding to preserve local semantic coherence. To realize this insight, we introduce two key components for lossless parallel decoding: (1) a latent global planning mechanism that automatically learns the event-level structure and produces compact tokens encoding global inter-event causality while adaptively aggregating event-level audio-visual semantics, guiding subsequent dependency restructuring and parallel decoding; and (2) an event-factorized parallel decoding mechanism that effectively balances local focus with global inter-event awareness. Experiments on various benchmarks demonstrate the clear advantage of our approach in both efficiency and performance in omni-modal event grounding and captioning. Project website: https://github.com/showlab/PadCaptioner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。