用时序缩放的状态空间模型实现长视频实时字幕生成
Time-Scaling State-Space Models for Dense Video Captioning
- 通过转移状态机制增强状态空间模型的长期记忆能力
- 处理长视频时计算量减少7倍，支持流式在线生成
- 适合需要实时响应的视频字幕场景，如直播分析
密集视频字幕是一项挑战性任务，需同时将视频分割为连续有意义的事件，并为每个事件生成详细描述。现有方法在处理长视频时受限于计算复杂度和内存瓶颈，且通常需完整视频输入才能输出，无法支持在线处理。本文提出一种时序缩放的状态空间模型（SSMs），引入转移状态机制，结合了SSM的长序列建模与循环特性，克服了其长期上下文维持能力弱的问题，显著扩展了模型的时间跨度。该方法特别适用于流式或在线场景下的字幕生成，无需等待全视频输入即可实时输出。在密集视频字幕任务中，模型可高效处理更长视频，计算量降低7倍。
原文摘要 · Abstract (English)
Dense video captioning is a challenging video understanding task which aims to simultaneously segment the video into a sequence of meaningful consecutive events and to generate detailed captions to accurately describe each event. Existing methods often encounter difficulties when working with the long videos associated with dense video captioning, due to the computational complexity and memory limitations. Furthermore, traditional approaches require the entire video as input, in order to produce an answer, which precludes online processing of the video. We address these challenges by time-scaling State-Space Models (SSMs) to even longer sequences than before. Our approach, State-Space Models with Transfer State, combines both the long-sequence and recurrent properties of SSMs and addresses the main limitation of SSMs which are otherwise not able to sustain their state for very long contexts, effectively scaling SSMs further in time. The proposed model is particularly suitable for generating captions on-the-fly, in an online or streaming manner, without having to wait for the full video to be processed, which is more beneficial in practice. When applied to dense video captioning, our approach scales well with video lengths and uses 7x fewer FLOPs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。