FlowNar让长视频流实时解说更高效，内存和计算开销不随视频变长而增长。
FlowNar: Scalable Streaming Narration for Long-Form Videos
- 动态清理历史视觉信息，结合跨线性注意力记忆模块保留关键内容。
- 在多个数据集上实现3倍吞吐量提升，支持10倍更长视频处理。
- 专为真实部署环境设计评估方案，适合需要低延迟长视频处理的场景。
近期大型多模态模型（LMMs）主要面向离线场景设计，难以满足视频流的动态需求。尽管已有在线适配方法提升了实时处理能力，但仍面临关键可扩展性挑战，资源消耗通常随视频时长至少呈线性增长。为突破这一瓶颈，我们提出FlowNar，一种可扩展的长视频流解说框架。其核心是动态上下文管理策略，用于移除历史视觉信息，结合我们提出的CLAM（跨线性注意力记忆）模块，实现对流式视觉历史的有效保留，确保视觉记忆使用和计算复杂度保持有界，这对高效流式处理至关重要。我们还引入了贴近实际部署条件的自条件评估协议及补充评估指标。在Ego4D、EgoExo4D和EpicKitchens100数据集上的实验表明，FlowNar显著优于强基线，在保证高质量解说的同时，支持处理长达10倍的视频，并实现3倍更高的吞吐量（FPS）。代码已公开于https://github.com/zeyun-zhong/FlowNar。
原文摘要 · Abstract (English)
Recent Large Multimodal Models (LMMs), primarily designed for offline settings, are ill-suited for the dynamic requirements of streaming video. While recent online adaptations improve real-time processing, they still face critical scalability challenges, with resource demands typically growing at least linearly with video duration. To overcome this bottleneck, we propose FlowNar, a novel framework for scalable streaming video narration. The core of FlowNar is a dynamic context management strategy for historical visual context removal, combined with our CLAM (Cross Linear Attentive Memory) module for streaming visual history retention, ensuring bounded visual memory usage and computational complexity, crucial for efficient streaming. We also introduce a realistic self-conditioned evaluation protocol and complementary evaluation metrics to assess streaming narration models under deployment-like conditions. Experiments on the Ego4D, EgoExo4D, and EpicKitchens100 datasets demonstrate that FlowNar substantially improves narration quality over strong baselines while being highly efficient, supporting processing of 10$\times$ longer videos and achieving 3$\times$ higher throughput (FPS). The code is available at https://github.com/zeyun-zhong/FlowNar.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。