提出StreamFormer模型，实现低延迟视频流的高效理解
Learning Streaming Video Representation via Multitask Training
- 用因果时序注意力改造视觉Transformer，支持逐帧处理
- 多任务对齐训练使模型同时捕捉语义、动态与空间关系
- 在实时动作检测等任务中表现优异，适合自动驾驶场景
连续视频流的理解在具身智能和自动驾驶等实时应用中至关重要。与离线视频理解不同，流式视频理解需逐帧处理、保留历史信息并实现低延迟决策。为此，本文提出三项贡献：(i) 提出新型流式视频骨干网络StreamFormer，通过将因果时序注意力引入预训练视觉Transformer，实现高效流式处理并保持图像表征能力；(ii) 设计统一的多任务视觉-语言对齐框架，整合多种时空视频理解任务进行联合训练，使StreamFormer同步学习全局语义、时序动态与细粒度空间关系；(iii) 在在线动作检测、在线视频实例分割和视频问答任务上进行大量实验，结果表明该模型在保持高效性的同时取得有竞争力的表现，展现出在实时应用中的潜力。
原文摘要 · Abstract (English)
Understanding continuous video streams plays a fundamental role in real-time applications including embodied AI and autonomous driving. Unlike offline video understanding, streaming video understanding requires the ability to process video streams frame by frame, preserve historical information, and make low-latency decisions. To address these challenges, our main contributions are three-fold. (i) We develop a novel streaming video backbone, termed as StreamFormer, by incorporating causal temporal attention into a pre-trained vision transformer. This enables efficient streaming video processing while maintaining image representation capability. (ii) To train StreamFormer, we propose to unify diverse spatial-temporal video understanding tasks within a multitask visual-language alignment framework. Hence, StreamFormer learns global semantics, temporal dynamics, and fine-grained spatial relationships simultaneously. (iii) We conduct extensive experiments on online action detection, online video instance segmentation, and video question answering. StreamFormer achieves competitive results while maintaining efficiency, demonstrating its potential for real-time applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。