提出WAT框架,让视频模型先看后想,实时处理长视频流
WAT: Online Video Understanding Needs Watching Before Thinking

- 分两阶段:先无查询地积累视频记忆,再根据问题检索历史信息
- 在StreamingBench上达77.7%准确率,实时运行且内存可控
- 适合需要实时理解长视频的应用,如直播分析、自动驾驶
多模态大语言模型在图像理解中表现优异,但现有视频大模型在在线流媒体场景下受限于严格内存约束,难以保持长时上下文。本文提出WAT(Watching Before Thinking)框架,将处理分为两个阶段:独立于查询的观看阶段与基于查询的思考阶段。观看阶段构建分层记忆系统,短期记忆(STM)缓冲近期帧,长期记忆(LTM)以去冗余策略固定容量存储历史内容摘要。思考阶段通过上下文感知检索机制,结合当前STM与查询,从LTM中召回相关历史帧进行跨时间推理。为支持训练,引入WAT-85K数据集,包含强调实时感知、回溯和预测的流式标注。实验表明,WAT在在线视频基准上达到领先性能,在StreamingBench上准确率达77.7%,OVO-Bench上达55.2%,优于现有开源视频大模型,并实现实时帧率运行。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown strong capabilities in image understanding, motivating recent efforts to extend them to video reasoning. However, existing Video LLMs struggle in online streaming scenarios, where long temporal context must be preserved under strict memory constraints. We propose WAT (Watching Before Thinking), a two-stage framework for online video reasoning. WAT separates processing into a query-independent watching stage and a query-triggered thinking stage. The watching stage builds a hierarchical memory system with a Short-Term Memory (STM) that buffers recent frames and a fixed-capacity Long-Term Memory (LTM) that maintains a diverse summary of historical content using a redundancy-aware eviction policy. In the thinking stage, a context-aware retrieval mechanism combines the query with the current STM context to retrieve relevant historical frames from the LTM for cross-temporal reasoning. To support training for online video tasks, we introduce WAT-85K, a dataset containing streaming-style annotations emphasizing real-time perception, backward tracing, and forecasting. Experiments show that WAT achieves state-of-the-art performance on online video benchmarks, including 77.7% accuracy on StreamingBench and 55.2% on OVO-Bench, outperforming existing open-source online Video LLMs while operating at real-time frame rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。