让视频大模型边看边思考,实时响应且推理更准。
Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously

- 设计边看边想机制,将推理延迟分摊到视频播放中。
- 在StreamingBench上达79.5%准确率,比Video-R1快15.7倍。
- 适合需要实时交互的视频理解场景,如智能客服、教育助手。
在线视频大语言模型在支持实时交互中至关重要。现有方法仅关注流式感知,缺乏同步逻辑推理。直接应用测试时扩展会引发不可接受的响应延迟。为此,我们提出视频流式思考(VST),一种新型流式视频理解范式,支持观看同时思考,可在视频流中激活对片段的推理。该设计通过将大模型推理延迟分摊至视频播放过程,实现及时理解与连贯认知,同时保持实时响应。我们还构建了完整后训练流程:VST-SFT结构化适配离线视频模型为因果流式推理;VST-RL通过多轮视频交互中的自我探索实现端到端优化。此外,我们设计自动化数据合成管道,利用视频知识图谱生成高质量流式问答对,并基于实体关系的流式思维链强制多证据推理与持续注意力。大量实验表明,VST-7B在在线基准上表现优异,如StreamingBench达79.5%,OVO-Bench达59.3%。同时,在离线长视频或推理基准上仍具竞争力。相比Video-R1,VST响应速度快15.7倍,且在VideoHolmes上提升5.4%,展现出更高效率与强泛化能力。代码、数据与模型将开源于https://github.com/1ranGuan/VST。
原文摘要 · Abstract (English)
Online Video Large Language Models (VideoLLMs) play a critical role in supporting responsive, real-time interaction. Existing methods focus on streaming perception, lacking a synchronized logical reasoning stream. However, directly applying test-time scaling methods incurs unacceptable response latency. To address this trade-off, we propose Video Streaming Thinking (VST), a novel paradigm for streaming video understanding. It supports a thinking while watching mechanism, which activates reasoning over incoming video clips during streaming. This design improves timely comprehension and coherent cognition while preserving real-time responsiveness by amortizing LLM reasoning latency over video playback. Furthermore, we introduce a comprehensive post-training pipeline that integrates VST-SFT, which structurally adapts the offline VideoLLM to causal streaming reasoning, and VST-RL, which provides end-to-end improvement through self-exploration in a multi-turn video interaction environment. Additionally, we devise an automated training-data synthesis pipeline that uses video knowledge graphs to generate high-quality streaming QA pairs, with an entity-relation grounded streaming Chain-of-Thought to enforce multi-evidence reasoning and sustained attention to the video stream. Extensive evaluations show that VST-7B performs strongly on online benchmarks, e.g. 79.5% on StreamingBench and 59.3% on OVO-Bench. Meanwhile, VST remains competitive on offline long-form or reasoning benchmarks. Compared with Video-R1, VST responds 15.7 times faster and achieves +5.4% improvement on VideoHolmes, demonstrating higher efficiency and strong generalization across diverse video understanding tasks. Code, data, and models will be released at https://github.com/1ranGuan/VST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。