让多模态大模型边看视频边思考，支持连续多轮推理
Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models
- 采用分段记忆框架，边接收视频流边保持长期记忆
- 在StreamingBench上单轮准确率提升2.6%，多轮输出令牌减少56%
- 适合需要实时视频交互的场景，如智能客服、自动驾驶
多模态大语言模型在离线视频理解任务中表现优异，但多数仅支持离线推理或在线推理能力弱，难以实现对持续到达视频流的多轮交互。现有流式方法通常采用感知-生成交替模式，导致感知与生成无法并行，且随流增长出现早期记忆衰减，影响长程依赖建模。本文提出Think While Watching，一种基于记忆锚定的流式视频推理框架，可在多轮交互中持续保存分段级记忆。构建了三阶段、多轮链式思维数据集，并采用阶段匹配训练策略，通过分段级流式因果掩码和流式位置编码严格保证因果性。推理时引入高效流水线，实现观看与思考重叠，并自适应选择最优注意力后端。在单轮与多轮流式输入协议下均取得优异结果。基于Qwen3-VL，在StreamingBench上单轮准确率提升2.6%，在OVO-Bench上提升3.79%；多轮设置下性能稳定，输出令牌减少56%。代码已开源。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have shown strong performance on offline video understanding, but most are limited to offline inference or have weak online reasoning, making multi-turn interaction over continuously arriving video streams difficult. Existing streaming methods typically use an interleaved perception-generation paradigm, which prevents concurrent perception and generation and leads to early memory decay as streams grow, hurting long-range dependency modeling. We propose Think While Watching, a memory-anchored streaming video reasoning framework that preserves continuous segment-level memory during multi-turn interaction. We build a three-stage, multi-round chain-of-thought dataset and adopt a stage-matched training strategy, while enforcing strict causality through a segment-level streaming causal mask and streaming positional encoding. During inference, we introduce an efficient pipeline that overlaps watching and thinking and adaptively selects the best attention backend. Under both single-round and multi-round streaming input protocols, our method achieves strong results. Built on Qwen3-VL, it improves single-round accuracy by 2.6% on StreamingBench and by 3.79% on OVO-Bench. In the multi-round setting, it maintains performance while reducing output tokens by 56%. Code is available at: https://github.com/wl666hhh/Think_While_Watching/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。