arXiv:2603.12938cs.CVcs.AI2026-03被引 8

让AI像人一样实时理解连续视频流,边看边思考。

Thinking in Streaming Video

  • 采用看-想-说的渐进式推理框架,随新帧到达即时更新理解。
  • 在多个基准上表现超越现有模型,延迟和内存占用更低。
  • 适合需要低延迟交互的智能助手、多模态机器人等场景。

实时理解连续视频流对动态环境中运行的交互式助手和多模态智能体至关重要。然而,大多数现有视频推理方法采用批处理范式,需等待完整视频上下文才进行推理,导致高延迟和不断增长的计算开销,不适用于流式场景。本文提出ThinkStream框架,基于看-想-说范式,使模型能随着新视频观测的到达逐步更新理解。每一步中,模型执行简短推理更新,并判断是否已积累足够证据生成响应。为支持长时序流式处理,提出推理压缩流式记忆(RCSM),将中间推理轨迹作为紧凑语义记忆,替代过时的视觉标记,同时保留关键上下文。进一步采用可验证奖励的流式强化学习训练策略,使增量推理与响应时机符合流式交互需求。在多个流式视频基准上的实验表明,ThinkStream显著优于现有在线视频模型,同时保持低延迟与低内存消耗。代码、模型与数据将开源于https://github.com/johncaged/ThinkStream。

原文摘要 · Abstract (English)

Real-time understanding of continuous video streams is essential for interactive assistants and multimodal agents operating in dynamic environments. However, most existing video reasoning approaches follow a batch paradigm that defers reasoning until the full video context is observed, resulting in high latency and growing computational cost that are incompatible with streaming scenarios. In this paper, we introduce ThinkStream, a framework for streaming video reasoning based on a Watch--Think--Speak paradigm that enables models to incrementally update their understanding as new video observations arrive. At each step, the model performs a short reasoning update and decides whether sufficient evidence has accumulated to produce a response. To support long-horizon streaming, we propose Reasoning-Compressed Streaming Memory (RCSM), which treats intermediate reasoning traces as compact semantic memory that replaces outdated visual tokens while preserving essential context. We further train the model using a Streaming Reinforcement Learning with Verifiable Rewards scheme that aligns incremental reasoning and response timing with the requirements of streaming interaction. Experiments on multiple streaming video benchmarks show that ThinkStream significantly outperforms existing online video models while maintaining low latency and memory usage. Code, models and data will be released at https://github.com/johncaged/ThinkStream

视频理解实时推理多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。