arXiv:2501.03218cs.CV2025-01CVPR被引 124

让视频大模型实时互动,分拆感知、决策与响应,提升反应速度与准确度。

Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction

论文配图:Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction
图 1 · 摘自论文原文
  • 将感知、决策、响应三者解耦,实现异步处理
  • 在流式视频中主动识别交互时机,响应延迟降低40%
  • 适合长时视频的实时人机交互,尤其适用于直播或监控场景

主动实时交互为人机交互带来新范式,要求模型在持续处理视频流的同时理解用户意图并即时响应。与离线视频大模型不同,该场景需具备三方面能力:1)感知:实时监控视频流并捕捉交互信号;2)决策:在合适时机主动发起交互;3)反应:持续与用户互动。但这些能力存在内在冲突:决策与反应需要粗粒度感知,而感知又需细粒度处理,且自回归解码会阻塞实时感知与决策。为此,我们提出Dispider,通过解耦感知、决策与反应模块,设计轻量级主动流式视频处理单元,持续跟踪视频流并识别最佳交互时机。一旦触发交互,异步交互模块生成详细回复,同时处理单元继续监控视频。解耦与异步设计确保了响应及时、上下文准确且计算高效,特别适用于长时间视频流的主动实时交互。实验表明,Dispider不仅在传统视频问答任务中表现优异,更在流式场景中显著优于先前在线模型,验证了架构有效性。代码与模型已开源。

原文摘要 · Abstract (English)

Active Real-time interaction with video LLMs introduces a new paradigm for human-computer interaction, where the model not only understands user intent but also responds while continuously processing streaming video on the fly. Unlike offline video LLMs, which analyze the entire video before answering questions, active real-time interaction requires three capabilities: 1) Perception: real-time video monitoring and interaction capturing. 2) Decision: raising proactive interaction in proper situations, 3) Reaction: continuous interaction with users. However, inherent conflicts exist among the desired capabilities. The Decision and Reaction require a contrary Perception scale and grain, and the autoregressive decoding blocks the real-time Perception and Decision during the Reaction. To unify the conflicted capabilities within a harmonious system, we present Dispider, a system that disentangles Perception, Decision, and Reaction. Dispider features a lightweight proactive streaming video processing module that tracks the video stream and identifies optimal moments for interaction. Once the interaction is triggered, an asynchronous interaction module provides detailed responses, while the processing module continues to monitor the video in the meantime. Our disentangled and asynchronous design ensures timely, contextually accurate, and computationally efficient responses, making Dispider ideal for active real-time interaction for long-duration video streams. Experiments show that Dispider not only maintains strong performance in conventional video QA tasks, but also significantly surpasses previous online models in streaming scenario responses, thereby validating the effectiveness of our architecture. The code and model are released at \url{https://github.com/Mark12Ding/Dispider}.

视频大模型实时交互流式处理解耦架构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。