让视频理解模型提前预测关键信息位置,实现主动响应。
StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding
- 通过提示预判未来关键事件的时间与空间位置
- 在长视频流中提升响应准确率与实时效率
- 适合自动驾驶、智能监控等实时场景
在自动驾驶、智能监控等场景中,实时流式视频理解面临连续感知、主动决策和动态响应的挑战,传统方法依赖感知-反应交替或异步触发,缺乏任务驱动的规划与未来预判,限制了实时性与主动性。为此,我们提出 StreamAgent,通过提示机制让代理预判未来任务相关的关键事件发生的时间区间与空间区域,实现主动且目标导向的响应。具体而言,该模型融合问题语义与历史观测,预判事件发展进程,对齐当前观察与预期未来证据,并据此调整感知动作(如关注任务相关区域或持续跟踪)。为提升推理效率,设计了分层的流式 KV-cache 内存机制,构建可选择性召回的层次化记忆结构,在减少存储开销的同时实现高效语义检索。在流式与长视频理解任务上的大量实验表明,本方法在响应准确率与实时效率上均优于现有方法,凸显其在真实流式场景中的实用价值。
原文摘要 · Abstract (English)
Real-time streaming video understanding in domains such as autonomous driving and intelligent surveillance poses challenges beyond conventional offline video processing, requiring continuous perception, proactive decision making, and responsive interaction based on dynamically evolving visual content. However, existing methods rely on alternating perception-reaction or asynchronous triggers, lacking task-driven planning and future anticipation, which limits their real-time responsiveness and proactive decision making in evolving video streams. To this end, we propose a StreamAgent that anticipates the temporal intervals and spatial regions expected to contain future task-relevant information to enable proactive and goal-driven responses. Specifically, we integrate question semantics and historical observations through prompting the anticipatory agent to anticipate the temporal progression of key events, align current observations with the expected future evidence, and subsequently adjust the perception action (e.g., attending to task-relevant regions or continuously tracking in subsequent frames). To enable efficient inference, we design a streaming KV-cache memory mechanism that constructs a hierarchical memory structure for selective recall of relevant tokens, enabling efficient semantic retrieval while reducing the overhead of storing all tokens in the traditional KV-cache. Extensive experiments on streaming and long video understanding tasks demonstrate that our method outperforms existing methods in response accuracy and real-time efficiency, highlighting its practical value for real-world streaming scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。