让AI实时理解视频并即时回应,速度达100帧/秒
StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition
- 用事件触发机制只在关键动作时调用大模型
- 100帧/秒处理速度,单张A100显卡实现
- 适合游戏AI、交互媒体等高实时场景
随着人机交互应用(如AI助手)兴起,实时流式视频对话需求迫切。本文提出StreamMind,一种视频大模型框架,可在单张A100上实现100帧/秒的超高速视频处理,支持主动、持续响应,无需用户显式触发。为解决视频流速线性与Transformer计算量二次增长的矛盾,提出新型感知-认知交替范式:事件触发的大模型调用。在视频编码器与大模型间引入认知门控网络,仅当检测到相关事件时才激活大模型。为实现恒定开销的事件特征提取,提出基于状态空间方法的事件保持特征提取器(EPFE),生成单一感知标记表示时空特征。该技术使视频大模型具备全帧率感知与实时认知响应能力。在Ego4D和SoccerNet流任务及标准离线基准上的实验表明,该模型在性能与实时效率方面均达到当前最优水平,为游戏AI、互动媒体等超高帧率应用铺平道路。代码与数据已公开于https://aka.ms/StreamMind。
原文摘要 · Abstract (English)
With the rise of real-world human-AI interaction applications, such as AI assistants, the need for Streaming Video Dialogue is critical. To address this need, we introduce StreamMind, a video LLM framework that achieves ultra-FPS streaming video processing (100 fps on a single A100) and enables proactive, always-on responses in real time, without explicit user intervention. To solve the key challenge of the contradiction between linear video streaming speed and quadratic transformer computation cost, we propose a novel perception-cognition interleaving paradigm named ''event-gated LLM invocation'', in contrast to the existing per-time-step LLM invocation. By introducing a Cognition Gate network between the video encoder and the LLM, LLM is only invoked when relevant events occur. To realize the event feature extraction with constant cost, we propose Event-Preserving Feature Extractor (EPFE) based on state-space method, generating a single perception token for spatiotemporal features. These techniques enable the video LLM with full-FPS perception and real-time cognition response. Experiments on Ego4D and SoccerNet streaming tasks, as well as standard offline benchmarks, demonstrate state-of-the-art performance in both model capability and real-time efficiency, paving the way for ultra-high-FPS applications, such as Game AI and interactive media. The code and data is available at https://aka.ms/StreamMind.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。