提出StreamOV框架,实现视频流的实时感知与主动响应。
StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering

- 用多模态证据引导记忆压缩长期视听上下文
- 通过隐状态驱动触发机制实现适时响应,无需沉默标记
- 构建首个在线多轮评测基准SOVBench,适合实时交互场景
流式全模态视频理解需要持续感知与主动实时交互,但该领域仍研究不足。现有全模态方法本质为离线设计,难以适配流式场景,存在两大缺陷:一是缺乏对长期持续增长的音视频上下文的有效管理,无法自主选择时机发起响应;二是现有基准多限于离线单轮问答,无法刻画连续多轮流式交互。为此,我们提出StreamOV,一种新型流式全模态视频理解框架,支持有限内存下的高效在线音视频推理与主动响应触发。StreamOV引入多模态证据引导的长短时记忆机制,在固定预算下将历史音视频上下文压缩为紧凑信息证据;同时采用隐状态驱动的触发器决定响应时机,避免显式生成沉默标记或外部路由。我们还构建了首个面向在线多轮全模态评估的SOVBench基准。大量实验表明,StreamOV在多种流式与全模态基准上均达到领先性能,验证了其在在线与离线视频理解中的有效性。
原文摘要 · Abstract (English)
While streaming omni-video understanding demands continuous perception and proactive, real-time interaction, this crucial area remains largely under-explored. Current omni-modal methods are inherently designed for offline settings, limiting their applicability in streaming scenarios due to two fundamental flaws. First, they lack robust mechanisms to manage continuously growing audio-visual context over long horizons and cannot autonomously initiate responses at opportune moments. Second, existing benchmarks are predominantly confined to offline, single-turn question answering, failing to capture continuous, multi-turn streaming interactions. To bridge these gaps, we propose StreamOV, a novel Streaming Omni-Video understanding framework for efficient online audio-visual reasoning with bounded memory and proactive response triggering. Specifically, StreamOV introduces a multimodal evidence-guided long-short term memory that condenses historical audio-visual context into compact informative evidence under a fixed budget. It further employs a hidden-state-driven trigger to decide when to respond, avoiding explicit silence-token generation and external routers. We also curate SOVBench, the first comprehensive benchmark for online, multi-turn omni-modal evaluation. Extensive experiments show that StreamOV achieves state-of-the-art performance across diverse streaming and omni-video benchmarks, demonstrating its effectiveness for both online and offline video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。