让AI看视频时不停顿,实时生成语言回应。
Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding

- 用状态机决定何时说话、停顿或开始新回应。
- 动态调整文字生成速度,每帧只输出少量词,不卡顿。
- 适合需要实时交互的直播场景,如在线讲解或客服。
在线视频大语言模型通过逐帧处理和主动响应,推动了人机交互的流畅性。然而在流媒体场景中,现有模型通常在生成回应时暂停视觉感知,破坏实时同步并造成卡顿。为此,我们提出一种新型在线视频理解范式——流式视频-语言同步(SVLS),并构建了基于分层控制框架的实时助手LyraV,包含两项核心创新:其一,无需训练的帧驱动状态转移控制器(FDTC),基于验证机制的有限状态机,用于判断是否继续说话、启动新回应或保持沉默;其二,即插即用的轻量级预测模块——流式令牌速率器(SToP),动态调节语言生成速率以匹配视觉内容节奏。具体而言,LyraV采用每帧增量、预算内解码策略:在每个帧间隔内仅生成符合实时预算的少量词元,确保感知永不阻塞。二者协同使视频帧与生成词元无缝交织,实现细粒度同步。在五个在线和三个离线基准上的大量实验表明,LyraV在保持主干模型通用理解能力的同时,显著提升流媒体同步性和叙事连贯性,达到98.29%的视频播放同步率,实现实时处理速度3.89 FPS。有趣的是,我们观察到LyraV具备动态推理能力:能持续对流式词元进行解释与“思考”,伴随视觉输入不断演进。
原文摘要 · Abstract (English)
Online Video Large Language Models (Video-LLMs) have advanced toward seamless human-AI interaction through frame-by-frame processing and proactive responding. However, a critical challenge remains in streaming scenarios: existing models typically pause video perception while generating responses, breaking real-time video-language synchrony and causing stutters. To address this, we introduce a novel paradigm for online video understanding: Streaming Video-Language Synchrony (SVLS), and present LyraV, a live streaming assistant built upon a hierarchical control framework with two core innovations. First, the Frame-Driven Transition Controller (FDTC), a training-free verification-based finite-state machine, makes high-level semantic decisions on when to continue speaking, start a new response, or stay silent. Second, the Streaming Token Pacer (SToP), a plug-and-play lightweight predictive module, dynamically adapts the language generation rate to match the pace of the visual content. Concretely, LyraV performs \emph{per-frame incremental, sub-budget decoding}: within each frame interval it emits only a small chunk of tokens that fits the real-time budget, so perception is never blocked for a full sentence. Together, these components enable LyraV to seamlessly interleave incoming video frames with generated word tokens, achieving a fine-grained synchrony. Extensive experiments conducted on five online and three offline benchmarks demonstrate that LyraV preserves the backbone's general understanding ability while substantially improving streaming synchrony and narrative fluency, delivering a 98.29\% synchrony with video playback and a real-time processing speed of 3.89 FPS. Interestingly, we observe an empirical capability in LyraV: dynamic reasoning over streaming tokens, enabling continuous interpretation and "thinking" alongside visual input.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。