arXiv:2601.06843cs.CVcs.CL2026-01被引 7

让多模态大模型边看视频边说话,实现真正实时理解。

Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models

论文配图:Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models
图 1 · 摘自论文原文
  • 通过解耦位置编码,实现感知与生成并行处理
  • 在平衡负载下延迟降低50%,最高提速2倍
  • 适合需要实时交互的视频理解场景

多模态大语言模型(MLLMs)在众多任务中表现强劲,但多数系统仍受限于离线推理,需等待完整输入才生成输出。近期流式方法虽通过交替感知与生成降低延迟,但仍遵循顺序循环,难以实现真正实时交互。本文针对扩展MLLM至实时视频理解的核心瓶颈——标准位置编码带来的全局位置连续性约束提出解决方案。该约束将感知与生成紧密绑定,阻碍输入输出并行。为此,我们设计了一种并行流式框架,通过重叠、分组解耦与间隙隔离三项设计,放松位置连续性,实现感知与生成的同时进行。大量实验表明,分组解耦方案在效率与性能间取得最佳平衡,保持高流畅性与准确性的同时显著降低延迟。在感知与生成工作量均衡时,系统可实现最高2倍加速,为实现‘边看边说’的实时系统提供了理论基础。代码已公开:https://github.com/EIT-NLP/Speak-While-Watching。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have achieved strong performance across many tasks, yet most systems remain limited to offline inference, requiring complete inputs before generating outputs. Recent streaming methods reduce latency by interleaving perception and generation, but still enforce a sequential perception-generation cycle, limiting real-time interaction. In this work, we target a fundamental bottleneck that arises when extending MLLMs to real-time video understanding: the global positional continuity constraint imposed by standard positional encoding schemes. While natural in offline inference, this constraint tightly couples perception and generation, preventing effective input-output parallelism. To address this limitation, we propose a parallel streaming framework that relaxes positional continuity through three designs: Overlapped, Group-Decoupled, and Gap-Isolated. These designs enable simultaneous perception and generation, allowing the model to process incoming inputs while producing responses in real time. Extensive experiments reveal that Group-Decoupled achieves the best efficiency-performance balance, maintaining high fluency and accuracy while significantly reducing latency. We further show that the proposed framework yields up to 2x acceleration under balanced perception-generation workloads, establishing a principled pathway toward speak-while-watching real-time systems. We make all our code publicly available: https://github.com/EIT-NLP/Speak-While-Watching.

多模态实时推理视频理解大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。