arXiv:2605.26485cs.CVcs.CL2026-05

构建实时多模态交互基准,测试大模型在音视频流中边听边答的能力。

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants

论文配图:OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
图 1 · 摘自论文原文
  • 设计真实音视频流上的在线推理任务,要求模型实时响应触发信号。
  • 最佳模型响应准确率仅0.368,连续任务响应准确率低至0.052。
  • 适合研究实时多模态交互、人机协同系统的开发者与研究人员。

我们提出OmniInteract,一个面向实时多模态大语言模型的流式交互基准,通过原生在线推理评估模型对音视频流的处理能力。与离线视频理解或文本提示的流式问答不同,OmniInteract保留原始音视频流,要求模型在无法预知未来内容的情况下在线处理。用户问题和环境声音嵌入音频流中,模型需检测多模态触发信号、判断响应时机,并在流进行时作答。数据集包含250个视频,共1,430个时间对齐的响应槽:1,062个1Q1A槽(涵盖实时、主动及嵌套场景),368个1QnA槽用于持续任务监控与步骤引导。每个槽包含触发信号、响应窗口和目标答案。采用交互感知质量-时效性F1、中断诊断套件和嵌套链完成度评分评估响应准确性、及时性、无效输出、中断处理及上下文连贯性。实验表明,当前模型在流式交互中表现薄弱,最优整体IA-QTF1仅为0.368,1QnA最优值仅0.052。进一步研究表明,离线推理能力并不必然转化为全双工在线交互能力。代码与数据集将公开于https://github.com/Lucky-Lance/OmniInteract。

原文摘要 · Abstract (English)

We introduce OmniInteract, a streaming benchmark for real-time omnimodal large language models evaluated through native online inference over audio-visual streams. Unlike offline video understanding or text-prompted streaming QA, OmniInteract preserves the original audio-visual stream and requires models to process it online, without access to future content. User queries and ambient sounds are embedded in the audio track, requiring models to detect multimodal triggers, decide when to respond, and answer while the stream unfolds. OmniInteract contains 250 videos with 1,430 temporally grounded response slots: 1,062 1Q1A slots across real-time, proactive, and nested scenarios, and 368 1QnA slots for continuous task monitoring and step guidance. Each slot includes a trigger, response window, and target answer. We evaluate response correctness, timing, invalid outputs, interruption handling, and context continuity using Interaction-Aware Quality-Timeliness F1, Interruption Diagnostic Suite, and Nested Chain Completion Score. Experiments show that current models remain weak in streaming interaction, with the best overall IA-QTF1 reaching only 0.368 and the best 1QnA IA-QTF1 only 0.052. Further study on mathematical reasoning in full-duplex settings shows that offline capability does not necessarily transfer to online interaction. Code and datasets will be made publicly accessible at https://github.com/Lucky-Lance/OmniInteract.

多模态交互实时推理流式处理大模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。