arXiv:2601.22575cs.CVcs.CL2026-01被引 10

首个面向移动端多模态助手的实时流评测基准,测试模型在真实场景下的响应时机与理解能力。

PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarios

  • 构建自动化生成+人工验证的流水线,覆盖四种场景十项能力
  • 模型在即时与回溯任务得分超80,前向任务仅16.40分
  • 揭示当前大模型不缺内容回答,但难把握说话时机

多模态大语言模型在离线音视频理解上表现优异,但在连续真实流媒体场景中作为移动端助手的能力仍待探索。日常手机使用中,助手需持续追踪音视频输入并适时响应,但现有评测多限于选择题或短视频。本文提出PhoStream,首个以移动端为中心的实时流评测基准,融合屏内与屏外场景,评估视频、音频及时间推理能力。数据集包含578段视频中的5,572个开放问答对,覆盖4种场景与10项能力。通过自动化生成管道结合严格人工验证构建,并采用真实在线推理流程与大模型评分机制评估开放式回答。实验显示,大模型评分(0-100)存在时间不对称:即时与回溯任务得分较高(Gemini 3 Pro超80),而前向任务骤降至16.40,主因是模型在视觉与音频信号出现前过早回应。这揭示了当前多模态大模型的核心瓶颈:不是说什么,而是何时说。代码与数据集将公开于https://github.com/Lucky-Lance/PhoStream。

原文摘要 · Abstract (English)

Multimodal Large Language Models excel at offline audio-visual understanding, but their ability to serve as mobile assistants in continuous real-world streams remains underexplored. In daily phone use, mobile assistants must track streaming audio-visual inputs and respond at the right time, yet existing benchmarks are often restricted to multiple-choice questions or use shorter videos. In this paper, we introduce PhoStream, the first mobile-centric streaming benchmark that unifies on-screen and off-screen scenarios to evaluate video, audio, and temporal reasoning. PhoStream contains 5,572 open-ended QA pairs from 578 videos across 4 scenarios and 10 capabilities. We build it with an Automated Generative Pipeline backed by rigorous human verification, and evaluate models using a realistic Online Inference Pipeline and LLM-as-a-Judge evaluation for open-ended responses. Experiments reveal a temporal asymmetry in LLM-judged scores (0-100): models perform well on Instant and Backward tasks (Gemini 3 Pro exceeds 80), but drop sharply on Forward tasks (16.40), largely due to early responses before the required visual and audio cues appear. This highlights a fundamental limitation: current MLLMs struggle to decide when to speak, not just what to say. Code and datasets used in this work will be made publicly accessible at https://github.com/Lucky-Lance/PhoStream.

多模态实时流移动端评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。