为实时视觉助手模型设计新评估框架,提升响应及时性与稳定性测试能力。
VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant Models

- 构建时序密集标注数据集,覆盖18000+标注点,支持多任务动态评估
- 提出同步/异步双评估协议,量化响应及时性与长期一致性表现
- 验证通用视频模型可零训练适配流式场景,性能超越专用流式模型
流式视觉语言模型(VLM)在接收到指令和实时输入帧流时持续生成响应,是实时视觉助手的核心机制。现有VLM框架多在离线环境下评估,而流式VLM的表现还需考量响应及时性(proactiveness)和时间稳定性(consistency)等额外指标。为此,我们提出VSAS-Bench,一个面向视觉流助手的新框架与基准。相较于以往仅依赖视频单轮问答的基准,VSAS-Bench包含跨多样化输入域与任务类型的超18,000条时序密集标注。我们引入标准化的同步与异步评估协议,并设计可分离测量流式VLM不同能力的指标。利用该框架,我们对近期视频与流式VLM进行了大规模评估,分析了记忆缓冲长度、内存访问策略、输入分辨率等关键设计因素下的准确率-延迟权衡,获得多项实用洞见。最终实证表明,传统VLM无需额外训练即可适配流式场景,且经适配后的模型性能优于现有专用流式VLM。例如,Qwen3-VL-4B在异步协议下较最佳流式模型Dispider提升3%。基准与代码将开源于https://github.com/apple/ml-vsas-bench。
原文摘要 · Abstract (English)
Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core mechanism for real-time visual assistants. Existing VLM frameworks predominantly assess models in offline settings. In contrast, the performance of a streaming VLM depends on additional metrics beyond pure video understanding, including proactiveness, which reflects the timeliness of the model's responses, and consistency, which captures the robustness of its responses over time. To address this limitation, we propose VSAS-Bench, a new framework and benchmark for Visual Streaming Assistants. In contrast to prior benchmarks that primarily employ single-turn question answering on video inputs, VSAS-Bench features temporally dense annotations with over 18,000 annotations across diverse input domains and task types. We introduce standardized synchronous and asynchronous evaluation protocols, along with metrics that isolate and measure distinct capabilities of streaming VLMs. Using this framework, we conduct large-scale evaluations of recent video and streaming VLMs, analyzing the accuracy-latency trade-off under key design factors such as memory buffer length, memory access policy, and input resolution, yielding several practical insights. Finally, we show empirically that conventional VLMs can be adapted to streaming settings without additional training, and demonstrate that these adapted models outperform recent streaming VLMs. For example, Qwen3-VL-4B surpasses Dispider, the best streaming VLM on our benchmark, by 3% under the asynchronous protocol. The benchmark and code will be available at https://github.com/apple/ml-vsas-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。