arXiv:2502.10810cs.CV2025-02ICLR被引 60

构建首个面向流式视频理解的多轮时序问答基准,评估模型长期推理能力。

SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding

  • 设计半自动标注流程生成49,979条多轮问答链,覆盖1,353段流式视频
  • 14个模型测试显示开源模型普遍难处理长时序视频,闭源GPT-4o表现最优
  • 提出StreamingChat模型,在本基准上显著超越开源模型,兼容多场景

尽管大视觉语言模型(LVLMs)在现有基准上取得显著进展,但在长上下文流式视频理解这一新兴领域仍缺乏合适的评估体系。现有视频理解基准多聚焦孤立单实例文本输入,无法评估模型在视频全程中持续进行时序推理的能力。为此,我们提出SVBench,首个专为流式视频理解设计的时序多轮问答基准,用于全面评估当前LVLMs在流式场景下的能力。通过半自动化标注流程,构建了包含49,979个问答对的1,353段流式视频数据集,涵盖连续多轮对话链,并建立相邻问答链间的时序关联。14个模型在对话与流式评测中的实验结果表明,虽闭源GPT-4o表现领先,多数开源LVLMs仍难以应对长时序视频理解。我们进一步构建StreamingChat模型,在SVBench上显著优于开源模型,并在多个视觉语言基准上达到可比性能。我们期望该基准推动流式视频理解研究的发展。项目代码与数据已公开于https://github.com/sotayang/SVBench。

原文摘要 · Abstract (English)

Despite the significant advancements of Large Vision-Language Models (LVLMs) on established benchmarks, there remains a notable gap in suitable evaluation regarding their applicability in the emerging domain of long-context streaming video understanding. Current benchmarks for video understanding typically emphasize isolated single-instance text inputs and fail to evaluate the capacity to sustain temporal reasoning throughout the entire duration of video streams. To address these limitations, we introduce SVBench, a pioneering benchmark with temporal multi-turn question-answering chains specifically designed to thoroughly assess the capabilities of streaming video understanding of current LVLMs. We design a semi-automated annotation pipeline to obtain 49,979 Question-Answer (QA) pairs of 1,353 streaming videos, which includes generating QA chains that represent a series of consecutive multi-turn dialogues over video segments and constructing temporal linkages between successive QA chains. Our experimental results, obtained from 14 models in dialogue and streaming evaluations, reveal that while the closed-source GPT-4o outperforms others, most open-source LVLMs struggle with long-context streaming video understanding. We also construct a StreamingChat model, which significantly outperforms open-source LVLMs on our SVBench and achieves comparable performance on diverse vision-language benchmarks. We expect SVBench to advance the research of streaming video understanding by providing a comprehensive and in-depth analysis of current LVLMs. Our benchmark and model can be accessed at https://github.com/sotayang/SVBench.

视频理解多轮对话流式处理基准评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。