让大模型在长视频中精准回答密集事件问题，提升时序理解能力。
Question-Answering Dense Video Events
- 提出分层摘要+时序记忆+自洽验证的无训练框架，增强长视频事件理解
- 在DeVE-QA和NExT-GQA上分别提升4.8%和2.1%的问答准确率
- 适用于需要细粒度视频理解的多事件推理任务，如智能监控与教育分析
本文提出密集视频事件问答任务，旨在长视频中回答并定位多个事件相关问题，挑战大模型对长时间跨度事件的忠实理解和推理能力。为此，我们构建了DeVE-QA数据集，包含10.6K条长视频中的26,000个事件，以及78,000个相关问题。基准测试显示当前先进多模态大模型在该任务上表现不佳。为改进，我们提出DeVi——一种无需训练的多模态大模型方法，包含分层摘要模块、时序事件记忆模块和自洽检查模块，分别用于检测、上下文化与记忆、定位长视频中的密集事件。大量实验表明，DeVi在密集事件问答与视频片段定位上显著优于现有模型，在DeVE-QA和NExT-GQA上分别实现4.8%和2.1%的准确率提升。数据与代码已开源。
原文摘要 · Abstract (English)
This paper presents question-answering on dense video events, a novel task that answers and grounds dense-event questions in long videos, thus challenging MLLMs to faithfully comprehend and reason about multiple events over extended periods of time. To facilitate the study, we construct DeVE-QA -- a dataset featuring 78K questions about 26K events on 10.6K long videos. Our benchmarking shows that state-of-the-art MLLMs struggle on DeVE-QA. For improvement, we propose DeVi, a novel training-free MLLM approach that highlights a hierarchical captioning module, a temporal event memory module, and a self-consistency checking module to respectively detect, contextualize and memorize, and ground dense-events in long videos for question answering. Extensive experiments show that DeVi is superior at answering dense-event questions and grounding relevant video moments. Compared with existing MLLMs, it achieves a notable increase of 4.8% and 2.1% for G(round)QA accuracy on DeVE-QA and NExT-GQA, respectively. Data and code are available at https://github.com/QHUni/DeVE-QA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。