用大模型生成的推理过程,让视频问答更准
ReasVQA: Advancing VideoQA with Imperfect Reasoning Process
- 用多模态大模型生成视频问答的推理过程
- 在三个数据集上分别提升2.9、7.3、5.9分
- 适合想提升视频理解能力的研究者
视频问答(VideoQA)是一项挑战性任务,需理解视频中复杂的视觉与时间关系以准确回答问题。本文提出ReasVQA(增强推理的视频问答),利用多模态大语言模型(MLLMs)生成的推理过程来提升视频问答模型性能。方法包含三个阶段:推理生成、推理精炼和基于推理的学习。首先使用额外的MLLM生成详细推理过程,其次通过过滤步骤确保数据质量,最后将可能不完美的推理数据用于多任务学习,引导视频问答模型如何根据视频内容理解并作答。我们在三个主流基准上评估该方法,结果在NExT-QA、STAR、IntentQA上分别取得+2.9、+7.3、+5.9的显著提升,达到新最优性能。实验验证了各组件有效性,并在不同骨干网络和MLLM下保持优势,表明引入推理过程对视频问答具有显著监督价值。
原文摘要 · Abstract (English)
Video Question Answering (VideoQA) is a challenging task that requires understanding complex visual and temporal relationships within videos to answer questions accurately. In this work, we introduce \textbf{ReasVQA} (Reasoning-enhanced Video Question Answering), a novel approach that leverages reasoning processes generated by Multimodal Large Language Models (MLLMs) to improve the performance of VideoQA models. Our approach consists of three phases: reasoning generation, reasoning refinement, and learning from reasoning. First, we generate detailed reasoning processes using additional MLLMs, and second refine them via a filtering step to ensure data quality. Finally, we use the reasoning data, which might be in an imperfect form, to guide the VideoQA model via multi-task learning, on how to interpret and answer questions based on a given video. We evaluate ReasVQA on three popular benchmarks, and our results establish new state-of-the-art performance with significant improvements of +2.9 on NExT-QA, +7.3 on STAR, and +5.9 on IntentQA. Our findings demonstrate the supervising benefits of integrating reasoning processes into VideoQA. Further studies validate each component of our method, also with different backbones and MLLMs, and again highlight the advantages of this simple but effective method. We offer a new perspective on enhancing VideoQA performance by utilizing advanced reasoning techniques, setting a new benchmark in this research field.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。