arXiv:2510.14032cs.CV2025-10NeurIPS被引 31

用图结构增强长视频理解,提升模型推理准确性。

Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding

  • 将视频切片构建为带语义关系的图结构,提升检索效率。
  • 引入中间推理步骤,使模型在多片段间更准确地整合信息。
  • 适合需要长视频分析的场景,如视频问答与内容摘要。

长视频的理解与推理对大视频语言模型(LVLMs)构成挑战,主要源于超出上下文窗口的密集视频标记处理困难以及长期序列信息难以保持。检索增强生成(RAG)在大语言模型中已证明有效,但应用于长视频时存在时间依赖性断裂和无关信息干扰等问题。为此,我们提出Vgent:一种基于图结构的检索-推理-增强生成框架,以提升LVLM对长视频的理解能力。该方法引入两项关键创新:(i) 将视频表示为包含跨片段语义关系的结构化图,增强检索有效性;(ii) 引入中间推理环节,通过结构化验证减少检索噪声,促进跨片段相关信息的显式聚合,从而实现更准确、更具上下文意识的响应。我们在三个长视频理解基准上,使用多种开源LVLM进行了全面评估。结果表明,在MLVU上整体性能相较基线模型提升3.0%~5.4%,优于现有最先进视频RAG方法8.6%。代码已公开于https://xiaoqian-shen.github.io/Vgent。

原文摘要 · Abstract (English)

Understanding and reasoning over long videos pose significant challenges for large video language models (LVLMs) due to the difficulty in processing intensive video tokens beyond context window and retaining long-term sequential information. Retrieval-Augmented Generation (RAG) has demonstrated effectiveness in processing long context for Large Language Models (LLMs); however, applying RAG to long video faces challenges such as disrupted temporal dependencies and inclusion of irrelevant information that can hinder accurate reasoning. To address these limitations, we propose Vgent, a novel graph-based retrieval-reasoning-augmented generation framework to enhance LVLMs for long video understanding. Our approach introduces two key innovations: (i) It represents videos by structured graphs with semantic relationships across video clips preserved to improve retrieval effectiveness. (ii) It introduces an intermediate reasoning step to mitigate the reasoning limitation of LVLMs, which leverages structured verification to reduce retrieval noise and facilitate the explicit aggregation of relevant information across clips, resulting in more accurate and context-aware responses. We comprehensively evaluate our framework with various open-source LVLMs on three long-video understanding benchmarks. Our approach yielded an overall performance improvement of $3.0\%\sim 5.4\%$ over base models on MLVU, and outperformed state-of-the-art video RAG methods by $8.6\%$. Our code is publicly available at https://xiaoqian-shen.github.io/Vgent.

视频理解图神经网络检索增强长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。